Hey, I'm having several issues with regards to 3rdparty dependencies and how they behave with different lockfile generators and different goals. Maybe someone has an idea what I could try.

Can you please provide more info? • What command(s) are you running? • Whats the output? • What is the expected result?

In the attached folder, I try to use the requirements cryptography==36.0.2 and docker==5.0.3. I'm using Ubuntu 22.04, pants 2.11.0 and restrict the interpreter to CPython>=3.10,<4. I'm activating enable_resolves. When I activate "pex" as the lockfile_generator, I get the following error during generation of the python-default lockfile:

./pants generate-lockfiles --resolve=python-default
17:27:27.51 [INFO] Initializing scheduler...
17:27:27.61 [INFO] Scheduler initialized.
17:27:27.63 [WARN] Please either set `enabled = true` in the [anonymous-telemetry] section of pants.toml to enable sending anonymous stats to the Pants project to aid development, or set `enabled = false` to disable it. No telemetry sent for this run. An explicit setting will get rid of this message. See <https://www.pantsbuild.org/v2.11/docs/anonymous-telemetry> for details.
17:27:30.86 [INFO] Completed: Generate lockfile for python-default
17:27:30.86 [ERROR] 1 Exception encountered:

  ProcessExecutionFailure: Process 'Generate lockfile for python-default' failed with exit code 1.
stdout:

stderr:
ERROR: Could not find a version that satisfies the requirement pywin32==227; sys_platform == "win32" (from docker)
ERROR: No matching distribution found for pywin32==227; sys_platform == "win32"
pid 147208 -> /home/********/.cache/pants/named_caches/pex_root/venvs/4a17ee4a3e5aca6b7a9e86fbf3e8687b9afe3820/5985ed09b49a653d6596b0e14d134c5456cf1a9f/pex --disable-pip-version-check --no-python-version-warning --exists-action a --isolated -q --cache-dir /home/********/.cache/pants/named_caches/pex_root --log /tmp/pex-pip-logmx51wm31/pip.log download --dest /tmp/tmpc4n1o1oe/usr.bin.python3.10 cryptography==36.0.2 docker==5.0.3 --index-url <https://pypi.org/simple/> --retries 5 --timeout 15 exited with 1 and STDERR:
None



Use `--no-process-cleanup` to preserve process chroots for inspection.

When I switch over to

poetry

however, cryptography is causing problems for me when calling

./pants export ::

after the lockfile generation:

./pants export ::                                  
Traceback (most recent call last):
  File "/home/********/.pex/unzipped_pexes/479026d892c9306a8907938e1e49f73e1514eabe/.bootstrap/pex/pex.py", line 504, in execute
    exit_value = self._wrap_coverage(self._wrap_profiling, self._execute)
  File "/home/********/.pex/unzipped_pexes/479026d892c9306a8907938e1e49f73e1514eabe/.bootstrap/pex/pex.py", line 409, in _wrap_coverage
    return runner(*args)
  File "/home/********/.pex/unzipped_pexes/479026d892c9306a8907938e1e49f73e1514eabe/.bootstrap/pex/pex.py", line 440, in _wrap_profiling
    return runner(*args)
  File "/home/********/.pex/unzipped_pexes/479026d892c9306a8907938e1e49f73e1514eabe/.bootstrap/pex/pex.py", line 556, in _execute
    return self.execute_entry(self._pex_info_overrides.entry_point)
  File "/home/********/.pex/unzipped_pexes/479026d892c9306a8907938e1e49f73e1514eabe/.bootstrap/pex/pex.py", line 707, in execute_entry
    return self.execute_module(entry_point, alter_sys)
  File "/home/********/.pex/unzipped_pexes/479026d892c9306a8907938e1e49f73e1514eabe/.bootstrap/pex/pex.py", line 719, in execute_module
    runpy.run_module(module_name, run_name="__main__", alter_sys=alter_sys)
  File "/usr/lib/python3.8/runpy.py", line 207, in run_module
    return _run_module_code(code, init_globals, run_name, mod_spec)
  File "/usr/lib/python3.8/runpy.py", line 97, in _run_module_code
    _run_code(code, mod_globals, init_globals,
  File "/usr/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/home/********/.pex/installed_wheels/6a5463833452712ecf58a573b1b489c81d14aaed599e955c2af85cf793217a42/pex-2.1.84-py2.py3-none-any.whl/pex/tools/__main__.py", line 10, in <module>
    sys.exit(tools())
  File "/home/********/.pex/installed_wheels/6a5463833452712ecf58a573b1b489c81d14aaed599e955c2af85cf793217a42/pex-2.1.84-py2.py3-none-any.whl/pex/tools/main.py", line 91, in main
    result = catch(pex_command.run, pex)
  File "/home/********/.pex/installed_wheels/6a5463833452712ecf58a573b1b489c81d14aaed599e955c2af85cf793217a42/pex-2.1.84-py2.py3-none-any.whl/pex/result.py", line 101, in catch
    return func(*args, **kwargs)
  File "/home/********/.pex/installed_wheels/6a5463833452712ecf58a573b1b489c81d14aaed599e955c2af85cf793217a42/pex-2.1.84-py2.py3-none-any.whl/pex/tools/commands/venv.py", line 192, in run
    populate_venv(
  File "/home/********/.pex/installed_wheels/6a5463833452712ecf58a573b1b489c81d14aaed599e955c2af85cf793217a42/pex-2.1.84-py2.py3-none-any.whl/pex/venv/pex.py", line 121, in populate_venv
    record_provenance(_populate_deps(venv, pex, venv_python, symlink))
  File "/home/********/.pex/installed_wheels/6a5463833452712ecf58a573b1b489c81d14aaed599e955c2af85cf793217a42/pex-2.1.84-py2.py3-none-any.whl/pex/venv/pex.py", line 117, in record_provenance
    for src, dst in src_to_dst:
  File "/home/********/.pex/installed_wheels/6a5463833452712ecf58a573b1b489c81d14aaed599e955c2af85cf793217a42/pex-2.1.84-py2.py3-none-any.whl/pex/venv/pex.py", line 205, in _populate_deps
    for dist in pex.resolve():
  File "/home/********/.pex/installed_wheels/6a5463833452712ecf58a573b1b489c81d14aaed599e955c2af85cf793217a42/pex-2.1.84-py2.py3-none-any.whl/pex/pex.py", line 121, in resolve
    for dist in env.resolve():
  File "/home/********/.pex/installed_wheels/6a5463833452712ecf58a573b1b489c81d14aaed599e955c2af85cf793217a42/pex-2.1.84-py2.py3-none-any.whl/pex/environment.py", line 492, in resolve
    for fingerprinted_distribution in self.resolve_dists(all_reqs)
  File "/home/********/.pex/installed_wheels/6a5463833452712ecf58a573b1b489c81d14aaed599e955c2af85cf793217a42/pex-2.1.84-py2.py3-none-any.whl/pex/environment.py", line 579, in resolve_dists
    raise ResolveError(
pex.environment.ResolveError: Failed to resolve requirements from PEX environment @ /home/********/.pex/unzipped_pexes/c63e230443e6590864c162f6f047efeaa207d04d.
Needed cp38-cp38-manylinux_2_31_x86_64 compatible dependencies for:
 1: cffi>=1.12
    Required by:
      FingerprintedDistribution(distribution=cryptography 36.0.2 (/home/********/.pex/installed_wheels/2ac879e0574023bfdc8af856eec5a1a3ff6524f800bc38b11a8cc10a2e6267c8/cryptography-36.0.2-cp36-abi3-manylinux_2_24_x86_64.whl), fingerprint='2ac879e0574023bfdc8af856eec5a1a3ff6524f800bc38b11a8cc10a2e6267c8')
    But this pex had no 'cffi' distributions.
17:28:51.47 [ERROR] 1 Exception encountered:

  ExportError: Failed to write virtualenv for the resolve 'python-default' (using CPython==3.10.*) to dist/export/python/virtualenvs/python-default

The first one is

ERROR: Could not find a version that satisfies the requirement pywin32==227; sys_platform == "win32" (from docker)

So somehow either docker or cryptography has a req on

pywin32

?

docker seems to have that, but only when using pex as lockfile generator, not when using poetry

Looks like its `docker`: https://github.com/docker/docker-py/blob/a48a5a9647761406d66e8271f19fab7fa0c5f582/setup.py#L19

But that shouldn't apply on Ubuntu, right?

@enough-analyst-54434 would likely know better the semantics of that

pex lockfile generation is “universal” so it will try to resolve dependencies for all possible platforms, not just the platform you’re using to generate the lock

Pex doesn't force this, Pants does. The default Pex lockfile style is

--style strict

which just locks for the selected interpreters.

ohhhh, hmmm, can I somehow disable support for win32 then which is not required on our side? 🤔

Since neither Pex nor Pants supports Windows, this is definitely unfortunate. When they do support Windows, the only escape left would be a new feature that allows you to restrict target architecture like you can already restrict target Python version. In the meantime the only recourse is to adjust your pinned requirements if possible or else fork docker and adjust docker reqs and use a VCS requirement.

Hmmm, I guess I could also switch over to poetry for the lockfiles, but there I keep having issues with several dependencies as well as posted above 🤔

If I were in your position I'd fork the docker repo, adjust it pywin32 dep to be more modern, and use a VCS requirement on your fork. It looks like

pywin32 302

is the 1st release to support CPython 3.10: https://pypi.org/project/pywin32/302/#files I'd then file an issue against Pex to add support for restricting platforms in

--style universal

locks, and when that ships, undo your VCS requirement on docker.

Hey, thanks for the suggestion and help so far! I did it the way you suggested and created a fork of docker and use that now. Will try to create a ticket with pex when I have some more time

Filed https://github.com/pantsbuild/pex/issues/1821 for fixing this over in Pex; i.e.: allowing you to restrict your lock as

--style universal --target-system linux --target-system mac --interpreter-constraint "CPython>=3.10,<4"

.

Hey @enough-analyst-54434, sorry for the late response. If you cherry-pick the fix to 2.13.x, I'll try to upgrade to that version once it's released.

It is picked there. You can try 2.13.0rc0 if you want to give it a spin and report back: https://pypi.org/project/pantsbuild.pants/2.13.0rc0/