Hey, I'm having several issues with regards to 3rd...
# general
Hey, I'm having several issues with regards to 3rdparty dependencies and how they behave with different lockfile generators and different goals. Maybe someone has an idea what I could try.
Can you please provide more info? • What command(s) are you running? • Whats the output? • What is the expected result?
In the attached folder, I try to use the requirements cryptography==36.0.2 and docker==5.0.3. I'm using Ubuntu 22.04, pants 2.11.0 and restrict the interpreter to CPython>=3.10,<4. I'm activating enable_resolves. When I activate "pex" as the lockfile_generator, I get the following error during generation of the python-default lockfile:
Copy code
./pants generate-lockfiles --resolve=python-default
17:27:27.51 [INFO] Initializing scheduler...
17:27:27.61 [INFO] Scheduler initialized.
17:27:27.63 [WARN] Please either set `enabled = true` in the [anonymous-telemetry] section of pants.toml to enable sending anonymous stats to the Pants project to aid development, or set `enabled = false` to disable it. No telemetry sent for this run. An explicit setting will get rid of this message. See <https://www.pantsbuild.org/v2.11/docs/anonymous-telemetry> for details.
17:27:30.86 [INFO] Completed: Generate lockfile for python-default
17:27:30.86 [ERROR] 1 Exception encountered:

  ProcessExecutionFailure: Process 'Generate lockfile for python-default' failed with exit code 1.

ERROR: Could not find a version that satisfies the requirement pywin32==227; sys_platform == "win32" (from docker)
ERROR: No matching distribution found for pywin32==227; sys_platform == "win32"
pid 147208 -> /home/********/.cache/pants/named_caches/pex_root/venvs/4a17ee4a3e5aca6b7a9e86fbf3e8687b9afe3820/5985ed09b49a653d6596b0e14d134c5456cf1a9f/pex --disable-pip-version-check --no-python-version-warning --exists-action a --isolated -q --cache-dir /home/********/.cache/pants/named_caches/pex_root --log /tmp/pex-pip-logmx51wm31/pip.log download --dest /tmp/tmpc4n1o1oe/usr.bin.python3.10 cryptography==36.0.2 docker==5.0.3 --index-url <https://pypi.org/simple/> --retries 5 --timeout 15 exited with 1 and STDERR:

Use `--no-process-cleanup` to preserve process chroots for inspection.
When I switch over to
however, cryptography is causing problems for me when calling
./pants export ::
after the lockfile generation:
Copy code
./pants export ::                                  
Traceback (most recent call last):
  File "/home/********/.pex/unzipped_pexes/479026d892c9306a8907938e1e49f73e1514eabe/.bootstrap/pex/pex.py", line 504, in execute
    exit_value = self._wrap_coverage(self._wrap_profiling, self._execute)
  File "/home/********/.pex/unzipped_pexes/479026d892c9306a8907938e1e49f73e1514eabe/.bootstrap/pex/pex.py", line 409, in _wrap_coverage
    return runner(*args)
  File "/home/********/.pex/unzipped_pexes/479026d892c9306a8907938e1e49f73e1514eabe/.bootstrap/pex/pex.py", line 440, in _wrap_profiling
    return runner(*args)
  File "/home/********/.pex/unzipped_pexes/479026d892c9306a8907938e1e49f73e1514eabe/.bootstrap/pex/pex.py", line 556, in _execute
    return self.execute_entry(self._pex_info_overrides.entry_point)
  File "/home/********/.pex/unzipped_pexes/479026d892c9306a8907938e1e49f73e1514eabe/.bootstrap/pex/pex.py", line 707, in execute_entry
    return self.execute_module(entry_point, alter_sys)
  File "/home/********/.pex/unzipped_pexes/479026d892c9306a8907938e1e49f73e1514eabe/.bootstrap/pex/pex.py", line 719, in execute_module
    runpy.run_module(module_name, run_name="__main__", alter_sys=alter_sys)
  File "/usr/lib/python3.8/runpy.py", line 207, in run_module
    return _run_module_code(code, init_globals, run_name, mod_spec)
  File "/usr/lib/python3.8/runpy.py", line 97, in _run_module_code
    _run_code(code, mod_globals, init_globals,
  File "/usr/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/home/********/.pex/installed_wheels/6a5463833452712ecf58a573b1b489c81d14aaed599e955c2af85cf793217a42/pex-2.1.84-py2.py3-none-any.whl/pex/tools/__main__.py", line 10, in <module>
  File "/home/********/.pex/installed_wheels/6a5463833452712ecf58a573b1b489c81d14aaed599e955c2af85cf793217a42/pex-2.1.84-py2.py3-none-any.whl/pex/tools/main.py", line 91, in main
    result = catch(pex_command.run, pex)
  File "/home/********/.pex/installed_wheels/6a5463833452712ecf58a573b1b489c81d14aaed599e955c2af85cf793217a42/pex-2.1.84-py2.py3-none-any.whl/pex/result.py", line 101, in catch
    return func(*args, **kwargs)
  File "/home/********/.pex/installed_wheels/6a5463833452712ecf58a573b1b489c81d14aaed599e955c2af85cf793217a42/pex-2.1.84-py2.py3-none-any.whl/pex/tools/commands/venv.py", line 192, in run
  File "/home/********/.pex/installed_wheels/6a5463833452712ecf58a573b1b489c81d14aaed599e955c2af85cf793217a42/pex-2.1.84-py2.py3-none-any.whl/pex/venv/pex.py", line 121, in populate_venv
    record_provenance(_populate_deps(venv, pex, venv_python, symlink))
  File "/home/********/.pex/installed_wheels/6a5463833452712ecf58a573b1b489c81d14aaed599e955c2af85cf793217a42/pex-2.1.84-py2.py3-none-any.whl/pex/venv/pex.py", line 117, in record_provenance
    for src, dst in src_to_dst:
  File "/home/********/.pex/installed_wheels/6a5463833452712ecf58a573b1b489c81d14aaed599e955c2af85cf793217a42/pex-2.1.84-py2.py3-none-any.whl/pex/venv/pex.py", line 205, in _populate_deps
    for dist in pex.resolve():
  File "/home/********/.pex/installed_wheels/6a5463833452712ecf58a573b1b489c81d14aaed599e955c2af85cf793217a42/pex-2.1.84-py2.py3-none-any.whl/pex/pex.py", line 121, in resolve
    for dist in env.resolve():
  File "/home/********/.pex/installed_wheels/6a5463833452712ecf58a573b1b489c81d14aaed599e955c2af85cf793217a42/pex-2.1.84-py2.py3-none-any.whl/pex/environment.py", line 492, in resolve
    for fingerprinted_distribution in self.resolve_dists(all_reqs)
  File "/home/********/.pex/installed_wheels/6a5463833452712ecf58a573b1b489c81d14aaed599e955c2af85cf793217a42/pex-2.1.84-py2.py3-none-any.whl/pex/environment.py", line 579, in resolve_dists
    raise ResolveError(
pex.environment.ResolveError: Failed to resolve requirements from PEX environment @ /home/********/.pex/unzipped_pexes/c63e230443e6590864c162f6f047efeaa207d04d.
Needed cp38-cp38-manylinux_2_31_x86_64 compatible dependencies for:
 1: cffi>=1.12
    Required by:
      FingerprintedDistribution(distribution=cryptography 36.0.2 (/home/********/.pex/installed_wheels/2ac879e0574023bfdc8af856eec5a1a3ff6524f800bc38b11a8cc10a2e6267c8/cryptography-36.0.2-cp36-abi3-manylinux_2_24_x86_64.whl), fingerprint='2ac879e0574023bfdc8af856eec5a1a3ff6524f800bc38b11a8cc10a2e6267c8')
    But this pex had no 'cffi' distributions.
17:28:51.47 [ERROR] 1 Exception encountered:

  ExportError: Failed to write virtualenv for the resolve 'python-default' (using CPython==3.10.*) to dist/export/python/virtualenvs/python-default
My expectation is that both commands would work for both lockfile generators. Poetry works when I exclude cryptography at least.
The first one is
Copy code
ERROR: Could not find a version that satisfies the requirement pywin32==227; sys_platform == "win32" (from docker)
So somehow either docker or cryptography has a req on
docker seems to have that, but only when using pex as lockfile generator, not when using poetry
But that shouldn't apply on Ubuntu, right?
@enough-analyst-54434 would likely know better the semantics of that
pex lockfile generation is “universal” so it will try to resolve dependencies for all possible platforms, not just the platform you’re using to generate the lock
Pex doesn't force this, Pants does. The default Pex lockfile style is
--style strict
which just locks for the selected interpreters.
👍 1
The specific issue with
pywin32 227
is it only supports up to CPython 3.9 and you require 3.10: https://pypi.org/project/pywin32/227/#files
coke 1
(There is no sdist)
ohhhh, hmmm, can I somehow disable support for win32 then which is not required on our side? 🤔
Since neither Pex nor Pants supports Windows, this is definitely unfortunate. When they do support Windows, the only escape left would be a new feature that allows you to restrict target architecture like you can already restrict target Python version. In the meantime the only recourse is to adjust your pinned requirements if possible or else fork docker and adjust docker reqs and use a VCS requirement.
Hmmm, I guess I could also switch over to poetry for the lockfiles, but there I keep having issues with several dependencies as well as posted above 🤔
If I were in your position I'd fork the docker repo, adjust it pywin32 dep to be more modern, and use a VCS requirement on your fork. It looks like
pywin32 302
is the 1st release to support CPython 3.10: https://pypi.org/project/pywin32/302/#files I'd then file an issue against Pex to add support for restricting platforms in
--style universal
locks, and when that ships, undo your VCS requirement on docker.
Hey, thanks for the suggestion and help so far! I did it the way you suggested and created a fork of docker and use that now. Will try to create a ticket with pex when I have some more time
🤞 1
Filed https://github.com/pantsbuild/pex/issues/1821 for fixing this over in Pex; i.e.: allowing you to restrict your lock as
--style universal --target-system linux --target-system mac --interpreter-constraint "CPython>=3.10,<4"
🙏 1
Ok, this default restriction to
--target-system linux --target-system mac
is up for review here: https://github.com/pantsbuild/pants/pull/16110 and should be in this weeks weekly dev release of 2.14.0.dev2. @abundant-hospital-56388 you noted you were using Pants 2.11.0 but it seems like you have a workaround. I was just planning on cherry-picking the fix to the 2.13.x release candidate. Let me know how that works for you.
Hey @enough-analyst-54434, sorry for the late response. If you cherry-pick the fix to 2.13.x, I'll try to upgrade to that version once it's released.
It is picked there. You can try 2.13.0rc0 if you want to give it a spin and report back: https://pypi.org/project/pantsbuild.pants/2.13.0rc0/