Hey I m having several issues with regards to 3rdparty depen

Question

Hey I m having several issues with regards to 3rdparty dependencies and how they behave with different lockfile generators and different goals Maybe someone has an idea what I could try

bitter-ability-32190 · Answer

Can you please provide more info What command s are you running Whats the output What is the expected result

abundant-hospital-56388 · Answer

In the attached folder I try to use the requirements cryptography==36 0 2 and docker==5 0 3 I m using Ubuntu 22 04 pants 2 11 0 and restrict the interpreter to CPython gt =3 10 lt 4 I m activating enable resolves When I activate pex as the lockfile generator I get the following error during generation of the python default lockfile ``` pants generate lockfiles resolve=python default 17 27 27 51 INFO Initializing scheduler 17 27 27 61 INFO Scheduler initialized 17 27 27 63 WARN Please either set `enabled = true` in the anonymous telemetry section of pants toml to enable sending anonymous stats to the Pants project to aid development or set `enabled = false` to disable it No telemetry sent for this run An explicit setting will get rid of this message See for details 17 27 30 86 INFO Completed Generate lockfile for python default 17 27 30 86 ERROR 1 Exception encountered ProcessExecutionFailure Process Generate lockfile for python default failed with exit code 1 stdout stderr ERROR Could not find a version that satisfies the requirement pywin32==227 sys platform == win32 from docker ERROR No matching distribution found for pywin32==227 sys platform == win32 pid 147208 gt home cache pants named caches pex root venvs 4a17ee4a3e5aca6b7a9e86fbf3e8687b9afe3820 5985ed09b49a653d6596b0e14d134c5456cf1a9f pex disable pip version check no python version warning exists action a isolated q cache dir home cache pants named caches pex root log tmp pex pip logmx51wm31 pip log download dest tmp tmpc4n1o1oe usr bin python3 10 cryptography==36 0 2 docker==5 0 3 index url retries 5 timeout 15 exited with 1 and STDERR None Use ` no process cleanup` to preserve process chroots for inspection ``` When I switch over to `poetry` however cryptography is causing problems for me when calling ` pants export ` after the lockfile generation ``` pants export Traceback most recent call last File home pex unzipped pexes 479026d892c9306a8907938e1e49f73e1514eabe bootstrap pex pex py line 504 in execute exit value = self wrap coverage self wrap profiling self execute File home pex unzipped pexes 479026d892c9306a8907938e1e49f73e1514eabe bootstrap pex pex py line 409 in wrap coverage return runner args File home pex unzipped pexes 479026d892c9306a8907938e1e49f73e1514eabe bootstrap pex pex py line 440 in wrap profiling return runner args File home pex unzipped pexes 479026d892c9306a8907938e1e49f73e1514eabe bootstrap pex pex py line 556 in execute return self execute entry self pex info overrides entry point File home pex unzipped pexes 479026d892c9306a8907938e1e49f73e1514eabe bootstrap pex pex py line 707 in execute entry return self execute module entry point alter sys File home pex unzipped pexes 479026d892c9306a8907938e1e49f73e1514eabe bootstrap pex pex py line 719 in execute module runpy run module module name run name= main alter sys=alter sys File usr lib python3 8 runpy py line 207 in run module return run module code code init globals run name mod spec File usr lib python3 8 runpy py line 97 in run module code run code code mod globals init globals File usr lib python3 8 runpy py line 87 in run code exec code run globals File home pex installed wheels 6a5463833452712ecf58a573b1b489c81d14aaed599e955c2af85cf793217a42 pex 2 1 84 py2 py3 none any whl pex tools main py line 10 in lt module gt sys exit tools File home pex installed wheels 6a5463833452712ecf58a573b1b489c81d14aaed599e955c2af85cf793217a42 pex 2 1 84 py2 py3 none any whl pex tools main py line 91 in main result = catch pex command run pex File home pex installed wheels 6a5463833452712ecf58a573b1b489c81d14aaed599e955c2af85cf793217a42 pex 2 1 84 py2 py3 none any whl pex result py line 101 in catch return func args kwargs File home pex installed wheels 6a5463833452712ecf58a573b1b489c81d14aaed599e955c2af85cf793217a42 pex 2 1 84 py2 py3 none any whl pex tools commands venv py line 192 in run populate venv File home pex installed wheels 6a5463833452712ecf58a573b1b489c81d14aaed599e955c2af85cf793217a42 pex 2 1 84 py2 py3 none any whl pex venv pex py line 121 in populate venv record provenance populate deps venv pex venv python symlink File home pex installed wheels 6a5463833452712ecf58a573b1b489c81d14aaed599e955c2af85cf793217a42 pex 2 1 84 py2 py3 none any whl pex venv pex py line 117 in record provenance for src dst in src to dst File home pex installed wheels 6a5463833452712ecf58a573b1b489c81d14aaed599e955c2af85cf793217a42 pex 2 1 84 py2 py3 none any whl pex venv pex py line 205 in populate deps for dist in pex resolve File home pex installed wheels 6a5463833452712ecf58a573b1b489c81d14aaed599e955c2af85cf793217a42 pex 2 1 84 py2 py3 none any whl pex pex py line 121 in resolve for dist in env resolve File home pex installed wheels 6a5463833452712ecf58a573b1b489c81d14aaed599e955c2af85cf793217a42 pex 2 1 84 py2 py3 none any whl pex environment py line 492 in resolve for fingerprinted distribution in self resolve dists all reqs File home pex installed wheels 6a5463833452712ecf58a573b1b489c81d14aaed599e955c2af85cf793217a42 pex 2 1 84 py2 py3 none any whl pex environment py line 579 in resolve dists raise ResolveError pex environment ResolveError Failed to resolve requirements from PEX environment home pex unzipped pexes c63e230443e6590864c162f6f047efeaa207d04d Needed cp38 cp38 manylinux 2 31 x86 64 compatible dependencies for 1 cffi gt =1 12 Required by FingerprintedDistribution distribution=cryptography 36 0 2 home pex installed wheels 2ac879e0574023bfdc8af856eec5a1a3ff6524f800bc38b11a8cc10a2e6267c8 cryptography 36 0 2 cp36 abi3 manylinux 2 24 x86 64 whl fingerprint= 2ac879e0574023bfdc8af856eec5a1a3ff6524f800bc38b11a8cc10a2e6267c8 But this pex had no cffi distributions 17 28 51 47 ERROR 1 Exception encountered ExportError Failed to write virtualenv for the resolve python default using CPython==3 10 to dist export python virtualenvs python default```

abundant-hospital-56388 · Answer

My expectation is that both commands would work for both lockfile generators Poetry works when I exclude cryptography at least

bitter-ability-32190 · Answer

The first one is ```ERROR Could not find a version that satisfies the requirement pywin32==227 sys platform == win32 from docker ``` So somehow either docker or cryptography has a req on `pywin32`

abundant-hospital-56388 · Answer

docker seems to have that but only when using pex as lockfile generator not when using poetry

bitter-ability-32190 · Answer

Looks like its `docker` <https github com docker docker py blob a48a5a9647761406d66e8271f19fab7fa0c5f582 setup py L19>

abundant-hospital-56388 · Answer

But that shouldn t apply on Ubuntu right

bitter-ability-32190 · Answer

< enough analyst 54434> would likely know better the semantics of that

sparse-lifeguard-95737 · Answer

pex lockfile generation is universal so it will try to resolve dependencies for all possible platforms not just the platform you re using to generate the lock

enough-analyst-54434 · Answer

Pex doesn t force this Pants does The default Pex lockfile style is ` style strict` which just locks for the selected interpreters

enough-analyst-54434 · Answer

The specific issue with `pywin32 227` is it only supports up to CPython 3 9 and you require 3 10 <https pypi org project pywin32 227 files|https pypi org project pywin32 227 files>

enough-analyst-54434 · Answer

There is no sdist

abundant-hospital-56388 · Answer

ohhhh hmmm can I somehow disable support for win32 then which is not required on our side thinking face

enough-analyst-54434 · Answer

Since neither Pex nor Pants supports Windows this is definitely unfortunate When they do support Windows the only escape left would be a new feature that allows you to restrict target architecture like you can already restrict target Python version In the meantime the only recourse is to adjust your pinned requirements if possible or else fork docker and adjust docker reqs and use a VCS requirement

abundant-hospital-56388 · Answer

Hmmm I guess I could also switch over to poetry for the lockfiles but there I keep having issues with several dependencies as well as posted above thinking face

enough-analyst-54434 · Answer

If I were in your position I d fork the docker repo adjust it pywin32 dep to be more modern and use a VCS requirement on your fork It looks like `pywin32 302` is the 1st release to support CPython 3 10 <https pypi org project pywin32 302 files|https pypi org project pywin32 302 files> I d then file an issue against Pex to add support for restricting platforms in ` style universal` locks and when that ships undo your VCS requirement on docker

abundant-hospital-56388 · Answer

Hey thanks for the suggestion and help so far I did it the way you suggested and created a fork of docker and use that now Will try to create a ticket with pex when I have some more time

enough-analyst-54434 · Answer

Filed <https github com pantsbuild pex issues 1821> for fixing this over in Pex i e allowing you to restrict your lock as ` style universal target system linux target system mac interpreter constraint CPython gt =3 10 lt 4 `

enough-analyst-54434 · Answer

Ok this default restriction to ` target system linux target system mac` is up for review here and should be in this weeks weekly dev release of 2 14 0 dev2 < abundant hospital 56388> you noted you were using Pants 2 11 0 but it seems like you have a workaround I was just planning on cherry picking the fix to the 2 13 x release candidate Let me know how that works for you

abundant-hospital-56388 · Answer

Hey < enough analyst 54434> sorry for the late response If you cherry pick the fix to 2 13 x I ll try to upgrade to that version once it s released

enough-analyst-54434 · Answer

It is picked there You can try 2 13 0rc0 if you want to give it a spin and report back <https pypi org project pantsbuild pants 2 13 0rc0 >