I have a pyproject toml that depends on some local wheels in

Question

I have a pyproject toml that depends on some local wheels internal 3rdparty deps Whenever I generate the lockfile pex I see that it uses the paths on my local computer this means that I can t really commit that lockfile to git and thus have to manually regenerate on CI Is there a way to get around this Would the answer be to use a different resolve for local wheels and only generate that lockfile every time

enough-analyst-54434 · Answer

Are you in a position to try 2 14 0a1 Although Pex has supported ` path mapping` for just this case for a long time now Pants just picked up support in

enough-analyst-54434 · Answer

Docs are here <https www pantsbuild org v2 14 docs python third party dependencies working around absolute paths>

cold-sugar-54376 · Answer

oh nice I m a little hestiant to try and alpha build but am fine with waiting till 2 14 comes out

future-oxygen-10553 · Answer

Could this be done with config file templating using e g ` buildroot ` pre 2 14 For example ``` python repos Use an absolute path to a directory containing ` whl` and or sdist files find links = file buildroot dist ```

future-oxygen-10553 · Answer

Or would that result in the same problem of writing the templated absolute path into the lockfile

enough-analyst-54434 · Answer

Yeah the ` buildroot s`is expanded early by Pants before it s handed as a real absolute path to Pex Pex doesn t try to guess and requires you give it a ` path mapping`to indicate a path that should be symbolized

cold-sugar-54376 · Answer

< enough analyst 54434> is there a full example of how this is meant to work somewhere

cold-sugar-54376 · Answer

i m trying this with pants 2 14 ```find links = file buildroot s prebuilt wheels path mappings = WHEELS DIR| buildroot s prebuilt wheels ```

enough-analyst-54434 · Answer

I m not sure but I don t think so If that s not working a quick thing to try is to get rid of the `file ` prefix in `find links`

cold-sugar-54376 · Answer

nvm I figured it out I thought you still had to use direct references to the file when defining dependencies

cold-sugar-54376 · Answer

realized now that you just need to define the dep as normal and it worked thanks

enough-analyst-54434 · Answer

Excellent

cold-sugar-54376 · Answer

hmm i m seeing this in my lockfile though ``` path mappings WHEELS DIR null ```

enough-analyst-54434 · Answer

That sounds like a bug Can you re run with ` pants generate lockfiles ldebug` and look for the command line including the word `universal` That will tell us if Pants hand s off badly to Pex or if Pex fumbles

cold-sugar-54376 · Answer

sure fwiw everything still works shrug

cold-sugar-54376 · Answer

```14 11 59 35 DEBUG spawned local process as Some 26736 for Process argv Users ashu pyenv versions 3 10 3 bin python pex lock create tmpdir tmp python path Users ashu pyenv versions 3 10 3 bin Users ashu pyenv versions 3 8 12 bin Users ashu pyenv versions 3 9 10 bin Users ashu pyenv versions 3 9 11 bin Users ashu pyenv versions 3 9 12 bin opt python3 output=lock json no emit warnings style=universal resolver version pip 2020 resolver target system linux target system mac indent=2 no pypi index=<https pypi org simple > find links=file Users ashu Documents backend prebuilt wheels manylinux manylinux2014 path mapping=WHEELS DIR| Users ashu Documents backend prebuilt wheels interpreter constraint CPython==3 9 10 pytest==7 0 1 pytest cov gt =2 12 =2 12 1 lt 3 1 pytest xdist gt =2 5 lt 3 fakeredis lupa freezegun pytest asyncio pytest django pytest env pytest freezegun pytest mock env CPPFLAGS I opt homebrew opt openssl include I opt homebrew opt openblas include GRPC PYTHON BUILD SYSTEM OPENSSL 1 LANG en US UTF 8 LDFLAGS L opt homebrew opt openssl lib L opt homebrew opt openblas lib PATH Users ashu pyenv shims Users ashu yarn bin Users ashu config yarn global node modules bin Users ashu nvm versions node v14 17 6 bin Users ashu go bin opt homebrew Caskroom google cloud sdk latest google cloud sdk bin opt homebrew bin opt homebrew sbin usr local bin usr bin bin usr sbin sbin usr local go bin PEX IGNORE RCFILES true PEX PYTHON PATH Users ashu pyenv versions 3 10 3 bin Users ashu pyenv versions 3 8 12 bin Users ashu pyenv versions 3 9 10 bin Users ashu pyenv versions 3 9 11 bin Users ashu pyenv versions 3 9 12 bin opt python3 PEX ROOT cache pex root PEX SCRIPT pex3 working directory None input digests InputDigests complete DirectoryDigest digest Digest hash Fingerprint lt 17cfe36e90b7216296f0651d9c72ef150c00d24bb1b99c8fe07134012366ce0b gt size bytes 158 tree Some nailgun DirectoryDigest digest Digest hash Fingerprint lt e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 gt size bytes 0 tree Some input files DirectoryDigest digest Digest hash Fingerprint lt 17cfe36e90b7216296f0651d9c72ef150c00d24bb1b99c8fe07134012366ce0b gt size bytes 158 tree Some immutable inputs use nailgun output files RelativePath lock json output directories timeout None execution slot variable None concurrency available 0 description Generate lockfile for pytest level Info append only caches CacheName pex root RelativePath cache pex root jdk home None platform constraint None cache scope PerSession remote cache speculation delay 0ns ```

cold-sugar-54376 · Answer

< enough analyst 54434> I m also running into errors where the hashes of the wheels being built on different machines are different expectedly and pants is complaining is there a way to ignore the hashes

enough-analyst-54434 · Answer

There is not

enough-analyst-54434 · Answer

Ok ` path mapping=WHEELS DIR| Users ashu Documents backend prebuilt wheels` looks like Pants is doing the right thing when creating the lock You say though that the lock file json has a path mappings with null valued entries

cold-sugar-54376 · Answer

correct

cold-sugar-54376 · Answer

so i d have to upload the wheels as part of my git repo

enough-analyst-54434 · Answer

I have not paged back in full context but if you want to stand a chance at a consistent resolve lock file you must have a stable set of wheels no matter where they come from

cold-sugar-54376 · Answer

got it I guess since our wheels are platform independent its fine I m not entirely sure how it would work for a platfoorm dependent wheel though

cold-sugar-54376 · Answer

it looks like `pip wheel` is non deterministic which is annoying

enough-analyst-54434 · Answer

Have you tried <https reproducible builds org docs source date epoch > I think I did this before and found it doesn t help with `pip wheel` but worth a quick try

enough-analyst-54434 · Answer

Ok Sorry for the delay there This is my own code and I forget it

enough-analyst-54434 · Answer

The null is correct So the key is the name of the path mapping that must be defined to use the lock the null value is an optional descrption of the path mapping that Pants doesn t currently fill in

enough-analyst-54434 · Answer

The idea was that if you had 3 path mappings you could add text to say what each was to help someone set up the proper mappings when they later went to use the lock on some other machine with perhaps wildly different paths

enough-analyst-54434 · Answer

So ``` path mappings WHEELS DIR XYZ Corp standard wheels ML WHEELS DIR XYZ Corp GPU specific wheels ```