thinking face After upgrading to 2 9 I m seeing some failure

Question

thinking face After upgrading to 2 9 I m seeing some failures when running Pytest tests ```ProcessExecutionFailure Process Building pytest runner pex failed with exit code 1 ``` Followed by a bunch of these ```pex environment ResolveError Failed to resolve requirements from PEX environment home maier cache pants named caches pex root unzipped pexes f00d5546a767d36a195e863e80031df5797b7eb8 Needed manylinux 2 28 x86 64 cp 37 cp37m compatible dependencies for 1 boto3 Required by FingerprintedDistribution distribution=grapl common 0 0 6 home maier cache pants named caches pex root installed wheels 7f5c555e91114b099cc3946517adba787c2ede3e grapl common 0 0 6 py3 none any whl fingerprint= a76df1c3f9ecd1ae5ae2352850c933b30140ba93 But this pex had no boto3 distributions 2 botocore Required by FingerprintedDistribution distribution=grapl common 0 0 6 home maier cache pants named caches pex root installed wheels 7f5c555e91114b099cc3946517adba787c2ede3e grapl common 0 0 6 py3 none any whl fingerprint= a76df1c3f9ecd1ae5ae2352850c933b30140ba93 But this pex had no botocore distributions 3 mypy boto3 cloudwatch Required by FingerprintedDistribution distribution=grapl common 0 0 6 home maier cache pants named caches pex root installed wheels 7f5c555e91114b099cc3946517adba787c2ede3e grapl common 0 0 6 py3 none any whl fingerprint= a76df1c3f9ecd1ae5ae2352850c933b30140ba93 But this pex had no mypy boto3 cloudwatch distributions 4 mypy boto3 dynamodb Required by FingerprintedDistribution distribution=grapl common 0 0 6 home maier cache pants named caches pex root installed wheels 7f5c555e91114b099cc3946517adba787c2ede3e grapl common 0 0 6 py3 none any whl fingerprint= a76df1c3f9ecd1ae5ae2352850c933b30140ba93 But this pex had no mypy boto3 dynamodb distributions ``` plus 8 similar missing dependencies Any pointers on where to look I ve regenerated my pytest lockfile Thanks in advance

hundreds-father-404 · Answer

Huh What was the last Pants version that worked What s interesting is that `pytest runner pex` is actually the fusion of a few PEXes including the `pytest pex` your tool lockfile and the `requirements pex` your code It uses the ` pex path` feature which allows you to merge PEXes

brash-baker-91190 · Answer

I upgraded from 2 8 to 2 9 today

brash-baker-91190 · Answer

And 2 8 worked without a hitch

brash-baker-91190 · Answer

stepping away for a bit but will be back soon

brash-baker-91190 · Answer

Some more information this is only happening for a small subset of my tests most run fine everything that s missing is required by a Python library `grapl common` which exists within this monorepo that is a dependency of the code under test The tests for `grapl common` run just fine with the exception of `boto3` and `botocore` numbers 1 and 2 in the output above all the missing dependencies are `mypy boto XXX` type libraries and `typing extensions` Not sure if this is useful but this is what stands out to me at this point

brash-baker-91190 · Answer

Tracing backwards it appears that something in version 2 9 0 dev3 triggers the failure

brash-baker-91190 · Answer

Looking at the changelog possible candidates might be Upgrade to Pex 2 1 56 Fix dup 3rdparty dists introduced by local dists those jumped out at me as possibly related in a sea of Java Go updates

hundreds-father-404 · Answer

Those do both look related thanks for bisecting this What happens btw if you temporarily remove the ~ cache pants named caches so it s unused preferred if you can preserve the folder for now in case we need it to debug

brash-baker-91190 · Answer

Moving the `named caches` directory aside creatively renamed it to `named caches orig` and running my test yields the same failure

hundreds-father-404 · Answer

Hm and then how about deleting the just created `named caches` and using ` no local cache no pantsd` Want to make sure this isn t from a corrupted Pex cache or anything which has happened in the past iirc

brash-baker-91190 · Answer

Still failed in the same way bummer

brash-baker-91190 · Answer

< hundreds father 404> Just curious if you might have any other ideas on this issue

hundreds-father-404 · Answer

Hm I ve been trying to think of ideas What were the changes to the Pytest lockfile and any other lockfiles

brash-baker-91190 · Answer

one sec let me go see

brash-baker-91190 · Answer

pytest diff

brash-baker-91190 · Answer

black diff mypy diff isort diff

hundreds-father-404 · Answer

Hm thanks None of those mention the requirements giving you issues I m wondering if this is an issue with `manylinux 2 28 x86 64 cp 37 cp37m` by chance < polite garden 50641> found that upgrading to Pants 2 10 is resulting in this error message for us ```Failed to resolve requirements from PEX environment home toolchain cache pants named caches pex root unzipped pexes 11c095ad69b7a3c1a757dbfcd20347104c1a6840 Needed manylinux 2 28 x86 64 cp 38 cp38 compatible dependencies for 1 py gt =1 8 2 Required by FingerprintedDistribution distribution=pytest 6 2 5 home toolchain cache pants named caches pex root installed wheels d8dbe72f24eaeea8b51e0f353336f9bd4915dc4a pytest 6 2 5 py3 none any whl fingerprint= c81a0f09ee54ea9496a98d7b1b12d41286248226 But this pex had no py distributions ``` To confirm do you get this issue on macOS if you have access

hundreds-father-404 · Answer

gt I m wondering if this is an issue with manylinux 2 28 x86 64 cp 37 cp37m by chance Probably red herring Asher also has the issue on a mac

polite-garden-50641 · Answer

keep in mind we don t use the lock file feature on our 2 9 branch so our branch that upgrades to 2 10 dev also switches from constraints file to a lock file

brash-baker-91190 · Answer

I don t have easy access to a mac for this sorry

hundreds-father-404 · Answer

That s okay Is this one of your open source projects that I could try running I m wanting to try to reproduce with Pex directly but am not sure how to do that off the top of my head

brash-baker-91190 · Answer

Yup This ought to reproduce it that gnarly `pants` invocation hits the specific test files that trigger this ```git clone <https github com grapl security grapl> git checkout cm pants 2 9 pants tag= needs work test pytest args= m not integration test src python graplctl tests tests```

hundreds-father-404 · Answer

Thank you I m about to log off for the long weekend but will play with this on Tuesday Have a great weekend

brash-baker-91190 · Answer

Sounds good thanks for your help Have a good weekend yourself wave

brash-baker-91190 · Answer

< hundreds father 404> any more ideas about this by chance

hundreds-father-404 · Answer

Ah thanks for the ping I don t any ideas < enough analyst 54434>

enough-analyst-54434 · Answer

You guys already found the idea <https github com pantsbuild pants pull 13690> that s exactly it I comment on that PR how I hate such a long winded explanation but its required reading unfortunately The book there talks about hand rolled setup py pyproject toml though and here grapl is using the auto setup py I m looking at that angle presently

brash-baker-91190 · Answer

Cool thanks very much < enough analyst 54434> < hundreds father 404> bow

enough-analyst-54434 · Answer

< brash baker 91190> just for background is depending on the src python grapl common python distribution instead of the src python grapl common grapl common pythoon sources intentional This diff gets your test command above passing and eliminates the middle man of packaging up grapl common into an sdist wheel ```diff git a src python graplctl graplctl BUILD b src python graplctl graplctl BUILD index 1e8a0ce9 050a642b 100644 a src python graplctl graplctl BUILD +++ b src python graplctl graplctl BUILD 1 14 +1 7 python sources name= graplctl lib dependencies= src python grapl common grapl common pex binary entry point= cli py main TODO Once lands remove this work around See also dependencies= src python grapl common grapl common ```

enough-analyst-54434 · Answer

Although its surprising this case isn t working since you don t use a custom setup py pyproject toml it is a bit surprising you re going through the middle man here instead of depending directly on the sources since that provides better fine grained invalidation a benefit of Pants + monorepo you re missing out on

brash-baker-91190 · Answer

Hrmm yeah it looks like we could do that sweat smile Oh man it even says it in the comments face palm

brash-baker-91190 · Answer

I m putting out about 3 fires at the moment but once things settle I ll give this a try in our repo Thanks

enough-analyst-54434 · Answer

OK I m still looking into the issue sans workaround since maybe its conceivable someone might have a use case where they actually want to depend on the wheel generated from the sources instead of the sources I can t come up with one off the top of my head but the world is a big place

brash-baker-91190 · Answer

Testing against the artifacts you d ultimately end up shipping

enough-analyst-54434 · Answer

I guess That s effectively testing Pants works though in this case of auto generated setup py

brash-baker-91190 · Answer

Just an update here I can now run my unit tests but our integration tests are now complaining I m not 100 sure what s happening but we have some Python services that are basically a PEX in a container not managed by Pants Docker integration at the moment FWIW and they re not starting up in Nomad just seeing errors like this ` bin bash line 1 analyzer executor pex Permission denied` I m continuing to bisect things to figure out at what point in the 2 9 dev rc line our system starts to fall over Our test process is kind of long though so I m just picking at this between other tasks we re not blocked or anything Pants 2 8 continues to work like a champ for us in this repository Just wanted to keep y all in the loop I ll report back if when I find anything

enough-analyst-54434 · Answer

gt they re not starting up in Nomad just seeing errors like this ` bin bash line 1 analyzer executor pex Permission denied` That looks like it can only really be that the 755 bits added to `analyzer executor pex` by Pex when it creates the PEX have been wiped out That shouldn t be related to any Pex or Pants upgrades Are you sure there are no other variable in play for that error re perms cp etc

brash-baker-91190 · Answer

It s possible been a while since I ve been in this bit of code and there s been a lot of change in this area recently We re probably doing something non standard somewhere sweat smile

brash-baker-91190 · Answer

I ll do some digging and report back

brash-baker-91190 · Answer

I finally got a chance to get back to this it looks like in Pants 2 8 PEX files were created with ` rwxr xr x` permissions but in 2 9 they get ` rwxr r ` in our container our PEX files are owned by root but run by a separate user Easy enough fix in our containers Is this an expected change in behavior though

enough-analyst-54434 · Answer

Pex itself still generates PEX files 755 This smells like a Pants behavior change And it is ```$ rm rf dist amp amp pants package src python pants bin 07 52 22 51 INFO Completed Installing 3rdparty python lockfiles user reqs txt 07 52 28 06 INFO Completed Extracting 15 requirements to build src python pants bin pants pex from lockfile pex PyYAML lt 7 0 gt =6 0 ansicolors==1 1 8 fasteners==0 16 3 humbug==0 2 7 ijson==3 1 4 packaging==21 3 pex==2 1 65 168 characters truncated 07 52 28 08 INFO Wrote dist src python pants bin pants pex $ ls l dist src python pants bin pants pex rwxr r 1 jsirois jsirois 53595441 Jan 27 07 52 dist src python pants bin pants pex``` That seems worth a bug < brash baker 91190>

brash-baker-91190 · Answer

< enough analyst 54434> OK I ll file an issue

enough-analyst-54434 · Answer

Thanks Chris

brash-baker-91190 · Answer

I ll also see if I can narrow it down to a specific dev or rc release of 2 9

enough-analyst-54434 · Answer

Awesome Thanks for digging

brash-baker-91190 · Answer

Yeah sure thing thumbsup

brash-baker-91190 · Answer

< enough analyst 54434>

brash-baker-91190 · Answer

Seems like Pants 2 9 0 dev4 is where this entered but I don t see what change in that release might be responsible