hello after a little break to fight fires I m back to switch

Question

hello after a little break to fight fires I m back to switching our tooling over to pants slightly smiling face Something I ve noticed is that running individual test files appears to take roughly twice as long with pants as opposed to just running them with straight pytest I ran this twice in a row so all the build steps would be cached and it s only spending time running the actual tests ``` remit nate ragin cajun remit srv $ time pants test force tests Scrubbed PYTHONPATH= home nate wave remit srv src from the environment 09 56 24 43 INFO Completed test tests test terrapay py succeeded tests test terrapay py succeeded real 2m23 628s user 0m0 429s sys 0m0 042s``` compared with ``` remit nate ragin cajun remit srv $ time pytest tests test terrapay py ============================================== test session starts ============================================== platform linux Python 3 8 5 pytest 6 0 1 py 1 9 0 pluggy 0 13 1 rootdir home nate wave remit srv plugins celery 4 4 7 requests mock 1 8 0 Faker 4 1 1 cov 2 10 1 collected 596 items tests test terrapay py 13 31 48 66 84 100 ======================================== 596 passed in 67 94s 0 01 07 ========================================= real 1m13 382s user 0m59 747s sys 0m2 068s```

average-australia-85137 · Answer

it does not appear to be based on something specific in that test file either ``` remit nate ragin cajun remit srv $ time pants test force tests Scrubbed PYTHONPATH= home nate wave remit srv src from the environment 10 06 27 88 INFO Completed test tests test aml actions py succeeded tests test aml actions py succeeded real 0m13 151s user 0m0 446s sys 0m0 041s remit nate ragin cajun remit srv $ time pytest tests test aml actions py ============================================== test session starts ============================================== platform linux Python 3 8 5 pytest 6 0 1 py 1 9 0 pluggy 0 13 1 rootdir home nate wave remit srv plugins celery 4 4 7 requests mock 1 8 0 Faker 4 1 1 cov 2 10 1 collected 37 items tests test aml actions py 100 ============================================== 37 passed in 1 50s =============================================== real 0m6 756s user 0m5 098s sys 0m1 233s```

hundreds-father-404 · Answer

Hi Try pants 2 3 0rc1 instead of 2 2 We made a big performance gain to remove nearly all overhead from Pex

average-australia-85137 · Answer

hmmm ``` ProcessExecutionFailure Process Building pytest runner pex failed with exit code 1 stdout stderr Traceback most recent call last File home nate cache pants named caches pex root unzipped pexes ba3fdfe2d776ac7517a031da663a86b1b88caef1 bootstrap pex pex py line 477 in execute exit value = self wrap coverage self wrap profiling self execute File home nate cache pants named caches pex root unzipped pexes ba3fdfe2d776ac7517a031da663a86b1b88caef1 bootstrap pex pex py line 394 in wrap coverage return runner args File home nate cache pants named caches pex root unzipped pexes ba3fdfe2d776ac7517a031da663a86b1b88caef1 bootstrap pex pex py line 425 in wrap profiling return runner args File home nate cache pants named caches pex root unzipped pexes ba3fdfe2d776ac7517a031da663a86b1b88caef1 bootstrap pex pex py line 533 in execute return self execute entry self pex info entry point File home nate cache pants named caches pex root unzipped pexes ba3fdfe2d776ac7517a031da663a86b1b88caef1 bootstrap pex pex py line 650 in execute entry return runner entry point File home nate cache pants named caches pex root unzipped pexes ba3fdfe2d776ac7517a031da663a86b1b88caef1 bootstrap pex pex py line 668 in execute pkg resources return runner File home nate cache pants named caches pex root unzipped pexes ba3fdfe2d776ac7517a031da663a86b1b88caef1 deps pex 2 1 32 py2 py3 none any whl pex bin pex py line 1097 in main execute cached args = seed cache options pex File home nate cache pants named caches pex root unzipped pexes ba3fdfe2d776ac7517a031da663a86b1b88caef1 deps pex 2 1 32 py2 py3 none any whl pex bin pex py line 1139 in seed cache venv pex = ensure venv pex File home nate cache pants named caches pex root unzipped pexes ba3fdfe2d776ac7517a031da663a86b1b88caef1 deps pex 2 1 32 py2 py3 none any whl pex pex bootstrapper py line 421 in ensure venv shenbang = populate venv with pex File home nate cache pants named caches pex root unzipped pexes ba3fdfe2d776ac7517a031da663a86b1b88caef1 deps pex 2 1 32 py2 py3 none any whl pex tools commands venv py line 114 in populate venv with pex for dist in pex activate File home nate cache pants named caches pex root unzipped pexes ba3fdfe2d776ac7517a031da663a86b1b88caef1 deps pex 2 1 32 py2 py3 none any whl pex pex py line 132 in activate self activated dists = self activate File home nate cache pants named caches pex root unzipped pexes ba3fdfe2d776ac7517a031da663a86b1b88caef1 deps pex 2 1 32 py2 py3 none any whl pex pex py line 119 in activate activated dists extend env activate File home nate cache pants named caches pex root unzipped pexes ba3fdfe2d776ac7517a031da663a86b1b88caef1 deps pex 2 1 32 py2 py3 none any whl pex environment py line 365 in activate self activated dists = self activate File home nate cache pants named caches pex root unzipped pexes ba3fdfe2d776ac7517a031da663a86b1b88caef1 deps pex 2 1 32 py2 py3 none any whl pex environment py line 689 in activate resolved = self resolve all reqs File home nate cache pants named caches pex root unzipped pexes ba3fdfe2d776ac7517a031da663a86b1b88caef1 deps pex 2 1 32 py2 py3 none any whl pex environment py line 596 in resolve raise ResolveError pex environment ResolveError Failed to resolve requirements from PEX environment tmp process executionzVTk6f requirements pex Needed manylinux 2 33 x86 64 cp 38 cp38 compatible dependencies for 1 python gt =2 6 0 Required by et xmlfile 1 0 1 But this pex had no python distributions ```

hundreds-father-404 · Answer

Huh < enough analyst 54434> I m a bit confused by python being the distribution I haven t seen that Do you know what might be going on

average-australia-85137 · Answer

Honestly I m not sure whats going on here so I don t know if I can be much help I m running python 3 8 7 and heres my proc version file ```Linux version 5 11 2 arch1 1 linux archlinux gcc GCC 10 2 0 GNU ld GNU Binutils 2 36 1 1 SMP PREEMPT Fri 26 Feb 2021 18 26 41 +0000```

average-australia-85137 · Answer

let me know if there s anything I can do to help debug

average-australia-85137 · Answer

< hundreds father 404> package works ``` nate ragin cajun remit srv $ pants package src remit dist 10 36 41 11 INFO Completed Building setuptools pex with 2 requirements setuptools gt =50 3 0 lt 54 0 wheel gt =0 35 1 lt 0 37 10 36 44 34 INFO Wrote dist remit srv 0 tar gz 10 36 44 34 INFO Wrote dist remit srv 0 py3 none any whl```

enough-analyst-54434 · Answer

The et xmlfile package is bogus Details here <https github com pantsbuild pex issues 1201|https github com pantsbuild pex issues 1201> Do you need it or is it a transitive dep I m not sure the workaround I gave over in Pex is applicable with Pants

average-australia-85137 · Answer

woof it looks like we are pulling that in from openpyxl disappointed the workaround doesn t seem to work but removing that package means the tests will run

hundreds-father-404 · Answer

It looks like the project is very actively maintained I wonder if they would accept a PR to replace that problematic dep with something else <https foss heptapod net openpyxl openpyxl blob branch 3 0 setup py L79> Not sure if you ve contributed to OSS before but if you re interested in proposing that change to them I recommend starting with an issue before submitting a PR

enough-analyst-54434 · Answer

I did not have a prompt interlocutor in that Pex issue so closed it answered If this package is really needed I can re open that issue if you ping it with your needs and lack of workaround applicability At the end of the day Pex could keep a database of known bad dist metadata

average-australia-85137 · Answer

I ll see what happens with openpyxl first

busy-vase-39202 · Answer

Is this something to mention in Pants docs Someone test driving Pants may appreciate head s up e g in rare circumstances an external package may cause Pants to run slowly c mon over to the slack for help identifying which one

hundreds-father-404 · Answer

gt Is this something to mention in Pants docs We call attention to MyPy running slowly which is particularly egregious due to not using MyPy s cache But generally I don t think the perf issues are egregious enough on Pants 2 3+ to need to call attention to it unless more users report it s still concerning enough we need to address it And we re planning work on addressing 1 and 3 from gt in rare circumstances an external package may cause Pants to run slowly That external dep isn t causing Pants to run slowly it s causing things to fail to build

average-australia-85137 · Answer

I created a new package with a fixed setup py <https test pypi org project et xmlfile 2021 > i hope I can get openpyxl to update their dependency

average-australia-85137 · Answer

enough-analyst-54434 · Answer

OK I just read a bunch of pip and setuptools code They account for but ignore the legacy `Requires` metadata associated with et xml s `requires` setuptools arg That seems completely broken but I want to confirm this with an experiment If true I ll have Pex follow their lead and ignore it too which would solve this

enough-analyst-54434 · Answer

Yup they ignore it and install brokenly don t install requires dependencies face palm OK I ll make Pex do so too confused

enough-analyst-54434 · Answer

Sorry for the trouble < helpful lunch 92084> I assumed doing the right thing in this code was right

helpful-lunch-92084 · Answer

wrong nate but i m following along wink

average-australia-85137 · Answer

< enough analyst 54434> thank you

enough-analyst-54434 · Answer

Thanks for actually sticking around I would have been nice to nail this sooner but the original reporter fled Re opened <https github com pantsbuild pex issues 1201> with all the details I ll get a fix out in a Pex release and consume in Pants stable releases with cherry picks