https://pantsbuild.org/ logo
Join Slack
Powered by
hello! after a little break to fight fires I'm bac...
# general
a
hello! after a little break to fight fires I'm back to switching our tooling over to pants šŸ™‚ Something I've noticed is that running individual test files appears to take roughly twice as long with pants as opposed to just running them with straight pytest: (I ran this twice in a row so all the build steps would be cached and it's only spending time running the actual tests)
Copy code
(remit) [nate@ragin-cajun remit-srv]$ time ./pants test --force tests
Scrubbed PYTHONPATH=/home/nate/wave/remit-srv/src: from the environment.
09:56:24.43 [INFO] Completed: test - tests/test_terrapay.py succeeded.

āœ“ tests/test_terrapay.py succeeded.

real	2m23.628s
user	0m0.429s
sys	0m0.042s
compared with:
Copy code
(remit) [nate@ragin-cajun remit-srv]$ time pytest tests/test_terrapay.py
============================================== test session starts ==============================================
platform linux -- Python 3.8.5, pytest-6.0.1, py-1.9.0, pluggy-0.13.1
rootdir: /home/nate/wave/remit-srv
plugins: celery-4.4.7, requests-mock-1.8.0, Faker-4.1.1, cov-2.10.1
collected 596 items

tests/test_terrapay.py .................................................................................. [ 13%]
......................................................................................................... [ 31%]
......................................................................................................... [ 48%]
......................................................................................................... [ 66%]
......................................................................................................... [ 84%]
..............................................................................................            [100%]

======================================== 596 passed in 67.94s (0:01:07) =========================================

real	1m13.382s
user	0m59.747s
sys	0m2.068s
(it does not appear to be based on something specific in that test file either):
Copy code
(remit) [nate@ragin-cajun remit-srv]$ time ./pants test --force tests
Scrubbed PYTHONPATH=/home/nate/wave/remit-srv/src: from the environment.
10:06:27.88 [INFO] Completed: test - tests/test_aml_actions.py succeeded.

āœ“ tests/test_aml_actions.py succeeded.

real	0m13.151s
user	0m0.446s
sys	0m0.041s
(remit) [nate@ragin-cajun remit-srv]$ time pytest tests/test_aml_actions.py
============================================== test session starts ==============================================
platform linux -- Python 3.8.5, pytest-6.0.1, py-1.9.0, pluggy-0.13.1
rootdir: /home/nate/wave/remit-srv
plugins: celery-4.4.7, requests-mock-1.8.0, Faker-4.1.1, cov-2.10.1
collected 37 items

tests/test_aml_actions.py .....................................                                           [100%]

============================================== 37 passed in 1.50s ===============================================

real	0m6.756s
user	0m5.098s
sys	0m1.233s
h
Hi! Try pants 2.3.0rc1 instead of 2.2. We made a big performance gain to remove nearly all overhead from Pex
šŸ‘€ 1
a
hmmm
Copy code
ProcessExecutionFailure: Process 'Building pytest_runner.pex' failed with exit code 1.
stdout:

stderr:
Traceback (most recent call last):
  File "/home/nate/.cache/pants/named_caches/pex_root/unzipped_pexes/ba3fdfe2d776ac7517a031da663a86b1b88caef1/.bootstrap/pex/pex.py", line 477, in execute
    exit_value = self._wrap_coverage(self._wrap_profiling, self._execute)
  File "/home/nate/.cache/pants/named_caches/pex_root/unzipped_pexes/ba3fdfe2d776ac7517a031da663a86b1b88caef1/.bootstrap/pex/pex.py", line 394, in _wrap_coverage
    return runner(*args)
  File "/home/nate/.cache/pants/named_caches/pex_root/unzipped_pexes/ba3fdfe2d776ac7517a031da663a86b1b88caef1/.bootstrap/pex/pex.py", line 425, in _wrap_profiling
    return runner(*args)
  File "/home/nate/.cache/pants/named_caches/pex_root/unzipped_pexes/ba3fdfe2d776ac7517a031da663a86b1b88caef1/.bootstrap/pex/pex.py", line 533, in _execute
    return self.execute_entry(self._pex_info.entry_point)
  File "/home/nate/.cache/pants/named_caches/pex_root/unzipped_pexes/ba3fdfe2d776ac7517a031da663a86b1b88caef1/.bootstrap/pex/pex.py", line 650, in execute_entry
    return runner(entry_point)
  File "/home/nate/.cache/pants/named_caches/pex_root/unzipped_pexes/ba3fdfe2d776ac7517a031da663a86b1b88caef1/.bootstrap/pex/pex.py", line 668, in execute_pkg_resources
    return runner()
  File "/home/nate/.cache/pants/named_caches/pex_root/unzipped_pexes/ba3fdfe2d776ac7517a031da663a86b1b88caef1/.deps/pex-2.1.32-py2.py3-none-any.whl/pex/bin/pex.py", line 1097, in main
    execute_cached_args = seed_cache(options, pex)
  File "/home/nate/.cache/pants/named_caches/pex_root/unzipped_pexes/ba3fdfe2d776ac7517a031da663a86b1b88caef1/.deps/pex-2.1.32-py2.py3-none-any.whl/pex/bin/pex.py", line 1139, in seed_cache
    venv_pex = ensure_venv(pex)
  File "/home/nate/.cache/pants/named_caches/pex_root/unzipped_pexes/ba3fdfe2d776ac7517a031da663a86b1b88caef1/.deps/pex-2.1.32-py2.py3-none-any.whl/pex/pex_bootstrapper.py", line 421, in ensure_venv
    shenbang = populate_venv_with_pex(
  File "/home/nate/.cache/pants/named_caches/pex_root/unzipped_pexes/ba3fdfe2d776ac7517a031da663a86b1b88caef1/.deps/pex-2.1.32-py2.py3-none-any.whl/pex/tools/commands/venv.py", line 114, in populate_venv_with_pex
    for dist in pex.activate():
  File "/home/nate/.cache/pants/named_caches/pex_root/unzipped_pexes/ba3fdfe2d776ac7517a031da663a86b1b88caef1/.deps/pex-2.1.32-py2.py3-none-any.whl/pex/pex.py", line 132, in activate
    self._activated_dists = self._activate()
  File "/home/nate/.cache/pants/named_caches/pex_root/unzipped_pexes/ba3fdfe2d776ac7517a031da663a86b1b88caef1/.deps/pex-2.1.32-py2.py3-none-any.whl/pex/pex.py", line 119, in _activate
    activated_dists.extend(env.activate())
  File "/home/nate/.cache/pants/named_caches/pex_root/unzipped_pexes/ba3fdfe2d776ac7517a031da663a86b1b88caef1/.deps/pex-2.1.32-py2.py3-none-any.whl/pex/environment.py", line 365, in activate
    self._activated_dists = self._activate()
  File "/home/nate/.cache/pants/named_caches/pex_root/unzipped_pexes/ba3fdfe2d776ac7517a031da663a86b1b88caef1/.deps/pex-2.1.32-py2.py3-none-any.whl/pex/environment.py", line 689, in _activate
    resolved = self.resolve(all_reqs)
  File "/home/nate/.cache/pants/named_caches/pex_root/unzipped_pexes/ba3fdfe2d776ac7517a031da663a86b1b88caef1/.deps/pex-2.1.32-py2.py3-none-any.whl/pex/environment.py", line 596, in resolve
    raise ResolveError(
pex.environment.ResolveError: Failed to resolve requirements from PEX environment @ /tmp/process-executionzVTk6f/requirements.pex.
Needed manylinux_2_33_x86_64-cp-38-cp38 compatible dependencies for:
 1: python>=2.6.0
    Required by:
      et-xmlfile 1.0.1
    But this pex had no 'python' distributions.
šŸ‘€ 1
h
Huh. @enough-analyst-54434 I'm a bit confused by python being the distribution, I haven't seen that. Do you know what might be going on?
a
Honestly I'm not sure whats going on here so I don't know if I can be much help - I'm running python 3.8.7 and heres my /proc/version file
Copy code
Linux version 5.11.2-arch1-1 (linux@archlinux) (gcc (GCC) 10.2.0, GNU ld (GNU Binutils) 2.36.1) #1 SMP PREEMPT Fri, 26 Feb 2021 18:26:41 +0000
let me know if there's anything I can do to help debug
šŸ‘ 1
@hundreds-father-404 package works
Copy code
[nate@ragin-cajun remit-srv]$ ./pants package src:remit-dist
10:36:41.11 [INFO] Completed: Building setuptools.pex with 2 requirements: setuptools>=50.3.0,<54.0, wheel>=0.35.1,<0.37
10:36:44.34 [INFO] Wrote dist/remit-srv-0.tar.gz
10:36:44.34 [INFO] Wrote dist/remit_srv-0-py3-none-any.whl
e
The et-xmlfile package is bogus. Details here: https://github.com/pantsbuild/pex/issues/1201 Do you need it or is it a transitive dep? I'm not sure the workaround I gave over in Pex is applicable with Pants.
šŸ™Œ 2
a
woof it looks like we are pulling that in from openpyxl šŸ˜ž the workaround doesn't seem to work but removing that package means the tests will run
āœ”ļø 1
h
It looks like the project is very actively maintained. I wonder if they would accept a PR to replace that problematic dep with something else? https://foss.heptapod.net/openpyxl/openpyxl/-/blob/branch/3.0/setup.py#L79 (Not sure if you've contributed to OSS before - but if you're interested in proposing that change to them, I recommend starting with an issue before submitting a PR)
āœ… 1
e
I did not have a prompt interlocutor in that Pex issue so closed it answered. If this package is really needed I can re-open that issue if you ping it with your needs and lack of workaround applicability. At the end of the day, Pex could keep a database of known bad dist metadata.
šŸ‘ 1
a
I'll see what happens with openpyxl first
b
Is this something to mention in Pants docs? Someone test driving Pants may appreciate head's up. e.g. in rare circumstances an external package may cause Pants to run slowly, c'mon over to the slack for help identifying which one.
h
Is this something to mention in Pants docs?
We call attention to MyPy running slowly, which is particularly egregious due to not using MyPy's cache https://www.pantsbuild.org/docs/python-typecheck-goal#performance-may-be-slower-than-normal But generally,, I don't think the perf issues are egregious enough (on Pants 2.3+) to need to call attention to it, unless more users report it's still concerning enough we need to address it. And we're planning work on addressing #1 and #3 from https://pantsbuild.slack.com/archives/C046T6T9U/p1614962378139800?thread_ts=1614956429.132400&amp;cid=C046T6T9U
in rare circumstances an external package may cause Pants to run slowly
That external dep isn't causing Pants to run slowly, it's causing things to fail to build
a
I created a new package with a fixed setup.py: https://test.pypi.org/project/et-xmlfile-2021/ i hope I can get openpyxl to update their dependency
šŸ¤ž 2
e
OK - I just read a bunch of pip and setuptools code. They account for, but ignore the legacy 
Requires
metadata associated with et-xml's 
requires
setuptools arg. That seems completely broken, but I want to confirm this with an experiment. If true, I'll have Pex follow their lead and ignore it too, which would "solve" this.
šŸ‘ 1
Yup - they ignore it and install brokenly (don't install requires dependencies)! šŸ¤¦ OK - I'll make Pex do so too šŸ˜•
šŸ™Œ 1
Sorry for the trouble @helpful-lunch-92084 - I assumed doing the right thing in this code was right.
h
wrong nate but iā€™m following along šŸ˜‰
a
@enough-analyst-54434 thank you!
e
Thanks for actually sticking around. I would have been nice to nail this sooner, but the original reporter fled. Re-opened https://github.com/pantsbuild/pex/issues/1201 with all the details. I'll get a fix out in a Pex release and consume in Pants stable releases with cherry-picks.
Open in Slack
PreviousNext
Powered by