https://pantsbuild.org/ logo
#development
Title
# development
m

melodic-thailand-99227

10/31/2021, 12:31 PM
It it possible to make
python_tests
use the virtualenv instead of pex?
e

enough-analyst-54434

10/31/2021, 1:18 PM
It already is using a venv. All PEXes Pants builds internally use the Pex
--venv
mode. That mode creates a venv (using
python -mvenv ...
), populates it with the contents of the PEX and runs from there. Are you seeing unexpected behavior?
m

melodic-thailand-99227

10/31/2021, 2:34 PM
It takes about 25-50 seconds to run a single test, it seems to building a pex file
Copy code
298.88s Building requirements.pex with 94 requirements: ...
e

enough-analyst-54434

10/31/2021, 2:57 PM
Ok. There are two things to track down: 1. Is that 'Building requirements.pex ...' happening more often than expected? It should only happen when requirements are edited. 2. Is that time markedly different from a Pip resolve of the same requirements?
Actually, I neglected to ask - are you using a lock file? That has a big impact on resolve performance: https://www.pantsbuild.org/docs/python-third-party-dependencies#user-lockfile
m

melodic-thailand-99227

10/31/2021, 3:05 PM
I have tried to use a lock file, but I can’t seem to make it work. We have a big lock file (~410 packages) and pants doesn’t finish locking them even though they are fully locked (e.g. exported from poetry)
e

enough-analyst-54434

10/31/2021, 3:06 PM
Aha - ok, that needs to be solved then.
So, what do you mean by Pants doesn't finish locking them? Pants just hangs?
If there is an issue filed for this already I can check that out.
m

melodic-thailand-99227

10/31/2021, 3:10 PM
It usually hangs, but when it finally finished creating the
repository.pex
I’m getting
OSError: [Errno 36] File name too long:
on this call stack
Copy code
ProcessExecutionFailure: Process 'Extracting urllib3==1.26.7; python_version >= "2.7" and python_full_version < "3.0.0" or python_full_version >= "3.6.0" and python_version < "4" or python_full_version >= "3.5.3" and python_version < "4" or python_full_version >= "3.5.0" and python_version < "4" from repository.pex' failed with exit code 1.
stdout:

stderr:
Traceback (most recent call last):
  File "/home/ubuntu/.cache/pants/named_caches/pex_root/unzipped_pexes/592827b562c4aa09c1e9af6b3ed7b797eef1a4dc/.bootstrap/pex/pex.py", line 483, in execute
    exit_value = self._wrap_coverage(self._wrap_profiling, self._execute)
  File "/home/ubuntu/.cache/pants/named_caches/pex_root/unzipped_pexes/592827b562c4aa09c1e9af6b3ed7b797eef1a4dc/.bootstrap/pex/pex.py", line 401, in _wrap_coverage
    return runner(*args)
  File "/home/ubuntu/.cache/pants/named_caches/pex_root/unzipped_pexes/592827b562c4aa09c1e9af6b3ed7b797eef1a4dc/.bootstrap/pex/pex.py", line 432, in _wrap_profiling
    return runner(*args)
  File "/home/ubuntu/.cache/pants/named_caches/pex_root/unzipped_pexes/592827b562c4aa09c1e9af6b3ed7b797eef1a4dc/.bootstrap/pex/pex.py", line 539, in _execute
    return self.execute_entry(self._pex_info.entry_point)
  File "/home/ubuntu/.cache/pants/named_caches/pex_root/unzipped_pexes/592827b562c4aa09c1e9af6b3ed7b797eef1a4dc/.bootstrap/pex/pex.py", line 655, in execute_entry
    return self.execute_pkg_resources(entry_point)
  File "/home/ubuntu/.cache/pants/named_caches/pex_root/unzipped_pexes/592827b562c4aa09c1e9af6b3ed7b797eef1a4dc/.bootstrap/pex/pex.py", line 687, in execute_pkg_resources
    return runner()
  File "/home/ubuntu/.cache/pants/named_caches/pex_root/unzipped_pexes/592827b562c4aa09c1e9af6b3ed7b797eef1a4dc/.deps/pex-2.1.44-py2.py3-none-any.whl/pex/bin/pex.py", line 1159, in main
    pex_builder.build(
  File "/home/ubuntu/.cache/pants/named_caches/pex_root/unzipped_pexes/592827b562c4aa09c1e9af6b3ed7b797eef1a4dc/.deps/pex-2.1.44-py2.py3-none-any.whl/pex/pex_builder.py", line 651, in build
    with safe_open(tmp_zip, "ab") as pexfile:
  File "/home/ubuntu/.cache/pants/named_caches/pex_root/unzipped_pexes/592827b562c4aa09c1e9af6b3ed7b797eef1a4dc/.deps/pex-2.1.44-py2.py3-none-any.whl/pex/common.py", line 282, in safe_open
    return open(filename, *args, **kwargs)  # noqa: T802
Copy code
OSError: [Errno 36] File name too long: '__reqs/urllib3==1.26.7_ python_version >= _2.7_ and python_full_version < _3.0.0_ or python_full_version >= _3.6.0_ and python_version < _4_ or python_full_version >= _3.5.3_ and python_version < _4_ or python_full_version >= _3.5.0_ and python_version < _4_.pex~'
This is the full filename
e

enough-analyst-54434

10/31/2021, 3:17 PM
Aha - ok, that method of resolution / building internal PEXes was very short lived, Are you in a position to try upgrading to 2.7.1?: https://pypi.org/project/pantsbuild.pants/2.7.1
m

melodic-thailand-99227

10/31/2021, 3:20 PM
@enough-analyst-54434 what’s the current recommended version of pants?
e

enough-analyst-54434

10/31/2021, 3:21 PM
The latest stable is always a good bet and that happens to be 2.7.1: https://pypi.org/project/pantsbuild.pants/#history
What are you using now?
m

melodic-thailand-99227

10/31/2021, 3:23 PM
2.7.0
but I’m doing a PoC so I can use any version which you think will make my journey better
e

enough-analyst-54434

10/31/2021, 3:23 PM
Yeah - please try 2.7.1. The 0 -> 1 contains an important change to requirement handling mostly encapsulated here: https://github.com/pantsbuild/pants/pull/13078
m

melodic-thailand-99227

10/31/2021, 3:31 PM
OK, now it works 😐
❤️ 1
@enough-analyst-54434 Thanks!
e

enough-analyst-54434

10/31/2021, 3:32 PM
Excellent.
You're welcome.
That should also improve your remote caching effectiveness a bit, so hopefully you notice a good deal better perf.