https://pantsbuild.org/ logo
#general
Title
# general
r

rapid-exabyte-76685

04/04/2022, 5:30 AM
Is there any way to get Pants to somehow 'share' the binaries built from an sdist between the PEXs that get created for different sets of tests? Or is the correct way to do this to build a wheel/bdist for the dependency that is otherwise expressed as an sdist (accessed via a git reference)
Just finding that my CI process is taking forever to build the PEXs, and it is essentially doing the same work multiple times.
Yes, I will be caching the directories, but even for the first run, feels like it is doing more work than it needs to?
Maybe I need to publish a wheel for this 3rd party dependency that is only available as an sdist?
h

happy-kitchen-89482

04/04/2022, 3:34 PM
Pants shouldn't be rebuilding a 3rdparty sdist over and over, so that does seem wrong.
The sharing you refer to should already be happening
Which pexes are taking a long time to build? The
pytest_runner.pex
?
Can you share your
pants.toml
, so we have some context about whether and how you're using lockfiles for example? The Python 3rdparty deps/lockfiles support has been in a lot of flux from 2.8 to 2.11, but it should be stable in 2.11.
r

rapid-exabyte-76685

04/04/2022, 10:08 PM
My
pants.toml
is fairly simple at the moment...
Copy code
[GLOBAL]
pants_version = "2.10.0"
backend_packages = [
    "pants.backend.python",
    "pants.backend.python.lint.black",
]
use_deprecated_python_macros = false

[anonymous-telemetry]
enabled = false # Apologies about this, I'm not being a good member of the Pants community having this disabled.

[tailor]
ignore_paths = ["scripts/**"]

[python]
interpreter_constraints = ["==3.10.2"]

[source]
root_patterns = ["/"]

[python-bootstrap]
search_path = ["<PYENV>"]
I'll give that
run_against_entire_lockfile
option a go (also, the web page has some weird behaviour on Safari at least where it loads at the anchor but then scrolls back to the top of the page)
Which pexes are taking a long time to build? The
pytest_runner.pex
?
requirements.pex
...
Copy code
09:27:08.88 [INFO] Initializing scheduler...
09:27:09.00 [INFO] Scheduler initialized.
09:27:14.98 [INFO] Canceled: Building pytest.pex from pytest_default_lockfile.txt
09:27:15.03 [INFO] Canceled: Building pytest.pex from pytest_default_lockfile.txt
09:27:15.09 [INFO] Canceled: Building pytest.pex from pytest_default_lockfile.txt
09:27:15.28 [INFO] Canceled: Building requirements.pex with 3 requirements: quickfix@ git+<https://github.com/jimmcslim/quickfix-python.git@v1.15.1>, structlog==21.5.0, tabulate==0.8.9
09:27:15.36 [INFO] Canceled: Building requirements.pex with 6 requirements: freezegun==1.1.0, google-cloud-datastore==2.4.0, mysql-connector-python==8.0.27, quickfix@ git+<https://github.com/jimmcslim/quickfix-python.git@v1.15.1>, ... (37 characters truncated)
09:27:15.36 [INFO] Canceled: Building requirements.pex with 3 requirements: quickfix@ git+<https://github.com/jimmcslim/quickfix-python.git@v1.15.1>, structlog==21.5.0, tabulate==0.8.9
09:27:15.36 [INFO] Canceled: Building requirements.pex with 4 requirements: ccy==1.2.2, quickfix@ git+<https://github.com/jimmcslim/quickfix-python.git@v1.15.1>, structlog==21.5.0, tabulate==0.8.9
09:27:27.30 [INFO] Completed: Building pytest.pex from pytest_default_lockfile.txt
28.33s Building requirements.pex with 6 requirements: freezegun==1.1.0, google-cloud-datastore==2.4.0, mysql-connector-python==8.0.27, quickfix@ git+<https://github.com/jimmcslim/quickfix-python.git@v1.15.1>, setup
84.29s Building requirements.pex with 6 requirements: freezegun==1.1.0, google-cloud-datastore==2.4.0, mysql-connector-python==8.0.27, quickfix@ git+<https://github.com/jimmcslim/quickfix-python.git@v1.15.1>, setup
72.25s Building requirements.pex with 9 requirements: firebase-admin==4.5.3, freezegun==1.1.0, mock-firestore==0.11.0, mysql-connector-python==8.0.27, quickfix@ git+<https://github.com/jimmcslim/quickfix-python.gi>
84.62s Building requirements.pex with 3 requirements: ccy==1.2.2, quickfix@ git+<https://github.com/jimmcslim/quickfix-python.git@v1.15.1>, structlog==21.5.0
84.62s Building requirements.pex with 2 requirements: quickfix@ git+<https://github.com/jimmcslim/quickfix-python.git@v1.15.1>, structlog==21.5.0
84.19s Building requirements.pex with 3 requirements: quickfix@ git+<https://github.com/jimmcslim/quickfix-python.git@v1.15.1>, structlog==21.5.0, tabulate==0.8.9
84.19s Building requirements.pex with 4 requirements: ccy==1.2.2, quickfix@ git+<https://github.com/jimmcslim/quickfix-python.git@v1.15.1>, structlog==21.5.0, tabulate==0.8.9
83.97s Building requirements.pex with 4 requirements: quickfix@ git+<https://github.com/jimmcslim/quickfix-python.git@v1.15.1>, structlog==21.5.0, tabulate==0.8.9, watchdog==2.1.6
83.60s Building requirements.pex with 8 requirements: firebase-admin==4.5.3, freezegun==1.1.0, mysql-connector-python==8.0.27, quickfix@ git+<https://github.com/jimmcslim/quickfix-python.git@v1.15.1>, strenum==0.4.
83.81s Building requirements.pex with 7 requirements: firebase-admin==4.5.3, mysql-connector-python==8.0.27, quickfix@ git+<https://github.com/jimmcslim/quickfix-python.git@v1.15.1>, strenum==0.4.7, structlog==21.5
83.96s Building requirements.pex with 8 requirements: ccy==1.2.2, google-cloud-datastore==2.4.0, mysql-connector-python==8.0.27, quickfix@ git+<https://github.com/jimmcslim/quickfix-python.git@v1.15.1>, setuptools=
83.64s Building requirements.pex with 8 requirements: firebase-admin==4.5.3, mock-firestore==0.11.0, mysql-connector-python==8.0.27, quickfix@ git+<https://github.com/jimmcslim/quickfix-python.git@v1.15.1>, strenum
so its the
quickfix@ git+<https://github.com/jimmcslim/quickfix-python.git@v1.15.1>
dependency (which is forked into my own repo) that takes time to build from the sdist and seems to be happening multiple times
the above log comes from having started a test run after clearing the contents of
~/.cache/pants
I've gone down the path of enabling lockfile support so my
[python]
section now looks like...
Copy code
[python]
interpreter_constraints = [">=3.10.2"]
enable_resolves = true
resolves = { python-default = "pants.lock" }
run_against_entire_lockfile = true
1. I needed to change the interpreter constraint from ==3.10.2 to >=3.10.2 otherwise
generate-lockfiles
wouldn't work 2. The addition of
run_against_entire_lockfile
does indeed result in a much faster build when running tests. 3. But many tests are now failing due to
requests library is not installed
even though my lockfile contains...
Copy code
requests==2.27.1; python_version >= "3.7" and python_full_version < "3.0.0" and platform_python_implementation != "PyPy" or python_full_version >= "3.6.0" and python_version >= "3.7" and platform_python_implementation != "PyPy" \
    --hash=sha256:f22fa1e554c9ddfd16e6e41ac79759e17be9e492b3587efa038054674760e72d \
    --hash=sha256:68d7c56fd5a8999887728ef304a6d12edc7be74f1cfa47714fc8b414525c9a61
On #1
Copy code
ProcessExecutionFailure: Process 'Generate lockfile for python-default' failed with exit code 1.
stdout:
The currently activated Python version 3.10.3 is not supported by the project (==3.10.2).
Trying to find and use a compatible version.

  NoCompatiblePythonVersionFound

  Poetry was unable to find a compatible version. If you have one, you can explicitly use it via the "env use" command.

  at ~/.cache/pants/named_caches/pex_root/venvs/s/fa271d32/venv/lib/python3.10/site-packages/poetry/utils/env.py:736 in create_venv
       732│                     python_minor = ".".join(python_patch.split(".")[:2])
       733│                     break
       734│
       735│             if not executable:
    →  736│                 raise NoCompatiblePythonVersionFound(
       737│                     self._poetry.package.python_versions
       738│                 )
       739│
       740│         if root_venv:
So Poetry has a dependency on 3.10.3 I guess?
With the presence of the lockfile, I can remove
run_against_entire_lockfile
and things seem much faster? Apart from the
requests
problem, which goes away if I drop the lockfile
h

happy-kitchen-89482

04/05/2022, 3:24 AM
Ah yes, with a "proper" lockfile those individual requirements pexes are created via fast subsetting instead of slow re-resolving
👍 1
And
run_against_entire_lockfile
removes the need for per-test resolves if you have an old-style lockfile, and removes the need for subsetting if you have a new-style lockfile
Hmm that
python_version >= "3.7" and python_full_version < "3.0.0" and platform_python_implementation != "PyPy"
is pretty suspicious though
Normally even the per-test resolve would use wheel caches and such, but maybe those are bypassed for VCS requirements, you may have hit a perfect storm there...
Are you using poetry to generate the lockfile?
Try
lockfile_generator = "pex"
under
[python]
to use the latest and greatest in lockfile generation?
That should sidestep any poetry issues
r

rapid-exabyte-76685

04/05/2022, 3:34 AM
I used
./pants generate-lockfiles
to generate the lockfile - and I'm on
pants_version = "2.10.0"
- I'll try
lockfile_generator = "pex"
lockfile_generator = "pex"
might be 2.11RC though?
yeah, updating to
2.11.0rc0
and running with
lockfile_generator = "pex"
has a problem with the git dependency 😢
I can avoid using the git dependency for the moment however
so... 1. Pants
2.11.0rc0
2.
lockfile_generator = "pex"
3.
quickfix
from PyPI and not from my git fork doesn't have the
requests
issue but, 1. Pants
2.10.0
2. no
lockfile_generator = "pex"
(as this is 2.11 only) 3.
quickfix
from PyPI and not from my git fork DOES have the
requests
issue
I haven't yet used poetry to generate the lockfile, I will do that now
h

happy-kitchen-89482

04/05/2022, 4:11 AM
Without
lockfile_generator = "pex"
poetry will be used to generate the lockfile in 2.10 and 2.11
2.11 should support git dependencies, what issue are you seeing there?
r

rapid-exabyte-76685

04/05/2022, 4:18 AM
A bunch of tests fail saying that
requests library is not installed
yet
requests
appears in the lockfile
Just to clarify...
lockfile_generator = "pex"
is not supported under 2.10?
Copy code
[python]
interpreter_constraints = ["==3.10.2"]
enable_resolves = true
lockfile_generator = "pex"
Copy code
❯ ./pants generate-lockfiles
14:25:35.99 [INFO] Initializing scheduler...
14:25:36.11 [INFO] Scheduler initialized.
14:25:36.13 [ERROR] Invalid option 'lockfile_generator' under [python] in pants.toml
14:25:36.15 [ERROR] Invalid config entries detected. See log for details on which entries to update or remove.
(Specify --no-verify-config to disable this check.)
h

happy-kitchen-89482

04/05/2022, 3:30 PM
Correct, that requires 2.11
I just want to clarify what the problem you're seeing with the git dependency is?
Is that the same as the requests issue?
And does the requests issue happen even with
run_against_entire_lockfile=true
?
Since, as you say, requests is in the lockfile, that would be odd
h

hundreds-father-404

04/05/2022, 3:45 PM
use 2.11.0rc1 for git suppport w/ pex lockfiles
3 Views