Just finding that my CI process is taking forever to build the PEXs, and it is essentially doing the same work multiple times.

Yes, I will be caching the directories, but even for the first run, feels like it is doing more work than it needs to?

Maybe I need to publish a wheel for this 3rd party dependency that is only available as an sdist?

The sharing you refer to should already be happening

Which pexes are taking a long time to build? The

pytest_runner.pex

?

Can you share your

pants.toml

, so we have some context about whether and how you're using lockfiles for example? The Python 3rdparty deps/lockfiles support has been in a lot of flux from 2.8 to 2.11, but it should be stable in 2.11.

Also, this option is relevant: https://www.pantsbuild.org/docs/reference-python#section-run-against-entire-lockfile

I'll give that

run_against_entire_lockfile

option a go (also, the web page has some weird behaviour on Safari at least where it loads at the anchor but then scrolls back to the top of the page)

Which pexes are taking a long time to build? The

pytest_runner.pex

?

requirements.pex

...

09:27:08.88 [INFO] Initializing scheduler...
09:27:09.00 [INFO] Scheduler initialized.
09:27:14.98 [INFO] Canceled: Building pytest.pex from pytest_default_lockfile.txt
09:27:15.03 [INFO] Canceled: Building pytest.pex from pytest_default_lockfile.txt
09:27:15.09 [INFO] Canceled: Building pytest.pex from pytest_default_lockfile.txt
09:27:15.28 [INFO] Canceled: Building requirements.pex with 3 requirements: quickfix@ git+<https://github.com/jimmcslim/quickfix-python.git@v1.15.1>, structlog==21.5.0, tabulate==0.8.9
09:27:15.36 [INFO] Canceled: Building requirements.pex with 6 requirements: freezegun==1.1.0, google-cloud-datastore==2.4.0, mysql-connector-python==8.0.27, quickfix@ git+<https://github.com/jimmcslim/quickfix-python.git@v1.15.1>, ... (37 characters truncated)
09:27:15.36 [INFO] Canceled: Building requirements.pex with 3 requirements: quickfix@ git+<https://github.com/jimmcslim/quickfix-python.git@v1.15.1>, structlog==21.5.0, tabulate==0.8.9
09:27:15.36 [INFO] Canceled: Building requirements.pex with 4 requirements: ccy==1.2.2, quickfix@ git+<https://github.com/jimmcslim/quickfix-python.git@v1.15.1>, structlog==21.5.0, tabulate==0.8.9
09:27:27.30 [INFO] Completed: Building pytest.pex from pytest_default_lockfile.txt
28.33s Building requirements.pex with 6 requirements: freezegun==1.1.0, google-cloud-datastore==2.4.0, mysql-connector-python==8.0.27, quickfix@ git+<https://github.com/jimmcslim/quickfix-python.git@v1.15.1>, setup
84.29s Building requirements.pex with 6 requirements: freezegun==1.1.0, google-cloud-datastore==2.4.0, mysql-connector-python==8.0.27, quickfix@ git+<https://github.com/jimmcslim/quickfix-python.git@v1.15.1>, setup
72.25s Building requirements.pex with 9 requirements: firebase-admin==4.5.3, freezegun==1.1.0, mock-firestore==0.11.0, mysql-connector-python==8.0.27, quickfix@ git+<https://github.com/jimmcslim/quickfix-python.gi>
84.62s Building requirements.pex with 3 requirements: ccy==1.2.2, quickfix@ git+<https://github.com/jimmcslim/quickfix-python.git@v1.15.1>, structlog==21.5.0
84.62s Building requirements.pex with 2 requirements: quickfix@ git+<https://github.com/jimmcslim/quickfix-python.git@v1.15.1>, structlog==21.5.0
84.19s Building requirements.pex with 3 requirements: quickfix@ git+<https://github.com/jimmcslim/quickfix-python.git@v1.15.1>, structlog==21.5.0, tabulate==0.8.9
84.19s Building requirements.pex with 4 requirements: ccy==1.2.2, quickfix@ git+<https://github.com/jimmcslim/quickfix-python.git@v1.15.1>, structlog==21.5.0, tabulate==0.8.9
83.97s Building requirements.pex with 4 requirements: quickfix@ git+<https://github.com/jimmcslim/quickfix-python.git@v1.15.1>, structlog==21.5.0, tabulate==0.8.9, watchdog==2.1.6
83.60s Building requirements.pex with 8 requirements: firebase-admin==4.5.3, freezegun==1.1.0, mysql-connector-python==8.0.27, quickfix@ git+<https://github.com/jimmcslim/quickfix-python.git@v1.15.1>, strenum==0.4.
83.81s Building requirements.pex with 7 requirements: firebase-admin==4.5.3, mysql-connector-python==8.0.27, quickfix@ git+<https://github.com/jimmcslim/quickfix-python.git@v1.15.1>, strenum==0.4.7, structlog==21.5
83.96s Building requirements.pex with 8 requirements: ccy==1.2.2, google-cloud-datastore==2.4.0, mysql-connector-python==8.0.27, quickfix@ git+<https://github.com/jimmcslim/quickfix-python.git@v1.15.1>, setuptools=
83.64s Building requirements.pex with 8 requirements: firebase-admin==4.5.3, mock-firestore==0.11.0, mysql-connector-python==8.0.27, quickfix@ git+<https://github.com/jimmcslim/quickfix-python.git@v1.15.1>, strenum

so its the

quickfix@ git+<https://github.com/jimmcslim/quickfix-python.git@v1.15.1>

dependency (which is forked into my own repo) that takes time to build from the sdist and seems to be happening multiple times

the above log comes from having started a test run after clearing the contents of

~/.cache/pants

I've gone down the path of enabling lockfile support so my

[python]

section now looks like...

[python]
interpreter_constraints = [">=3.10.2"]
enable_resolves = true
resolves = { python-default = "pants.lock" }
run_against_entire_lockfile = true

1. I needed to change the interpreter constraint from ==3.10.2 to >=3.10.2 otherwise

generate-lockfiles

wouldn't work 2. The addition of

run_against_entire_lockfile

does indeed result in a much faster build when running tests. 3. But many tests are now failing due to

requests library is not installed

even though my lockfile contains...

requests==2.27.1; python_version >= "3.7" and python_full_version < "3.0.0" and platform_python_implementation != "PyPy" or python_full_version >= "3.6.0" and python_version >= "3.7" and platform_python_implementation != "PyPy" \
    --hash=sha256:f22fa1e554c9ddfd16e6e41ac79759e17be9e492b3587efa038054674760e72d \
    --hash=sha256:68d7c56fd5a8999887728ef304a6d12edc7be74f1cfa47714fc8b414525c9a61

On #1

ProcessExecutionFailure: Process 'Generate lockfile for python-default' failed with exit code 1.
stdout:
The currently activated Python version 3.10.3 is not supported by the project (==3.10.2).
Trying to find and use a compatible version.

  NoCompatiblePythonVersionFound

  Poetry was unable to find a compatible version. If you have one, you can explicitly use it via the "env use" command.

  at ~/.cache/pants/named_caches/pex_root/venvs/s/fa271d32/venv/lib/python3.10/site-packages/poetry/utils/env.py:736 in create_venv
       732│                     python_minor = ".".join(python_patch.split(".")[:2])
       733│                     break
       734│
       735│             if not executable:
    →  736│                 raise NoCompatiblePythonVersionFound(
       737│                     self._poetry.package.python_versions
       738│                 )
       739│
       740│         if root_venv:

So Poetry has a dependency on 3.10.3 I guess?

With the presence of the lockfile, I can remove

run_against_entire_lockfile

and things seem much faster? Apart from the

requests

problem, which goes away if I drop the lockfile

And

run_against_entire_lockfile

removes the need for per-test resolves if you have an old-style lockfile, and removes the need for subsetting if you have a new-style lockfile

Hmm that

python_version >= "3.7" and python_full_version < "3.0.0" and platform_python_implementation != "PyPy"

is pretty suspicious though

Normally even the per-test resolve would use wheel caches and such, but maybe those are bypassed for VCS requirements, you may have hit a perfect storm there...

Are you using poetry to generate the lockfile?

Try

lockfile_generator = "pex"

under

[python]

to use the latest and greatest in lockfile generation?

That should sidestep any poetry issues

lockfile_generator = "pex"

might be 2.11RC though?

yeah, updating to

2.11.0rc0

and running with

lockfile_generator = "pex"

has a problem with the git dependency 😢

I can avoid using the git dependency for the moment however

so... 1. Pants

2.11.0rc0

2.

lockfile_generator = "pex"

3.

quickfix

from PyPI and not from my git fork doesn't have the

requests

issue but, 1. Pants

2.10.0

2. no

lockfile_generator = "pex"

(as this is 2.11 only) 3.

quickfix

from PyPI and not from my git fork DOES have the

requests

issue

I haven't yet used poetry to generate the lockfile, I will do that now

2.11 should support git dependencies, what issue are you seeing there?

yet

requests

appears in the lockfile

Just to clarify...

lockfile_generator = "pex"

is not supported under 2.10?

[python]
interpreter_constraints = ["==3.10.2"]
enable_resolves = true
lockfile_generator = "pex"

❯ ./pants generate-lockfiles
14:25:35.99 [INFO] Initializing scheduler...
14:25:36.11 [INFO] Scheduler initialized.
14:25:36.13 [ERROR] Invalid option 'lockfile_generator' under [python] in pants.toml
14:25:36.15 [ERROR] Invalid config entries detected. See log for details on which entries to update or remove.
(Specify --no-verify-config to disable this check.)

I just want to clarify what the problem you're seeing with the git dependency is?

Is that the same as the requests issue?

And does the requests issue happen even with

run_against_entire_lockfile=true

?

Since, as you say, requests is in the lockfile, that would be odd