I'm seeing an unexpected behavioral change when updating from 2.10.0 to 2.11.0. I generally run Python tests in this particular repository with the following:

./pants filter --target-type=python_tests :: | xargs ./pants --tag="-e2e" test

. This worked fine with 2.10.0 and appears to have started failing with

2.11.0.dev0

. The failure comes with the tag-based filter: my tests that are tagged as "e2e" tests (which I don't want to run in CI when I'm trying to do unit tests, since they rely on external infrastructure resources) are also being run, and are not being filtered out. Even taking out the

xargs

bit and just running

./pants --tag="-e2e" test ::

fails. Has anyone encountered something similar? Any pointers on how to deal with this? Thanks 🙇

With the new pants-native lockfile generation, how is it expected for pants to stay aware if lockfiles are properly updated. As an example, I generated my lockfile, ran a test, changed a relevant line in

requirements.txt

that would invalidate the lockfile (for a transitive dependency of the test), but I did not get prompted at all when I reran my test.

Does Dockerfile dep inference work with

instructions

? I can't seem to get it to infer a dep on a

pex_binary

.

pex_binaries(
    name = "binaries",
    entry_points = [
        "doodad.py",
    ],
)

docker_image(
    name="img_name",
    instructions=[
        "FROM ubuntu:20.04",
        (
            "RUN apt-get -qq -y update &&"
            + " DEBIAN_FRONTEND=noninteractive apt-get -qq -y install"
            + " python3.8"
        ),
        "COPY path.to.thingamabob/doodad_py.pex /bin/helloworld",
    ],
)

If I

./pants package path/to/thingamabob/doodad.py

I see

Wrote dist/path.to.thingamabob/doodad_py.pex

another post-2.11-upgrade question: after upgrading,

./pants export ::

started to fail with:

unknown command: python. Perhaps you have to reshim?

which is an error from

asdf

. I saw a related fix had already been merged & cherry-picked, and pointing at the SHA fixed the issue for me:

PANTS_SHA=9ac327d46d8ef98a6f635c9795e26433b50e62b8

but even though it fixed the error, running

./pants export ::

still produced warnings that weren’t there in 2.10:

16:01:16.46 [INFO] Completed: Building requirements.pex with 177 requirements: agefromname==0.0.6, amqp==2.6.1, authlib==1.0.0, awslambdaric==2.0.1, beautifulsoup4==4.10.0, biocommons.seqrepo==0.6.5, biopython==1.76, bioutils==0.5... (3765 characters truncated)
16:01:16.55 [INFO] Completed: Get interpreter version
/Users/dan.moran/.pex/installed_wheels/6a5463833452712ecf58a573b1b489c81d14aaed599e955c2af85cf793217a42/pex-2.1.84-py2.py3-none-any.whl/pex/dist_metadata.py:332: PEXWarning: Ignoring 1 `Requires` field in pysam 0.15.4 metadata:
1.) Requires: cython (>=0.29.12)

You may have issues using the 'pysam' distribution as a result.
More information on this workaround can be found here:
  <https://github.com/pantsbuild/pex/issues/1201#issuecomment-791715585>

  pex_warnings.warn(
/Users/dan.moran/.pex/installed_wheels/6a5463833452712ecf58a573b1b489c81d14aaed599e955c2af85cf793217a42/pex-2.1.84-py2.py3-none-any.whl/pex/dist_metadata.py:332: PEXWarning: Ignoring 2 `Requires` fields in rdflib 4.2.2 metadata:
1.) Requires: isodate
2.) Requires: pyparsing

You may have issues using the 'rdflib' distribution as a result.
More information on this workaround can be found here:
  <https://github.com/pantsbuild/pex/issues/1201#issuecomment-791715585>

  pex_warnings.warn(
/Users/dan.moran/.pex/installed_wheels/6a5463833452712ecf58a573b1b489c81d14aaed599e955c2af85cf793217a42/pex-2.1.84-py2.py3-none-any.whl/pex/venv/pex.py:160: PEXWarning: Encountered collisions building venv at /Users/dan.moran/color/dist/export/python/virtualenv/3.8.12 from /Users/dan.moran/color/dist/export/python/virtualenv/requirements.pex:
1. /Users/dan.moran/color/dist/export/python/virtualenv/3.8.12/lib/python3.8/site-packages/LICENSE was provided by:
        sha1:b88cc146668fa09ce5aa419bf7022c39d5cf52cb -> /Users/dan.moran/.pex/installed_wheels/7ca4e84ea192db2bafdec3c54d31518357073288fd3cf10dd87ead6784527bbf/pyppeteer-0.2.6-py3-none-any.whl/LICENSE
        sha1:a4466bfd6f8b645aebdf508c1b7ede0bc018f694 -> /Users/dan.moran/.pex/installed_wheels/427414fcd945d3992364ad129c76f6f0db1a5a0a13a08f84f12c1b7c33fa9a7a/rsa-4.8-py3-none-any.whl/LICENSE
2. /Users/dan.moran/color/dist/export/python/virtualenv/3.8.12/lib/python3.8/site-packages/README.md was provided by:
        sha1:d22f65503e9ee765f10290d7dc5c56c8d539e942 -> /Users/dan.moran/.pex/installed_wheels/7ca4e84ea192db2bafdec3c54d31518357073288fd3cf10dd87ead6784527bbf/pyppeteer-0.2.6-py3-none-any.whl/README.md
        sha1:72aad144e976ac1623c8f45f2cf6ca42134ad4b2 -> /Users/dan.moran/.pex/installed_wheels/427414fcd945d3992364ad129c76f6f0db1a5a0a13a08f84f12c1b7c33fa9a7a/rsa-4.8-py3-none-any.whl/README.md
  pex_warnings.warn(message)
Wrote virtualenv for the resolve 'None' (using Python 3.8.12) to dist/export/python/virtualenv

are any of these warnings a sign of an issue in my project setup? should I file a bug report?

I feel like this was asked before, but I can't seem to find the thread: is there a way to

./pants run

something without moving to a sandboxed area?

./pants package <docker image>

made my env-based image tags forced to lower case. Is that expected? Any way to avoid that?

Python 3rd party deps page rewritten to talk about Poetry vs Pex lockfiles, along w/

parametrize

in the context of multiple resolves. Feedback/questions welcomed! https://www.pantsbuild.org/docs/python-third-party-dependencies#lockfiles

I think

export

is much slower with a pex-based lockfile. Is this expected?

Before I switched over to generating pex lockfiles, I was seeing an export time of about 12 seconds on fresh runs and less than a second on successive runs. Now I'm getting anywhere from 44 seconds to over a minute.

No, I would not expect that. Note that we made a change in 2.11 to include PIP in the virtual environment. Do you have a checkpoint where you are using 2.11 but with poetry lockfiles? To confirm that this is about pex versus poetry rather than 2.10 versus 2.11

Actually. It might be because I was testing previously with the VCS requirement commented out. I know pip doesn't cache vcs installs, but does pants do anything intelligent with that? We specify the full commit hash so there's definitely enough information there.

removing the vcs dep from requirements.txt and regenerating the lockfile and running export didn't seem to change things. Still takes ~45 seconds best case.

I might have been working with 2.10 when testing previously

Will get some more data

In 2.10 we "exported" a symlink directly into the pex_root cache, which was dangerous for a few reasons, and also meant that the resulting virtualenv didn't include pip and couldn't be easily used in various ways

In 2.11 we actually create a "proper", separate virtualenv, so I can see how that would take longer

Although the resolve and downloads and all should still be used from cache

Okay here's my findings • v2.10.1rc0, no VCS, lockfile from pip-tools, export takes 43 seconds. Running immediately after and it takes 20 seconds. Ran again and it takes 1 second each successive time. • v2.10.1rc0, with VCS, lockfile from pip-tools, export takes 2m30s. Running immediately after and it takes 22 seconds. Running again takes 1 second and all successive runs take that long. okay so VCS isn't the issue (though weird that it takes three runs to reach its best caching). • v2.11.0, with VCS, lockfile from pants with pex, export takes 58 seconds. Running again takes 41 seconds. Again at 38 seconds. Again up to 1m2s. Again 46 so doesn't seem like it's converging like it used to. • v2.11.0, no VCS, lockfile from pants with pex, export takes 1m7s. Running again takes 48 seconds. Again at 46 seconds. Doesn't show signs of converging. Okay so 2.11 changes things but unclear if it's 2.11 or pex lockfile. Revert back to pip-tools controlled lockfile and see what happens. • v2.11.0, no VCS, lockfile from pip-tools, export takes 2m50s (a lot of time installing constraints file). Running again is 47 seconds and that seems to continue indefinitely. saving sanity and going to assume adding back VCS doesn't make it better. So I'm fairly confident it's just the change from 2.10 to 2.11.

I'd take a little danger for 40x speed improvement 😉

I do wonder why it is taking so long, when you already have a lockfile and all the wheels should already be downloaded/built and cached

@enough-analyst-54434 Any thoughts?

How many dists in the resulting virtualenv?

Somewhere in the neighborhood of 200

No clue from the Pex side. There is alot of complex stuff going on above though in Pants too. As always it would be best to get a shareable repro if possible.

I mean, the change you mentioned Benjy explains it all I think. Since there is no symlink, a VCS clone is happening every time. Pex does not trust any VCS URL to be stable and so both Pip cloning the repo and checking out the given branch, tag or commit happens every time, and then Pex sha256 hasing that repo tree happens every time to confirm no float.

To speed this up PEX would need to relax a bit and trust certain VCS urls, like commit ids for cryptographic VCS (git commit IDs but not svn, for example) but not tags or branches.

That's assuming the git clone time here roughly aligns with 1 minute, explaining the non-convergence. Is that the case @high-yak-85899?

I think this is slow even without the VCS requirement though?

I don't know, I'm having a hard time following. That, at least, is the only thing I can think of on the Pex side of the world.

Q26: Regarding my Q22 to have package metadata inside pex environments like editable installation, I'd like to access BUILD files inside the auto-generated pex environment. But it seems that they are not available there. I'm trying to scan the entrypoint from BUILD files manually as an alternative to editable installation. Could you suggest how to achieve it or better alternatives?