🤔 After upgrading to 2.9, I'm seeing some failures when running Pytest tests.

ProcessExecutionFailure: Process 'Building pytest_runner.pex' failed with exit code 1.

Followed by a bunch of these:

pex.environment.ResolveError: Failed to resolve requirements from PEX environment @ /home/maier/.cache/pants/named_caches/pex_root/unzipped_pexes/f00d5546a767d36a195e863e80031df5797b7eb8.
Needed manylinux_2_28_x86_64-cp-37-cp37m compatible dependencies for:
 1: boto3
    Required by:
      FingerprintedDistribution(distribution=grapl-common 0.0.6 (/home/maier/.cache/pants/named_caches/pex_root/installed_wheels/7f5c555e91114b099cc3946517adba787c2ede3e/grapl_common-0.0.6-py3-none-any.whl), fingerprint='a76df1c3f9ecd1ae5ae2352850c933b30140ba93')
    But this pex had no 'boto3' distributions.
 2: botocore
    Required by:
      FingerprintedDistribution(distribution=grapl-common 0.0.6 (/home/maier/.cache/pants/named_caches/pex_root/installed_wheels/7f5c555e91114b099cc3946517adba787c2ede3e/grapl_common-0.0.6-py3-none-any.whl), fingerprint='a76df1c3f9ecd1ae5ae2352850c933b30140ba93')
    But this pex had no 'botocore' distributions.
 3: mypy-boto3-cloudwatch
    Required by:
      FingerprintedDistribution(distribution=grapl-common 0.0.6 (/home/maier/.cache/pants/named_caches/pex_root/installed_wheels/7f5c555e91114b099cc3946517adba787c2ede3e/grapl_common-0.0.6-py3-none-any.whl), fingerprint='a76df1c3f9ecd1ae5ae2352850c933b30140ba93')
    But this pex had no 'mypy-boto3-cloudwatch' distributions.
 4: mypy-boto3-dynamodb
    Required by:
      FingerprintedDistribution(distribution=grapl-common 0.0.6 (/home/maier/.cache/pants/named_caches/pex_root/installed_wheels/7f5c555e91114b099cc3946517adba787c2ede3e/grapl_common-0.0.6-py3-none-any.whl), fingerprint='a76df1c3f9ecd1ae5ae2352850c933b30140ba93')
    But this pex had no 'mypy-boto3-dynamodb' distributions.

(plus 8 similar missing dependencies) Any pointers on where to look? I've regenerated my pytest lockfile. Thanks in advance.

Huh. What was the last Pants version that worked? What's interesting is that

pytest_runner.pex

is actually the fusion of a few PEXes, including the

pytest.pex

(your tool lockfile) and the

requirements.pex

(your code). It uses the

--pex-path

feature which allows you to merge PEXes

I upgraded from 2.8 to 2.9 today

Those do both look related, thanks for bisecting this. What happens btw if you temporarily remove the ~/.cache/pants/named_caches so it's unused? (preferred if you can preserve the folder for now in case we need it to debug)

Moving the

named_caches

directory aside (creatively renamed it to

named_caches_orig

) and running my test yields the same failure.

Hm, and then how about deleting the just created

named_caches

and using

--no-local-cache --no-pantsd

? Want to make sure this isn't from a "corrupted" Pex cache or anything, which has happened in the past iirc.

Still failed in the same way... bummer

Hm I've been trying to think of ideas. What were the changes to the Pytest lockfile and any other lockfiles?

one sec, let me go see

Hm, thanks! None of those mention the requirements giving you issues. I'm wondering if this is an issue with

manylinux_2_28_x86_64-cp-37-cp37m

by chance. @polite-garden-50641 found that upgrading to Pants 2.10 is resulting in this error message for us:

Failed to resolve requirements from PEX environment @ /home/toolchain/.cache/pants/named_caches/pex_root/unzipped_pexes/11c095ad69b7a3c1a757dbfcd20347104c1a6840.
Needed manylinux_2_28_x86_64-cp-38-cp38 compatible dependencies for:
 1: py>=1.8.2
    Required by:
      FingerprintedDistribution(distribution=pytest 6.2.5 (/home/toolchain/.cache/pants/named_caches/pex_root/installed_wheels/d8dbe72f24eaeea8b51e0f353336f9bd4915dc4a/pytest-6.2.5-py3-none-any.whl), fingerprint='c81a0f09ee54ea9496a98d7b1b12d41286248226')
    But this pex had no 'py' distributions.

To confirm, do you get this issue on macOS? (if you have access)

keep in mind we don't use the lock file feature on our 2.9 branch. so our branch that upgrades to 2.10 (dev) also switches from constraints file to a lock file.

I don't have easy access to a mac for this, sorry

That's okay! Is this one of your open source projects that I could try running? I'm wanting to try to reproduce with Pex directly but am not sure how to do that off the top of my head

Yup! This ought to reproduce it (that gnarly

pants

invocation hits the specific test files that trigger this)

git clone <https://github.com/grapl-security/grapl>
git checkout cm/pants-2.9
./pants --tag="-needs_work" test --pytest-args="-m \"not integration_test\"" src/python/graplctl/tests:tests

Thank you! I'm about to log off for the long weekend, but will play with this on Tuesday. Have a great weekend!

Sounds good; thanks for your help! Have a good weekend yourself! 👋

Ah thanks for the ping. I don't - any ideas @enough-analyst-54434?

You guys already found the idea: https://github.com/pantsbuild/pants/pull/13690 - that's exactly it. I comment on that PR how I hate such a long-winded explanation, but its required reading unfortunately. The book there talks about hand-rolled setup.py / pyproject.toml though and here grapl is using the auto-setup.py. I'm looking at that angle presently...

Cool, thanks very much @enough-analyst-54434, @hundreds-father-404 🙇

@brash-baker-91190 just for background, is depending on the src/python/grapl-common python_distribution instead of the src/python/grapl-common/grapl_common pythoon_sources intentional? This diff gets your test command above passing and eliminates the middle-man of packaging up grapl-common into an sdist / wheel:

diff --git a/src/python/graplctl/graplctl/BUILD b/src/python/graplctl/graplctl/BUILD
index 1e8a0ce9..050a642b 100644
--- a/src/python/graplctl/graplctl/BUILD
+++ b/src/python/graplctl/graplctl/BUILD
@@ -1,14 +1,7 @@
 python_sources(
     name="graplctl_lib",
-    dependencies=[
-        "src/python/grapl-common:grapl-common",
-    ],
 )
 
 pex_binary(
     entry_point="cli.py:main",
-    # TODO: Once <https://github.com/pantsbuild/pants/pull/13690> lands,
-    # remove this work-around
-    # See also <https://pantsbuild.slack.com/archives/C046T6T9U/p1637597807103900>
-    dependencies=["!!src/python/grapl-common:grapl-common"],
 )

Hrmm... yeah, it looks like we could do that 😅 Oh man, it even says it in the comments 🤦

OK. I'm still looking into the issue sans workaround since maybe its conceivable someone might have a use case where they actually want to depend on the wheel generated from the sources instead of the sources? I can't come up with one off the top of my head, but the world is a big place.

Testing against the artifacts you'd ultimately end up shipping?

I guess. That's effectively testing Pants works though in this case of auto generated setup.py.

Just an update here: I can now run my unit tests, but our integration tests are now complaining. I'm not 100% sure what's happening, but we have some Python services that are basically a PEX-in-a-container (not managed by Pants' Docker integration at the moment, FWIW) and they're not starting up in Nomad... just seeing errors like this:

/bin/bash: line 1: ./analyzer-executor.pex: Permission denied

I'm continuing to bisect things to figure out at what point in the 2.9 dev/rc line our system starts to fall over. Our test process is kind of long, though, so I'm just picking at this between other tasks (we're not blocked or anything; Pants 2.8 continues to work like a champ for us in this repository). Just wanted to keep y'all in the loop; I'll report back if/when I find anything.

they're not starting up in Nomad... just seeing errors like this: 

/bin/bash: line 1: ./analyzer-executor.pex: Permission denied

That looks like it can only really be that the 755 bits added to

analyzer-executor.pex

by Pex when it creates the PEX have been wiped out. That shouldn't be related to any Pex or Pants upgrades. Are you sure there are no other variable in play for that error re perms / cp, etc?

It's possible... been a while since I've been in this bit of code, and there's been a lot of change in this area recently. We're probably doing something "non-standard" somewhere 😅

Pex itself still generates PEX files 755. This smells like a Pants behavior change. And it is:

$ rm -rf dist && ./pants package src/python/pants/bin::
07:52:22.51 [INFO] Completed: Installing 3rdparty/python/lockfiles/user_reqs.txt
07:52:28.06 [INFO] Completed: Extracting 15 requirements to build src.python.pants.bin/pants.pex from lockfile.pex: PyYAML<7.0,>=6.0, ansicolors==1.1.8, fasteners==0.16.3, humbug==0.2.7, ijson==3.1.4, packaging==21.3, pex==2.1.65,... (168 characters truncated)
07:52:28.08 [INFO] Wrote dist/src.python.pants.bin/pants.pex
$ ls -l dist/src.python.pants.bin/pants.pex
-rwxr--r-- 1 jsirois jsirois 53595441 Jan 27 07:52 dist/src.python.pants.bin/pants.pex

That seems worth a bug @brash-baker-91190

@enough-analyst-54434 OK, I'll file an issue

Thanks Chris.

I'll also see if I can narrow it down to a specific dev or rc release of 2.9

Awesome. Thanks for digging.

Yeah, sure thing! 👍