https://pantsbuild.org/ logo
#general
Title
# general
b

brash-baker-91190

01/13/2022, 9:23 PM
🤔 After upgrading to 2.9, I'm seeing some failures when running Pytest tests.
Copy code
ProcessExecutionFailure: Process 'Building pytest_runner.pex' failed with exit code 1.
Followed by a bunch of these:
Copy code
pex.environment.ResolveError: Failed to resolve requirements from PEX environment @ /home/maier/.cache/pants/named_caches/pex_root/unzipped_pexes/f00d5546a767d36a195e863e80031df5797b7eb8.
Needed manylinux_2_28_x86_64-cp-37-cp37m compatible dependencies for:
 1: boto3
    Required by:
      FingerprintedDistribution(distribution=grapl-common 0.0.6 (/home/maier/.cache/pants/named_caches/pex_root/installed_wheels/7f5c555e91114b099cc3946517adba787c2ede3e/grapl_common-0.0.6-py3-none-any.whl), fingerprint='a76df1c3f9ecd1ae5ae2352850c933b30140ba93')
    But this pex had no 'boto3' distributions.
 2: botocore
    Required by:
      FingerprintedDistribution(distribution=grapl-common 0.0.6 (/home/maier/.cache/pants/named_caches/pex_root/installed_wheels/7f5c555e91114b099cc3946517adba787c2ede3e/grapl_common-0.0.6-py3-none-any.whl), fingerprint='a76df1c3f9ecd1ae5ae2352850c933b30140ba93')
    But this pex had no 'botocore' distributions.
 3: mypy-boto3-cloudwatch
    Required by:
      FingerprintedDistribution(distribution=grapl-common 0.0.6 (/home/maier/.cache/pants/named_caches/pex_root/installed_wheels/7f5c555e91114b099cc3946517adba787c2ede3e/grapl_common-0.0.6-py3-none-any.whl), fingerprint='a76df1c3f9ecd1ae5ae2352850c933b30140ba93')
    But this pex had no 'mypy-boto3-cloudwatch' distributions.
 4: mypy-boto3-dynamodb
    Required by:
      FingerprintedDistribution(distribution=grapl-common 0.0.6 (/home/maier/.cache/pants/named_caches/pex_root/installed_wheels/7f5c555e91114b099cc3946517adba787c2ede3e/grapl_common-0.0.6-py3-none-any.whl), fingerprint='a76df1c3f9ecd1ae5ae2352850c933b30140ba93')
    But this pex had no 'mypy-boto3-dynamodb' distributions.
(plus 8 similar missing dependencies) Any pointers on where to look? I've regenerated my pytest lockfile. Thanks in advance.
👀 1
h

hundreds-father-404

01/13/2022, 9:25 PM
Huh. What was the last Pants version that worked? What's interesting is that
pytest_runner.pex
is actually the fusion of a few PEXes, including the
pytest.pex
(your tool lockfile) and the
requirements.pex
(your code). It uses the
--pex-path
feature which allows you to merge PEXes
b

brash-baker-91190

01/13/2022, 9:25 PM
I upgraded from 2.8 to 2.9 today
And 2.8 worked without a hitch
(stepping away for a bit, but will be back soon)
Some more information: • this is only happening for a small subset of my tests; most run fine • everything that's missing is required by a Python library (
grapl-common
, which exists within this monorepo) that is a dependency of the code under test. The tests for
grapl-common
run just fine. • with the exception of
boto3
and
botocore
(numbers 1 and 2 in the output above), all the missing dependencies are
mypy-boto-XXX
type libraries and
typing-extensions
Not sure if this is useful, but this is what stands out to me at this point.
Tracing backwards, it appears that something in version 2.9.0.dev3 triggers the failure.
Looking at the changelog (https://pypi.org/project/pantsbuild.pants/2.9.0.dev3/), possible candidates might be: • https://github.com/pantsbuild/pants/pull/13713 (Upgrade to Pex 2.1.56) • https://github.com/pantsbuild/pants/pull/13690 (Fix dup 3rdparty dists introduced by local dists) (those jumped out at me as possibly related, in a sea of Java / Go updates)
h

hundreds-father-404

01/13/2022, 11:27 PM
Those do both look related, thanks for bisecting this. What happens btw if you temporarily remove the ~/.cache/pants/named_caches so it's unused? (preferred if you can preserve the folder for now in case we need it to debug)
b

brash-baker-91190

01/13/2022, 11:29 PM
Moving the
named_caches
directory aside (creatively renamed it to
named_caches_orig
) and running my test yields the same failure.
👍 1
h

hundreds-father-404

01/13/2022, 11:30 PM
Hm, and then how about deleting the just created
named_caches
and using
--no-local-cache --no-pantsd
? Want to make sure this isn't from a "corrupted" Pex cache or anything, which has happened in the past iirc.
b

brash-baker-91190

01/13/2022, 11:32 PM
Still failed in the same way... bummer
@hundreds-father-404 Just curious if you might have any other ideas on this issue?
h

hundreds-father-404

01/14/2022, 10:45 PM
Hm I've been trying to think of ideas. What were the changes to the Pytest lockfile and any other lockfiles?
b

brash-baker-91190

01/14/2022, 10:51 PM
one sec, let me go see
pytest.diff
black.diff,mypy.diff,isort.diff
h

hundreds-father-404

01/14/2022, 10:58 PM
Hm, thanks! None of those mention the requirements giving you issues. I'm wondering if this is an issue with
manylinux_2_28_x86_64-cp-37-cp37m
by chance. @polite-garden-50641 found that upgrading to Pants 2.10 is resulting in this error message for us:
Copy code
Failed to resolve requirements from PEX environment @ /home/toolchain/.cache/pants/named_caches/pex_root/unzipped_pexes/11c095ad69b7a3c1a757dbfcd20347104c1a6840.
Needed manylinux_2_28_x86_64-cp-38-cp38 compatible dependencies for:
 1: py>=1.8.2
    Required by:
      FingerprintedDistribution(distribution=pytest 6.2.5 (/home/toolchain/.cache/pants/named_caches/pex_root/installed_wheels/d8dbe72f24eaeea8b51e0f353336f9bd4915dc4a/pytest-6.2.5-py3-none-any.whl), fingerprint='c81a0f09ee54ea9496a98d7b1b12d41286248226')
    But this pex had no 'py' distributions.
To confirm, do you get this issue on macOS? (if you have access)
I'm wondering if this is an issue with manylinux_2_28_x86_64-cp-37-cp37m by chance.
Probably red-herring. Asher also has the issue on a mac
p

polite-garden-50641

01/14/2022, 11:05 PM
keep in mind we don't use the lock file feature on our 2.9 branch. so our branch that upgrades to 2.10 (dev) also switches from constraints file to a lock file.
👍 1
b

brash-baker-91190

01/14/2022, 11:27 PM
I don't have easy access to a mac for this, sorry
h

hundreds-father-404

01/14/2022, 11:29 PM
That's okay! Is this one of your open source projects that I could try running? I'm wanting to try to reproduce with Pex directly but am not sure how to do that off the top of my head
b

brash-baker-91190

01/14/2022, 11:46 PM
Yup! This ought to reproduce it (that gnarly
pants
invocation hits the specific test files that trigger this)
Copy code
git clone <https://github.com/grapl-security/grapl>
git checkout cm/pants-2.9
./pants --tag="-needs_work" test --pytest-args="-m \"not integration_test\"" src/python/graplctl/tests:tests
🙌 1
h

hundreds-father-404

01/14/2022, 11:49 PM
Thank you! I'm about to log off for the long weekend, but will play with this on Tuesday. Have a great weekend!
b

brash-baker-91190

01/14/2022, 11:50 PM
Sounds good; thanks for your help! Have a good weekend yourself! 👋
❤️ 1
@hundreds-father-404 any more ideas about this, by chance?
h

hundreds-father-404

01/19/2022, 6:19 PM
Ah thanks for the ping. I don't - any ideas @enough-analyst-54434?
e

enough-analyst-54434

01/19/2022, 6:58 PM
You guys already found the idea: https://github.com/pantsbuild/pants/pull/13690 - that's exactly it. I comment on that PR how I hate such a long-winded explanation, but its required reading unfortunately. The book there talks about hand-rolled setup.py / pyproject.toml though and here grapl is using the auto-setup.py. I'm looking at that angle presently...
👍 1
b

brash-baker-91190

01/19/2022, 6:59 PM
Cool, thanks very much @enough-analyst-54434, @hundreds-father-404 🙇
e

enough-analyst-54434

01/19/2022, 7:20 PM
@brash-baker-91190 just for background, is depending on the src/python/grapl-common python_distribution instead of the src/python/grapl-common/grapl_common pythoon_sources intentional? This diff gets your test command above passing and eliminates the middle-man of packaging up grapl-common into an sdist / wheel:
Copy code
diff --git a/src/python/graplctl/graplctl/BUILD b/src/python/graplctl/graplctl/BUILD
index 1e8a0ce9..050a642b 100644
--- a/src/python/graplctl/graplctl/BUILD
+++ b/src/python/graplctl/graplctl/BUILD
@@ -1,14 +1,7 @@
 python_sources(
     name="graplctl_lib",
-    dependencies=[
-        "src/python/grapl-common:grapl-common",
-    ],
 )
 
 pex_binary(
     entry_point="cli.py:main",
-    # TODO: Once <https://github.com/pantsbuild/pants/pull/13690> lands,
-    # remove this work-around
-    # See also <https://pantsbuild.slack.com/archives/C046T6T9U/p1637597807103900>
-    dependencies=["!!src/python/grapl-common:grapl-common"],
 )
Although its surprising this case isn't working since you don't use a custom setup.py / pyproject.toml, it is a bit surprising you're going through the middle-man here instead of depending directly on the sources since that provides better fine-grained invalidation - a benefit of Pants + monorepo you're missing out on.
b

brash-baker-91190

01/19/2022, 7:28 PM
Hrmm... yeah, it looks like we could do that 😅 Oh man, it even says it in the comments 🤦
I'm putting out about 3 fires at the moment, but once things settle, I'll give this a try in our repo. Thanks!
e

enough-analyst-54434

01/19/2022, 7:31 PM
OK. I'm still looking into the issue sans workaround since maybe its conceivable someone might have a use case where they actually want to depend on the wheel generated from the sources instead of the sources? I can't come up with one off the top of my head, but the world is a big place.
b

brash-baker-91190

01/19/2022, 7:32 PM
Testing against the artifacts you'd ultimately end up shipping?
e

enough-analyst-54434

01/19/2022, 7:35 PM
I guess. That's effectively testing Pants works though in this case of auto generated setup.py.
b

brash-baker-91190

01/24/2022, 6:55 PM
Just an update here: I can now run my unit tests, but our integration tests are now complaining. I'm not 100% sure what's happening, but we have some Python services that are basically a PEX-in-a-container (not managed by Pants' Docker integration at the moment, FWIW) and they're not starting up in Nomad... just seeing errors like this:
/bin/bash: line 1: ./analyzer-executor.pex: Permission denied
I'm continuing to bisect things to figure out at what point in the 2.9 dev/rc line our system starts to fall over. Our test process is kind of long, though, so I'm just picking at this between other tasks (we're not blocked or anything; Pants 2.8 continues to work like a champ for us in this repository). Just wanted to keep y'all in the loop; I'll report back if/when I find anything.
e

enough-analyst-54434

01/24/2022, 7:12 PM
they're not starting up in Nomad... just seeing errors like this: 
/bin/bash: line 1: ./analyzer-executor.pex: Permission denied
That looks like it can only really be that the 755 bits added to
analyzer-executor.pex
by Pex when it creates the PEX have been wiped out. That shouldn't be related to any Pex or Pants upgrades. Are you sure there are no other variable in play for that error re perms / cp, etc?
b

brash-baker-91190

01/24/2022, 7:15 PM
It's possible... been a while since I've been in this bit of code, and there's been a lot of change in this area recently. We're probably doing something "non-standard" somewhere 😅
I'll do some digging and report back.
I finally got a chance to get back to this... it looks like in Pants 2.8, PEX files were created with
-rwxr-xr-x
permissions, but in 2.9, they get
-rwxr--r--
... in our container, our PEX files are owned by root, but run by a separate user. Easy enough fix in our containers. Is this an expected change in behavior, though?
e

enough-analyst-54434

01/27/2022, 3:54 PM
Pex itself still generates PEX files 755. This smells like a Pants behavior change. And it is:
Copy code
$ rm -rf dist && ./pants package src/python/pants/bin::
07:52:22.51 [INFO] Completed: Installing 3rdparty/python/lockfiles/user_reqs.txt
07:52:28.06 [INFO] Completed: Extracting 15 requirements to build src.python.pants.bin/pants.pex from lockfile.pex: PyYAML<7.0,>=6.0, ansicolors==1.1.8, fasteners==0.16.3, humbug==0.2.7, ijson==3.1.4, packaging==21.3, pex==2.1.65,... (168 characters truncated)
07:52:28.08 [INFO] Wrote dist/src.python.pants.bin/pants.pex
$ ls -l dist/src.python.pants.bin/pants.pex
-rwxr--r-- 1 jsirois jsirois 53595441 Jan 27 07:52 dist/src.python.pants.bin/pants.pex
That seems worth a bug @brash-baker-91190
b

brash-baker-91190

01/27/2022, 3:54 PM
@enough-analyst-54434 OK, I'll file an issue
e

enough-analyst-54434

01/27/2022, 3:58 PM
Thanks Chris.
b

brash-baker-91190

01/27/2022, 3:59 PM
I'll also see if I can narrow it down to a specific dev or rc release of 2.9
🙌 1
e

enough-analyst-54434

01/27/2022, 3:59 PM
Awesome. Thanks for digging.
b

brash-baker-91190

01/27/2022, 3:59 PM
Yeah, sure thing! 👍
Seems like Pants 2.9.0.dev4 is where this entered, but I don't see what change in that release might be responsible.
5 Views