https://pantsbuild.org/ logo
Join Slack
Powered by
Is anyone experiencing something similar due to so...
# general
l
Is anyone experiencing something similar due to some upgrade that released recently?
b
A few context questions: 1. What version of pants? 2. Do you use lock files? 3. Have you changed anything recently to go from working to not working?
l
1. 2.15.0 (broken) 2.16.0 (working) 2. Not pants a generated one with checksums etc, but we do use requirements.txt 3. Not relating to pants configuration even our requirements.txt
b
Okay, without a full lock file (that pins versions of all transitive dependencies), I wouldn’t be surprised about inconsistent failures like this: if the dependencies have new releases, the built artifacts might change behaviour. That said, I don’t know why that particular problem would surface. Can you look in the zip file and confirm what’s in it? You can see the list of dependencies in 
.deps
In addition, you might be interested in https://github.com/pantsbuild/pants/discussions/18756
l
The 
.deps
. Folder has the correct dependencies when unzipped. The issue is the .bootstrap/pex folder’s dependencies which I don't think I have control over are if I'm not mistaken are added by pants?
Which is why the version bump to 2.16.0 might fix it?
Also I will note this is happening for a working lambda deployment
b
Ah, I see there’s two different errors. Upgrading pants will indeed use a newer version of PEX, which might’ve fixed an error like that, if there was an issue under the hood
l
We’re working on determining the other issue, I will let you know what we find out. Thanks for your help
Follow up: The 1st error message was solved by upgrading pants to 
2.16.0
The 2nd error message was resolved by using a 
complete_platforms
file instead of the 
runtime
with 
python_awslambda
👍 1
l
@late-manchester-49683 I get exactly the first issue in pants 2.16.0. Did you get it once you upgraded?
l
@lemon-yak-80782 no I got it when I was using 2.15.0. It went away after I upgraded to 2.16.0
l
The weird thing is I only get it for some lambdas, but not for all. Despite the fact I build all lambdas the same…
The contents of the .bootstrap/pex folder are totally different. In one attr is a module. In the working lambdas it is a package.
l
I was also experiencing that.
I'm on my phone. Can you paste into this thread the 2 error I was experiencing?
But I did list the 2 things that I did. Upgrade to 2.16 and start using complete_platforms.
l
Copy code
Runtime.ImportModuleError: Unable to import module 'lambdex_handler': cannot import name 'AttrsInstance' from 'pex.vendor._vendored.attrs.attr' (/var/task/.bootstrap/pex/vendor/_vendored/attrs/attr/__init__.py)
Copy code
[10:17 PM] {
  "errorMessage": "Failed to resolve requirements from PEX environment @ /var/task.\nNeeded cp39-cp39-manylinux_2_26_x86_64 compatible dependencies for:\n 1: bcrypt>=3.2\n    Required by:\n      paramiko 3.2.0\n    But this pex had no ProjectName(raw='bcrypt', normalized='bcrypt') distributions.\n 2: cryptography>=3.3\n    Required by:\n      paramiko 3.2.0\n    But this pex had no ProjectName(raw='cryptography', normalized='cryptography') distributions.",
I only get the first error.
l
If you like I can paste into this thread my lambda macro and the complete_platforms file I'm using.
l
This would be great.
l
I will ping you in a couple of hours when I'm back in front of my comp.
l
I still would expect the contents of .bootstrap/pex to be exactly the same for every lambda.
@late-manchester-49683 No problem, thanks!
l
But yes, that error was solved by upgrading because I believe the bootstrap was changed in a non backwards compatible way during the 2.16 release
Also keep in mind that when you build, it will used cached builds unless the inputs change
If you build all your lambdas without a local cache, maybe they’ll all have the same issue.
l
Already deleted the cache once. I have check this tomorrow…
l
👍
@lemon-yak-80782 Here is an example of a custom macro I use for creating dockerized lambda functions https://gist.github.com/Sailias/8e9e8a575f78becfcc7834070cc571a6
l
Thanks. Sadly it did not fix my issue. I try to find out the difference between the working lambdas and the failing ones.
Some findings: The version of /var/task/.bootstrap/pex/vendor/_vendored/attrs/ is 19.2 The lambdas that throw the error also contain attrs version 23.1 in the .deps folder This should not be an issue, since in earlier working versions this was also the case. After going over the diff off the last working version and my current state I found the issue:
Copy code
@@ -1996,48 +2058,69 @@
             "typing-extensions<5.0,>=3.7"
           ],
           "requires_python": "~=3.7",
-          "version": "1.84.0"
+          "version": "1.85.0"
         },
         {
           "artifacts": [
             {
               "algorithm": "sha256",
-              "hash": "a870ad254da1a8ca84b6a2905cac29d265f805acc57af304784962a2aa6508f6",
-              "url": ""
+              "hash": "971be834317c22daaa9132340a51c01b50910724082c2c1a2ac87eeec153a3fe",
+              "url": "/jsonschema-4.18.4-py3-none-any.whl"
             },
             {
               "algorithm": "sha256",
-              "hash": "0f864437ab8b6076ba6707453ef8f98a6a0d512a80e93f8abdb676f737ecb60d",
-              "url": "/jsonschema-4.17.3.tar.gz"
+              "hash": "fb3642735399fa958c0d2aad7057901554596c63349f4f6b283c493cf692a25d",
+              "url": "/jsonschema-4.18.4.tar.gz"
             }
           ],
           "project_name": "jsonschema",
           "requires_dists": [
-            "attrs>=17.4.0",
+            "attrs>=22.2.0",
             "fqdn; extra == \"format\"",
             "fqdn; extra == \"format-nongpl\"",
             "idna; extra == \"format\"",
             "idna; extra == \"format-nongpl\"",
-            "importlib-metadata; python_version < \"3.8\"",
             "importlib-resources>=1.4.0; python_version < \"3.9\"",
             "isoduration; extra == \"format\"",
             "isoduration; extra == \"format-nongpl\"",
             "jsonpointer>1.13; extra == \"format\"",
             "jsonpointer>1.13; extra == \"format-nongpl\"",
+            "jsonschema-specifications>=2023.03.6",
             "pkgutil-resolve-name>=1.3.10; python_version < \"3.9\"",
-            "pyrsistent!=0.17.0,!=0.17.1,!=0.17.2,>=0.14.0",
+            "referencing>=0.28.4",
             "rfc3339-validator; extra == \"format\"",
             "rfc3339-validator; extra == \"format-nongpl\"",
             "rfc3986-validator>0.1.0; extra == \"format-nongpl\"",
             "rfc3987; extra == \"format\"",
-            "typing-extensions; python_version < \"3.8\"",
+            "rpds-py>=0.7.1",
             "uri-template; extra == \"format\"",
             "uri-template; extra == \"format-nongpl\"",
             "webcolors>=1.11; extra == \"format\"",
             "webcolors>=1.11; extra == \"format-nongpl\""
           ],
-          "requires_python": ">=3.7",
-          "version": "4.17.3"
+          "requires_python": ">=3.8",
+          "version": "4.18.4"
+        },
Fixing the 
jsonschema
version at 4.17.3 resolved the error for now. I am not sure why the local dependencies interfere with lambdex setup. Further, I am not even sure if this is actually the case. @enough-analyst-54434 is it worth to open an ticket in the lambdex repository? Sorry, if I tagged the wrong person, but you have the most contributions to lambdex.
e
@lemon-yak-80782 Pex has been on pex/vendor/_vendored/attrs/ @ ~21.5.0 since Pex 2.1.91 released on May 30th, 2022 (See https://github.com/pantsbuild/pex/pull/1780). This does not match up with the 19.2 you observe at all.
l
@enough-analyst-54434 thanks for your quick. Sorry, I checked _version_info.py, this does not at all contain the version of attrs. The bug I experience still exist, if I do not add 
jsonschema == 4.17.3
to my requirements file. Can the 3rd party dependencies interfere with lambdex/pex in any way?
e
The thread is ~huge so I stopped short. Let me read up...
@lemon-yak-80782 there should be no interference is the short answer. From my quick read of the above there is nowhere near enough detail to debug this. Can you provide more on the nose data? For example, the actual PEX file with trouble, how its deployed (lambda layer or zip), the runtime environment (3.8, 3.9, ...?) and the full AWS failure log from the console?
l
@enough-analyst-54434, sure! AWS lambda failure log:
Copy code
[ERROR] Runtime.ImportModuleError: Unable to import module 'lambdex_handler': cannot import name 'AttrsInstance' from 'pex.vendor._vendored.attrs.attr' (/var/task/.bootstrap/pex/vendor/_vendored/attrs/attr/__init__.py)
Traceback (most recent call last):
We run the .pex in a docker lambda. The runtime environment is python 3.10.
Copy code
FROM public.ecr.aws/lambda/python:3.10

RUN yum install unzip -y && rm -rf /var/cache/yum
COPY some-path/lambda.zip .
RUN unzip -q lambda.zip -d "${LAMBDA_TASK_ROOT}"
RUN rm -r lambda.zip
CMD ["lambdex_handler.handler"]
We build .zip with:
Copy code
python_awslambda(
    name="lambda",
    runtime="python3.10",
    handler="lambda.py:handler",
)
Deployment is done via AWS-CDK. I also added the .zip, with our source code removed. Is there anything else I can provide you?
lambda.zip
e
Thanks. So the traceback is empty?
That seems unlikely.
Ok. Thanks. Can you include the top of the log? There should be more 
pex: ...
lines above.
The total log would be awesome.
l
Sry, it is getting late 😕
log.txt
e
Thanks @lemon-yak-80782 - I was able to repro with this plus a bit of docker exec wrangling. The Pex machinery works fine and fully scrubs its internal vendored attrs before launching the underlying user code but the Lambdex does not. Although I'm poking at what goes wrong in Lambdex right now, I will point out that Lambdex is a dead man walking. In fact, in Pants 2.17 you can add 
layout="zip"
to your 
python_awslambda
target to get Lambdex and Pex completely out of the picture at runtime. Are you in a position to try that with Pants 2.18.0.dev5?
This does fix things for Lambdex back in Pex:
Copy code
bash-4.2# diff -u .bootstrap/pex/pex_bootstrapper.py .bootstrap/pex/pex_bootstrapper.py.orig
--- .bootstrap/pex/pex_bootstrapper.py  2023-07-22 16:52:07.720031735 +0000
+++ .bootstrap/pex/pex_bootstrapper.py.orig     2023-07-22 16:43:47.570107642 +0000
@@ -700,11 +700,3 @@
     from .environment import PEXEnvironment

     PEXEnvironment.mount(entry_point, pex_info).activate()
-
-    from . import third_party
-    from .bootstrap import Bootstrap
-    third_party.uninstall()
-    bootstrap = Bootstrap.locate()
-    bootstrap.demote()
-    import pex
-
Since that's an easy fix and does not require new work on Lambdex - which, as I mentioned, is a dead man walking that I prefer not to invest time in, I'll get that fix rolled in to the Pex 2.1.139 release. I can provide more info here as soon as that's released that will allow you to upgrade to that Pex version while keeping your Pants version fixed at whatever it is today. That said - definitely switch to the 
layout="zip"
@broad-processor-92400 introduced in Pants 2.17.x as soon as you can.
l
@enough-analyst-54434 any idea why the issue went away for me after upgrading to 2.16.0?
e
There is a bit much going on here - I expect you and @lemon-yak-80782 have issues that look similar squinting but are totally unrelated. Let me read the thread again with your details as the focus.
l
I don't believe I ended up providing many details aside for the error message and the fact it resolved after upgrading. I did notice the attrs version in bootstrap/pex was the older version that did not export AttrsInstance, but I couldn't change the version of that in my code obv. So I upgraded to 2.16 which seemed to use a newer version of attrs.
I assumed a dependency problem that might have upgraded pex but held onto an older version of attrs, which 2.16 corrected.
e
Ok. Well, Pex last changed its vendored attrs in the 2.1.91 release as I researched for @lemon-yak-80782's issue and that is much older than the Pex versions used by Pants 2.15.1 (Pex 2.1.111) and 2.16.0 (Pex 2.1.134).
So @late-manchester-49683 I'm really not sure at all what is going on with your observed behavior. FWICT your error should have the same cause as @lemon-yak-80782’s and it should not have gone away in Pants 2.16.0.
But is it true @late-manchester-49683 that you're OK / you're just curious at this point?
l
Yes that's true. My builds are working again and I'm just following up.
I'm even more curious now as to why they are working. I did also add the interpreter to my pants file.
e
Ok. Well, the fix I mention above will go out in Pex 2.1.139; so you can use the same workaround of upping your configured Pex version when I post the magic 
pants.toml
setup here later if the issue comes back for you. But the same advice I offered @lemon-yak-80782 applies - switch to 
layout="zip"
as soon as you're on a Pants version that supports that. That kills Lambdex and gets you on a vanilla / as-reccommended by AWS zip.
l
Maybe this helped with dependency resolution?
Copy code
[python]
interpreter_constraints = ["CPython==3.9.*"]
As we are using 3.9 and not 3.10
e
No. At the point you get that error message the Pex has bootstrapped and found all the deps it needs to activate including your user-dependency on more-modern attrs than the Pex vendored attrs. If the ICs were wrong, it would fail to find matching platform-specific wheels earlier than that and never get to the wrong version of attrs issue you saw. Presumably you have at least one of those (cp39 or cp310 in wheel name which you can see quickly with 
zipinfo -1 my.pex | grep .deps/
).
l
@enough-analyst-54434 Thank you for addressing the issue in such detail! I would be able to update to 2.18.0.dev5. Currently, I get an error in my python_aws_lambda_function goal:
Copy code
Unrecognized field `layout=zip` in target src/python/mdl-history/mdl_history/lambda_handler:data-processor-lambda. Valid fields for the target type `python_aws_lambda_function`: ['complete_platforms', 'dependencies', 'description', 'environment', 'handler', 'include_requirements', 'output_path', 'resolve', 'runtime', 'tags'].
Anyway, my preferred way is to update the Pex version (when it is released) in my pants.toml until 2.17 is released as stable. Then I will switch to 
layout="zip"
. I don’t want to risk running our build tool on a potentially unstable version if I don’t have to.
e
Aha - yes. I was a lazy reader. So this is actually available in 2.17+ and its not set per-target but globally with:
Copy code
[lambdex]
layout = "zip"
See here: https://www.pantsbuild.org/v2.17/docs/reference-lambdex#layout As for the Pex fix, you can use that now by adding the following to `pants.toml`:
Copy code
[pex-cli]
version = "v2.1.139"
known_versions = [
    "v2.1.139|macos_arm64|56761f3feb10a632ed82f590ad1fd49f3bdeb6785a48956abe6e5da9488a5305|4103625",
    "v2.1.139|macos_x86_64|56761f3feb10a632ed82f590ad1fd49f3bdeb6785a48956abe6e5da9488a5305|4103625",
    "v2.1.139|linux_x86_64|56761f3feb10a632ed82f590ad1fd49f3bdeb6785a48956abe6e5da9488a5305|4103625",
    "v2.1.139|linux_arm64|56761f3feb10a632ed82f590ad1fd49f3bdeb6785a48956abe6e5da9488a5305|4103625"
]
l
@enough-analyst-54434 Thanks! Adding the 
pex-cli
to the 
pants.toml
fixed it for us. I am gonna try out the 
layout="zip"
approach if I find the time.
b
Hey, @enough-analyst-54434! I’m facing the same issue, even after setting the 
pex-cli
to v2.1.139. The error is:
Copy code
ImportError: cannot import name 'AttrsInstance' from 'pex.vendor._vendored.attrs.attr'
which happens when a transitive dependency tries to import 
AttrsInstance
from the 
attr
module. debugging: I increased the verbosity to confirm that the vendored attr was, at least according to the logs, un-imported
Copy code
pex: un-imported <module 'pex.vendor._vendored.attrs.attr' from '/.pex/unzipped_pexes/51625a6fbd01b68370a16c38cd520f842deca7a0/.bootstrap/pex/vendor/_vendored/attrs/attr/__init__.py'>
I’ve also checked that PYTHONPATH includes the 
attrs
package (non-vendored) before the 
.bootstrap
folder. The weird thing, is that the correct 
attrs
module is properly imported, but the 
attr
(which should be packaged together ) is pointing to 
pex.vendor._vendored.attrs.attr
here we can notice that indeed the things imported from the 
attr
namespace seem to be coming from the vendored 
attr
. Any suggestions on how to debug this further?
e
@lemon-yak-80782 was able to provide me with a lambda.zip, which allowed a repro. @billowy-tiger-59247 if you can do that that would be extremely helpful.
Basically this here: https://pantsbuild.slack.com/archives/C046T6T9U/p1689972675730709?thread_ts=1688761502.867459&amp;cid=C046T6T9U That's useful debug info he provided. Similar would be great.
@billowy-tiger-59247 if that's too hard / restricted by proprietary concerns too much and your diagnosis is correct - the transitive import bit, can you provide: + The version of unvendored attrs in question + The project and version that does the indirect import?
b
yes, I’ll try to create a reproducible example and send you here! I’m not using the 
python_awslambda
target, though. I’m using the 
pex_binary
directly • the unvendored attrs package version is 
23.1.0
• the import order is: ◦ 
sagemaker==2.182.0
imports 
jsonschema
◦ 
jsonschema==4.19.0
imports 
attrs
◦ 
attrs==23.1.0
imports 
attr
e
Ok, thanks - that may be enough. I'll experiment a little later today and report back.
b
thanks a lot! Don’t know if this will add to the investigation but, weirdly enough, this started happening today, and might be correlated with an attempt to upgrade pants from 2.16 to 2.17. Even after going back to 2.16 and setting the 
pex-bin
version to 2.1.139, the issue still appears
e
And, one last thing - what Python is in use CPython 3.10 or ... ?
b
we have a constraint of 
CPython==3.8.*
e
Ok
@billowy-tiger-59247 as a super quick workaround, you probably want to be setting 
execution_mode="venv"
anyway. I assume you're not.
✅ 1
Wow, sagemaker is a _beast_:
Copy code
$ pex --python python3.8 sagemaker==2.182.0 jsonschema==4.19.0 attrs==23.1.0 -o test.pex
$
$ # Noop:
$ time ./test.pex -c ''

real    0m0.905s
user    0m0.807s
sys     0m0.043s
$
$ # Just import sagemaker:
$ time ./test.pex -c 'import sagemaker'

real    0m13.468s
user    0m1.637s
sys     0m0.978s
b
yes, it takes quite some time to import haha
e
Do you happen to know the entry point or call stack that leads to the attrs import error?
Clearly just importing sagemaker doesn't trigger it.
b
let me get the full traceback here
https://gist.github.com/lecardozo/3b72f89539861e08e0af2054fb6e7f5f the first import that we do is 
from sagemaker.s3 import S3Uploader
e
I don't repro:
Copy code
$ pex -V
2.1.145
$ time ./test.pex -c 'from sagemaker.s3 import S3Uploader'

real    0m13.531s
user    0m1.681s
sys     0m1.015s
b
Yeah, I tried to reproduce it here in a simplified example, but could not reproduce as well 🤔
e
Ok, well, 2 things: 1. Since you're running from a PEX directly, the 2.1.139 fix does not apply / that only fixed alternate ways of running ~Pex like things (Lambdex is one of 2 such known cases) to work like running a PEX ~always has. 2. I'll need more repro data. Your theory about indirect import doesn't seem to cover all the nuance of your failure since it doesn't repro it.
l
@billowy-tiger-59247 On what platform are you building the .pex before uploading it to aws lambda?
e
But execution_mode venv would be an awesome sanity check. That should make the error go away since, at that point, the Pex runtime is out of the picture via a re-exec into the venv - which will contain 0 pex code.
@lemon-yak-80782 I don't think his case is a lambda at all.
He just pattern-matched the error to this thread.
l
@enough-analyst-54434 Ah, I misunderstood this.
e
I did too at first!
b
yes, sorry if that was misleading! I’m indeed executing the PEX directly on the our environment (mac arm64). The 
execution_mode=venv
is doing the trick for us, so I’ll change it while I gather more data to reproduce the issue.
l
@billowy-tiger-59247 Does the issue persist on a x86 architecture? I also have a M1 and noticed some issues from time to time.
e
Ok, great. @billowy-tiger-59247 if you don't mind filing a https://github.com/pantsbuild/pex issue with that data, that would probably be best. My work schedule is very erratic and blocky; so that helps / is a better way of async work.
Yeah, so:
Copy code
$ ./test.pex -c 'import sys; print("\n".join(mod for mod in sys.modules if mod.startswith("pex")))'
pex.version
pex
pex.typing
pex.compatibility
IOW, after the pex boots, those are the only 
pex*
modules. I don't see 
pex.vendor._vendored.attrs.attr
let alone 
pex.vendor
even.
@billowy-tiger-59247 what does your 
pex_binary
target look like? Can you paste that? Maybe better paste 
unzip -qc your.pex PEX-INFO | jq .
.
b
@lemon-yak-80782 haven’t tested on other platforms yet, but will do! @enough-analyst-54434 here is the output of the 
unzip -qc your.pex PEX-INFO | jq .
. I’ll add a breakpoint before the sagemaker import to see what are the 
pex*
modules available before the error
Copy code
{
  "bootstrap_hash": "a064045603ba298561942a56a3da3747949fa93f",
  "build_properties": {
    "pex_version": "2.1.139"
  },
  "code_hash": "a4a097655fef575bb0d71c9e0b2e1fefe64ebd52",
  "distributions": {
    "GitPython-3.1.34-py3-none-any.whl": "f69e1673d69bb3dd1358c0b66c6b65feb2a1366ad3316893517c0e03056d258e",
    "Jinja2-2.11.3-py2.py3-none-any.whl": "8a18e3284e3369148048fa9af2ae04b61c512038a56a4f34883c81cfc00f221a",
    "MarkupSafe-2.0.1-cp38-cp38-macosx_10_9_universal2.whl": "f160fc1d89b834c3c8284b88944fa1a7b6369644857dafaadf8cacbe2869647f",
    "PyYAML-6.0.1-cp38-cp38-macosx_13_0_arm64.whl": "b6c511fbc584cb75c53aaf7196f25455e3790735e177576ae4e80bd535e41fd9",
    "Pygments-2.16.1-py3-none-any.whl": "9a5ace6bc85edf3d79b456f7caa713ad39bba73b03f8b0da03e004dd22cfe624",
    "anyio-4.0.0-py3-none-any.whl": "925cf2cab75f7910d3ca514d28e203258692c39646ddb96a348f963725ecbdb1",
    "arrow-1.2.3-py3-none-any.whl": "093fc3cd56b645c0f4ef57c194d2222b27aa5147efb68ef6ccc14c7b6e571529",
    "attrs-23.1.0-py3-none-any.whl": "6c3861638ec4fd267653f203e53754b70438c2d0696a8955f9eb6345b1c3bb48",
    "awswrangler-3.3.0-py3-none-any.whl": "21d6cac3c5502eae00ae7a4d69184e5688e56a11d16e40a9be655e574ecbe34c",
    "binaryornot-0.4.4-py2.py3-none-any.whl": "7798e44237367e15ebfe0d25ca43ec0453a10290b401678e1beb4c242dabe1f4",
    "boto3-1.28.40-py3-none-any.whl": "c582724acddfdf2794cd41e8bd392f2e3a396cdcef2c29c7062a98786691b3b5",
    "botocore-1.31.40-py3-none-any.whl": "c186a150f4a1e28e8a98b81c17c5b1a88bf510f4f4022ff88e284aef6448fa9e",
    "certifi-2023.7.22-py3-none-any.whl": "41972ef3880e7aeb1cf9f271e4f80f9df401f5ceb5f585fb3c2ee8b6500b4788",
    "chardet-5.2.0-py3-none-any.whl": "3444f1caec835239744916f414f002a736e1e050efc52c1352ed87dd25592e60",
    "charset_normalizer-3.2.0-cp38-cp38-macosx_11_0_arm64.whl": "94f8f44c9787603f156c34b42c76b9de9f8ad0820728f756085726532da5615d",
    "click-8.1.7-py3-none-any.whl": "e0ff3997898ca5b6bfc3d8c11d1daff31c89eec22f38ceab6fc6f7d41c898d5d",
    "cloudpickle-2.2.1-py3-none-any.whl": "5e507b687a53f60ec719f5beecb87c447721de298cbd1ee0e34de2c94b72b03d",
    "contextlib2-21.6.0-py2.py3-none-any.whl": "022eb42df50d1724cdf14142be24650c8c0eee54f81d72fd3fd60f46830de93d",
    "cookiecutter-2.3.0-py3-none-any.whl": "14dd801060060dfb3dcccba6639657ac0af4f0c08c616667401d33606febb9f3",
    "databricks_sdk-0.8.0-py3-none-any.whl": "0531e75f59f737c344617a2ec64dd50f5a36c0711f49ad784575b80e976a2de2",
    "dill-0.3.7-py3-none-any.whl": "afaaddaf810630ef34227c2af6f927734691ebe91158f82c90ee1dd17c43b60e",
    "exceptiongroup-1.1.3-py3-none-any.whl": "843e07b90a2d689c6034c0ecd0bae83bd61bddc53a2fd25646cfc8f7c3f762b2",
    "gitdb-4.0.10-py3-none-any.whl": "dc2aa23fc023a7c1a29edd7b1d3c37078a388d3c6781a7f11896894f81e6939c",
    "google_pasta-0.2.0-py3-none-any.whl": "aae70d694fda15d7a6b20beb89c904c4ddbcb28d756f0db133c4e4d94727baea",
    "idna-3.4-py3-none-any.whl": "3baea0ffbdea9f349783bacbdb82dbd50eadfec75339388aaf1597754786e3ed",
    "importlib_metadata-6.8.0-py3-none-any.whl": "61e6097ed68695b9d7d77e89b8dcab92163d0c6d3a84a11e50d5deaa79b1632a",
    "importlib_resources-6.0.1-py3-none-any.whl": "dbd5cae79fd4ab8126a895b72ed2adfb19badca335be88a8f7e5de45aee4a3bf",
    "jmespath-1.0.1-py3-none-any.whl": "299c3a18595a39d4d54252e98a3c39343899b9bd1997e3eebfee63ea5757588b",
    "jsonschema-4.19.0-py3-none-any.whl": "48881fb45d271ae04c9540764c96525deb809bc05d49eea0b9518a0a08767aac",
    "jsonschema_specifications-2023.7.1-py3-none-any.whl": "19c57dbff23ee1e94b00322dc033a6e825535c7ad1115bddb19fe18b1404cdaf",
    "markdown_it_py-3.0.0-py3-none-any.whl": "5283bd369e168609df0b6af247b2b8e724a9a3d6d9812481b2eca026608d530b",
    "mdurl-0.1.2-py3-none-any.whl": "43d69334b85db19da8afaf9583f4b6033d1874e9b7419b6aa4cac68292f061f2",
    "multiprocess-0.70.15-py38-none-any.whl": "d58349bfbe6d500eba85b3cb815b44f7d7c3b200ca8decfe4d3d3dbb49367657",
    "numpy-1.23.5-cp38-cp38-macosx_11_0_arm64.whl": "606be144a2d9de59d4d6399ab648bef17dbff8dcfff7d395def25e9d868f1956",
    "packaging-23.1-py3-none-any.whl": "8b3aa0966620e42b8b5bddab8e11f6f65ba78f4d885786d7e2d50efbdfcda19e",
    "pandas-1.5.3-cp38-cp38-macosx_11_0_arm64.whl": "07dd2201bd4430682f69cae46c46a177233a57ba3ca360c2216166ac43af5727",
    "pathos-0.3.1-py3-none-any.whl": "48aeba80b7bac20ced27bed0108204852b6ecdf56e284cf6ebc30344ee707c8e",
    "pkgutil_resolve_name-1.3.10-py3-none-any.whl": "0121081f2eebae9a497df26f92610864ab4875a38beb3d5015cab3ecad39e9b2",
    "platformdirs-3.10.0-py3-none-any.whl": "5b27ddc6aa62c130ff012216374056073e011b56dc9569ce38bbc5fcb2acc1c0",
    "pox-0.3.3-py3-none-any.whl": "4c0f203126015c5cb4d34961b19e59660880ce00451e2199ef73c0253f84e503",
    "ppft-1.7.6.7-py3-none-any.whl": "30d11fd64aa37bccc552487fb5ec612a7358a435ad3aa290550daae20097ff7d",
    "protobuf-3.20.3-py2.py3-none-any.whl": "9a36cf1764b80d3b534a8829ebfd9f9e40be087c417fff94a63f6dd38edc1992",
    "pyarrow-13.0.0-cp38-cp38-macosx_11_0_arm64.whl": "2f1dacd91066d9e0d312f0b989a1296164f36f0997863496d3777b8879582cf9",
    "python_dateutil-2.8.2-py2.py3-none-any.whl": "202c27a293331dd8fa9d41d1fcdd5ba4f4d6d2de0a2f00fa8547adc7c1aac629",
    "python_slugify-8.0.1-py2.py3-none-any.whl": "70d881acc57e232e80027924886fe94a1bc980bb3f5635748b89b35fbd9ae044",
    "pytz-2023.3.post1-py2.py3-none-any.whl": "d1961115ba2d0eab86256042387b168cd0c4e9ab9aa56fcbdd07ac22250a1a5f",
    "referencing-0.30.2-py3-none-any.whl": "21994a35d4e452eee5e42e20ad387b2d346cb997d540cb11aa7aa245637b9107",
    "requests-2.31.0-py3-none-any.whl": "d1478462520ba69e97c257949e5c5b4552eac42920d4643a8625610aa1c4b999",
    "rich-13.5.2-py3-none-any.whl": "6917b84a6269c2c098705bc2cec4c3df9dfbbe12dfbaafe92a1127789203cc96",
    "rpds_py-0.10.2-cp38-cp38-macosx_11_0_arm64.whl": "81072a34c2da7c5da0b788b2b673e0c81851167e70f330e3963ff8ecd60562be",
    "s3transfer-0.6.2-py3-none-any.whl": "11074ffe5b076fb3d7570d90448e7daabf3faa05d29f38132b04f6352299d72a",
    "sagemaker-2.182.0-py2.py3-none-any.whl": "a2c15e882188ab4817db2a1761315c2a4f88062ded46a01034c1e53e8dd9e3ff",
    "schema-0.7.5-py2.py3-none-any.whl": "7bdc0e92742022ca7270178fc8a2914f5dba5aeb769b037ca67deb461af106e3",
    "six-1.16.0-py2.py3-none-any.whl": "3e1c439c88d2e7681372427bab751b3fc99969891e95a714fed9604bf7710213",
    "smdebug_rulesconfig-1.0.1-py2.py3-none-any.whl": "67baf8ca0032bf2a09d4008103fec198d870360cdfc0febeb5d31dc480ce2eb1",
    "smmap-5.0.0-py3-none-any.whl": "5fcddf5ee09f054ef4f8f6d903cf356d3bb93b3b1dec599f2eac198bec9ed1cb",
    "sniffio-1.3.0-py3-none-any.whl": "feffd920899e99d56efbafd1469bc06ccab4337c224960b4ca43f5c937f56a6a",
    "tblib-1.7.0-py2.py3-none-any.whl": "fde3148d690c9d474ef027b654ec90562a4fcdd20813a2d8d6f22fbe671879c8",
    "text_unidecode-1.3-py2.py3-none-any.whl": "b927a485a7c5ea3c959180514bb6c1a43d44e949292bdbe9865e37fdaed35c02",
    "typer-0.9.0-py3-none-any.whl": "d769c5e0cf70922639e8fcbf546959f9a4ab5d914d716ce117b14175968b546c",
    "typing_extensions-4.7.1-py3-none-any.whl": "5e6df9210d60789205e617f60bb181074200092ebed2796a63358a9c9f3f1c75",
    "urllib3-1.26.16-py2.py3-none-any.whl": "55ed1adea278529a458f440b5ed6ce3b7be6dac8041ee06ba2c06e83821a33d9",
    "watchfiles-0.20.0-cp37-abi3-macosx_11_0_arm64.whl": "d44879bf8e2c3294cb2dd1f15dbf1982c89477764119a724f75d934f937bafc5",
    "zipp-3.16.2-py3-none-any.whl": "f4d6d3db4feedf71e8e6fbdd30c0f51c59a63902f104c9f2fdf5d757cbada595"
  },
  "emit_warnings": false,
  "entry_point": "cli.main",
  "ignore_errors": false,
  "includes_tools": false,
  "inherit_path": "false",
  "inject_args": [],
  "inject_env": {},
  "interpreter_constraints": [
    "CPython==3.8.*"
  ],
  "pex_hash": "51625a6fbd01b68370a16c38cd520f842deca7a0",
  "pex_path": "",
  "pex_paths": [],
  "requirements": [
    "GitPython>=3.0.0",
    "PyYAML>=5.4.1",
    "awswrangler",
    "cookiecutter",
    "databricks-sdk",
    "pandas<2.0,>=1.4.0",
    "python-slugify",
    "rich",
    "sagemaker",
    "typer<1.0.0,>=0.3.2",
    "watchfiles"
  ],
  "strip_pex_env": true,
  "venv": false,
  "venv_bin_path": "false",
  "venv_copies": false,
  "venv_hermetic_scripts": true,
  "venv_site_packages_copies": false
}
Yeah, indeed something weird. Here are the 
pex*
packages on the path
Copy code
pex.version
pex
pex.typing
pex.common
pex.pex_warnings
pex.enum
pex.inherit_path
pex.compatibility
pex.orderedset
pex.venv
pex.venv.bin_path
pex.variables
pex.tracer
pex.vendor
pex.vendor._vendored
pex.vendor._vendored.attrs
pex.vendor._vendored.attrs.attr._compat
pex.vendor._vendored.attrs.attr._config
pex.vendor._vendored.attrs.attr.exceptions
pex.vendor._vendored.attrs.attr.setters
pex.vendor._vendored.attrs.attr._make
pex.vendor._vendored.attrs.attr.converters
pex.vendor._vendored.attrs.attr.filters
pex.vendor._vendored.attrs.attr.validators
pex.vendor._vendored.attrs.attr._cmp
pex.vendor._vendored.attrs.attr._funcs
pex.vendor._vendored.attrs.attr._version_info
pex.vendor._vendored.attrs.attr._next_gen
pex.vendor._vendored.attrs.attr
e
Ok, I'll try a repro with an entrypoint ~like yours. Perhaps that is the difference.
Hrm, no dice:
Copy code
$ cat requirements.txt
GitPython>=3.0.0
PyYAML>=5.4.1
awswrangler
cookiecutter
databricks-sdk
pandas<2.0,>=1.4.0
python-slugify
rich
sagemaker
typer<1.0.0,>=0.3.2
watchfiles
$ cat src/cli/main.py
import sys

from sagemaker.s3 import S3Uploader


if __name__ == "__main__":
    print("\n".join(mod for mod in sys.modules if mod.startswith("pex")))

$ pex --python python3.8 -r requirements.txt -Dsrc -e cli.main -o example.pex
$ ./example.pex
pex.version
pex
b
yeah, weird. I’ll try to clean up my environment to see if there’s any weird unexplainable thing happening. I’m also using locked dependencies. Do you think that the specific combination of versions of the whole locked resolve could somehow lead to this issue?
e
Perhaps. Is this a Pex lockfile (in other words, did you generate the lock using Pants)? If so, can you provide that?
I'm on Linux; so the only other difference would be in native code. The PEX-INFO you dumped does indicate several native wheels. That said - the native wheel for Mac for a given project is highly unlikely to contain different plain Python code than a native wheel for Linux. Almost always the only difference is in the embedded native shared library binary files.
Ok, and for completeness:
Copy code
$ pex --python python3.8 pex==2.1.139 -cpex --  --python python3.8 -r requirements.txt -Dsrc -e cli.main -o example.pex
$ ./example.pex
pex.version
pex
That gets us on the same Pex version.
b
Yes, the lockfile was generated with Pants!
default_lock.txt
tried restarting the computer and deleting all pants caches (that I know of), and I’m still having the issue without the 
execution_mode=venv
e
Ok ... that gives a repro:
Copy code
$ pex --python python3.8 pex==2.1.139 -cpex -- --python python3.8 -r requirements.txt --lock default_lock.txt -Dsrc -e cli.main -o example.pex
$ ./example.pex
/home/jsirois/.pex/unzipped_pexes/830409e07a10ffa3289ab6c001769a72362c7d02/.bootstrap/pex/environment.py:673: PEXWarning: The `pkg_resources` package was loaded from a pex vendored version when declaring namespace packages defined by:

1. protobuf==3.20.3 namespace packages:
  google

These distributions should fix their `install_requires` to include `setuptools`
  pex_warnings.warn(
Traceback (most recent call last):
  File "/home/jsirois/.pyenv/versions/3.8.18/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/home/jsirois/.pyenv/versions/3.8.18/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/home/jsirois/.pex/unzipped_pexes/830409e07a10ffa3289ab6c001769a72362c7d02/__main__.py", line 108, in <module>
    bootstrap_pex(__entry_point__, execute=__execute__, venv_dir=__venv_dir__)
  File "/home/jsirois/.pex/unzipped_pexes/830409e07a10ffa3289ab6c001769a72362c7d02/.bootstrap/pex/pex_bootstrapper.py", line 618, in bootstrap_pex
    pex.PEX(entry_point).execute()
  File "/home/jsirois/.pex/unzipped_pexes/830409e07a10ffa3289ab6c001769a72362c7d02/.bootstrap/pex/pex.py", line 560, in execute
    sys.exit(self._wrap_coverage(self._wrap_profiling, self._execute))
  File "/home/jsirois/.pex/unzipped_pexes/830409e07a10ffa3289ab6c001769a72362c7d02/.bootstrap/pex/pex.py", line 467, in _wrap_coverage
    return runner(*args)
  File "/home/jsirois/.pex/unzipped_pexes/830409e07a10ffa3289ab6c001769a72362c7d02/.bootstrap/pex/pex.py", line 498, in _wrap_profiling
    return runner(*args)
  File "/home/jsirois/.pex/unzipped_pexes/830409e07a10ffa3289ab6c001769a72362c7d02/.bootstrap/pex/pex.py", line 603, in _execute
    return self.execute_entry(
  File "/home/jsirois/.pex/unzipped_pexes/830409e07a10ffa3289ab6c001769a72362c7d02/.bootstrap/pex/pex.py", line 771, in execute_entry
    return self.execute_module(entry_point.module)
  File "/home/jsirois/.pex/unzipped_pexes/830409e07a10ffa3289ab6c001769a72362c7d02/.bootstrap/pex/pex.py", line 779, in execute_module
    runpy.run_module(module_name, run_name="__main__", alter_sys=True)
  File "/home/jsirois/.pyenv/versions/3.8.18/lib/python3.8/runpy.py", line 207, in run_module
    return _run_module_code(code, init_globals, run_name, mod_spec)
  File "/home/jsirois/.pyenv/versions/3.8.18/lib/python3.8/runpy.py", line 97, in _run_module_code
    _run_code(code, mod_globals, init_globals,
  File "/home/jsirois/.pyenv/versions/3.8.18/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/home/jsirois/.pex/unzipped_pexes/830409e07a10ffa3289ab6c001769a72362c7d02/cli/main.py", line 3, in <module>
    from sagemaker.s3 import S3Uploader
  File "/home/jsirois/.pex/installed_wheels/9c644caf1caa8bad76c4c8d6a051fa37948726d637ce406eafdc2f33d40928c5/sagemaker-2.182.0-py2.py3-none-any.whl/sagemaker/__init__.py", line 18, in <module>
    from sagemaker import estimator, parameter, tuner  # noqa: F401
  File "/home/jsirois/.pex/installed_wheels/9c644caf1caa8bad76c4c8d6a051fa37948726d637ce406eafdc2f33d40928c5/sagemaker-2.182.0-py2.py3-none-any.whl/sagemaker/estimator.py", line 30, in <module>
    from sagemaker import git_utils, image_uris, vpc_utils, s3
  File "/home/jsirois/.pex/installed_wheels/9c644caf1caa8bad76c4c8d6a051fa37948726d637ce406eafdc2f33d40928c5/sagemaker-2.182.0-py2.py3-none-any.whl/sagemaker/image_uris.py", line 23, in <module>
    from sagemaker import utils
  File "/home/jsirois/.pex/installed_wheels/9c644caf1caa8bad76c4c8d6a051fa37948726d637ce406eafdc2f33d40928c5/sagemaker-2.182.0-py2.py3-none-any.whl/sagemaker/utils.py", line 40, in <module>
    from sagemaker.config import validate_sagemaker_config
  File "/home/jsirois/.pex/installed_wheels/9c644caf1caa8bad76c4c8d6a051fa37948726d637ce406eafdc2f33d40928c5/sagemaker-2.182.0-py2.py3-none-any.whl/sagemaker/config/__init__.py", line 16, in <module>
    from sagemaker.config.config import load_sagemaker_config, validate_sagemaker_config  # noqa: F401
  File "/home/jsirois/.pex/installed_wheels/9c644caf1caa8bad76c4c8d6a051fa37948726d637ce406eafdc2f33d40928c5/sagemaker-2.182.0-py2.py3-none-any.whl/sagemaker/config/config.py", line 26, in <module>
    import jsonschema
  File "/home/jsirois/.pex/installed_wheels/043dc26a3845ff09d20e4420d6012a9c91c9aa8999fa184e7efcfeccb41e32cb/jsonschema-4.19.0-py3-none-any.whl/jsonschema/__init__.py", line 13, in <module>
    from jsonschema._format import FormatChecker
  File "/home/jsirois/.pex/installed_wheels/043dc26a3845ff09d20e4420d6012a9c91c9aa8999fa184e7efcfeccb41e32cb/jsonschema-4.19.0-py3-none-any.whl/jsonschema/_format.py", line 11, in <module>
    from jsonschema.exceptions import FormatError
  File "/home/jsirois/.pex/installed_wheels/043dc26a3845ff09d20e4420d6012a9c91c9aa8999fa184e7efcfeccb41e32cb/jsonschema-4.19.0-py3-none-any.whl/jsonschema/exceptions.py", line 14, in <module>
    from attrs import define
  File "/home/jsirois/.pex/installed_wheels/1f28b4522cdc2fb4256ac1a020c78acf9cba2c6b461ccd2c126f3aa8e8335d04/attrs-23.1.0-py3-none-any.whl/attrs/__init__.py", line 3, in <module>
    from attr import (
ImportError: cannot import name 'AttrsInstance' from 'pex.vendor._vendored.attrs.attr' (/home/jsirois/.pex/unzipped_pexes/830409e07a10ffa3289ab6c001769a72362c7d02/.bootstrap/pex/vendor/_vendored/attrs/attr/__init__.py)
And the warning printed at the top is the proximal cause I believe. I'll dig a bit more, file an issue and link it here. Thanks for the details @billowy-tiger-59247.
Ok, and to put a fine point on it, the unlocked pex gets protouf 4.x which appears to fix the problem:
Copy code
$ pex --python python3.8 pex==2.1.139 -cpex -- --python python3.8 -r requirements.txt -Dsrc -e cli.main -o example.no-lock.pex
$ ./example.no-lock.pex
pex.version
pex
$ unzip -qc example.no-lock.pex PEX-INFO | jq -r '.distributions | keys[]' | grep protobuf
protobuf-4.24.2-cp37-abi3-manylinux2014_x86_64.whl
Yeah, so @billowy-tiger-59247 another workaround is to help old protobuf out by manually declaring its missing dependency on setuptools. Here I add a 
setuptools
requirement in addition to your requirements in requiremets.txt. The result is both no warning and no failure:
Copy code
$ pex --python python3.8 pex==2.1.139 -cpex --  --python python3.8 -r requirements.txt setuptools --lock default_lock.txt -Dsrc -e cli.main -o example.plus-setuptools.pex
$ ./example.plus-setuptools.pex
pex.version
pex
I'm not sure there will be anything I can do about this! If Pex is forced to supply setuptools the vendoring system can't be fully un-imported. The only fix I can imagine is a new flag to Pex that says fail instead of warn. That would force you to fix the 3rdparty library problem by adding your own top-level setuptools requirement like I just did.
For further color, the 
pkg_resources
from setuptools is now gone in modern setuptools since the modern Python stdlib provides replacements for its functionality in 
importlib.*
code in addition to PEP-420 implicit namespace packages taking away the need for `pkg_resources.declare_namespace(...)`: https://packaging.python.org/en/latest/guides/packaging-namespace-packages/#pkg-resources-style-namespace-packages
b
oooh so that’s it! Thanks for the investigation, John. I gotta say though that it is not immediately clear the relationship between the `setuptools`/`protobuf` issue and the 
attr
import issue. In my understanding, protobuf requires 
pkg_resources
from 
setuptools
but doesn’t declare that as a requirement, so we end up importing that from a pex vendored setuptools instead. Does this import from vendored 
setuptools
have a side effect of including other vendored packages in the sys.path and then we end up with the 
attr
issue?
e
Yes, exactly. The vendoring system uses attrs, amongst other vendored packages. The attrs package, though, is special. It cannot work with its vendored self when it's making a data class instance. In that process it imports itself from 
attr
As a result that is the sole Pex vendored package exposed both under the vendored namespace, which is safe, and the unvendored attr namespace, which is not. In all cases except bad setuptools behavior, Pex still can manage to tidy up this unsafety.
Open in Slack
PreviousNext
Powered by