https://pantsbuild.org/ logo
Join Slack
Powered by
I'm having a weird issue with lockfile generation ...
# general
r
I'm having a weird issue with lockfile generation and verification w.r.t. 
python_requirement
. We have some requirements on 
boto===2.49.0a1
. I believe this is an internal boto fork that only exists in our private artifactory. However, lockfile generation turns this into 
boto===2.49a1
which then fails all the 
python_requirement
constraints. Changing the lockfile manually to 
boto===2.49.0a1
doesn't fix it either. Is the version number ill-formed or something?
h
Hi, is this using the Poetry lockfile generator or Pex generator?
r
Pex
It seems like an issue with Pex itself. Is there a good way to debug Pex? I tried unzipping the sandbox ./pex and PDBing into it but it eventually fails with an import resolution failure.
Basically a way to run
Copy code
/Users/navneeth.jayendran/.pyenv/versions/3.7.13/bin/python3.7 .pex --tmpdir .tmp --jobs 2 --python-path $'/Users/navneeth.jayendran/.pyenv/versions/2.7.14/bin:/Users/navneeth.jayendran/.pyenv/versions/3.7.13/bin' --output-file requirements.pex --no-emit-warnings --manylinux manylinux2014 --python /Users/navneeth.jayendran/.pyenv/versions/3.7.13/bin/python3.7 $'--sources-directory=source_files' $'boto3>1.17.0' --layout packed
but using a built-from source Pex or something
Okay I started debugging PEX directly from source with 
pex --lock mylockfile
. I've noticed that the normalization that PEX gives seems to be the root cause.
Copy code
LockedRequirement(pin=Pin(project_name=ProjectName(raw='boto', normalized='boto'), version=Version(raw='2.49.0a1', normalized='2.49a1')), ...
Ok it seems this package is not in canonical form according to the packaging module. That's unfortunate
@hundreds-father-404 Do you think it would be worth trying to handle this in Pex by normalizing discovered dependency versions before trying to match them to the normalized requirements?
Strangely enough, normalizing the version in the lockfile works with Pex directly, but not in Pants
e
@rough-vase-83553 although I don't have access to boto 2.49.0a1, I do have access to pantsbuild.pants 2.14.0a0 which has the same normalization properties. I find Pex handles that completely for the entire lifecycle. Lock creation works:
Copy code
$ pex3 lock create --style universal --resolver-version pip-2020-resolver pantsbuild.pants==2.14.0a0 --indent 2 -o lock.json --interpreter-constraint "CPython>=3.7,<3.10" --target-system linux --target-system mac
$ jq .requirements lock.json
[
  "pantsbuild.pants==2.14.0a0"
]
$ jq -r '.locked_resolves[] | .locked_requirements[] | select(.project_name == "pantsbuild-pants") | .version' lock.json
2.14a0
Lock consumption works wether via implicit (resolve the whole lock) or explicit with version normalized or not:
Copy code
$ pex --lock lock.json -ofull.pex --interpreter-constraint "CPython>=3.7,<3.10"
$ pex --lock lock.json pantsbuild.pants==2.14.0a0 -oexplicit-root-req.pex --interpreter-constraint "CPython>=3.7,<3.10"
$ pex --lock lock.json pantsbuild.pants==2.14a0 -oexplicit-root-req-normalized.pex --interpreter-constraint "CPython>=3.7,<3.10"
$ ls -lrt *.pex
-rwxr-xr-x 1 jsirois jsirois 158380109 Oct 18 11:53 full.pex
-rwxr-xr-x 1 jsirois jsirois 158380109 Oct 18 11:54 explicit-root-req.pex
-rwxr-xr-x 1 jsirois jsirois 158380107 Oct 18 11:55 explicit-root-req-normalized.pex
$ diff -u <(pex-tools full.pex info -i2) <(pex-tools explicit-root-req.pex info -i2)
$ diff -u <(pex-tools explicit-root-req-normalized.pex info -i2) <(pex-tools explicit-root-req.pex info -i2)
--- /dev/fd/63  2022-10-18 12:06:41.181807816 -0700
+++ /dev/fd/62  2022-10-18 12:06:41.181807816 -0700
@@ -52,11 +52,11 @@
   "interpreter_constraints": [
     "CPython<3.10,>=3.7"
   ],
-  "pex_hash": "d2bcc8934d8b8eb3659333598c2878b1ad2ee50a",
+  "pex_hash": "66738a3a3a993b8c5165248e2017c8876d5dff93",
   "pex_path": "",
   "pex_paths": [],
   "requirements": [
-    "pantsbuild.pants==2.14a0"
+    "pantsbuild.pants==2.14.0a0"
   ],
   "strip_pex_env": true,
   "venv": false,
$ ./full.pex -c 'import pants.version; print(pants.version.__file__)'
/home/jsirois/.pex/installed_wheels/7bd29822f3c7d8ef101cf687dd4bee6afed808fb143fd1b15dabb8c428a50e75/pantsbuild.pants-2.14.0a0-cp37-cp37m-manylinux2014_x86_64.whl/pants/version.py
$ ./explicit-root-req.pex -c 'import pants.version; print(pants.version.__file__)'
/home/jsirois/.pex/installed_wheels/7bd29822f3c7d8ef101cf687dd4bee6afed808fb143fd1b15dabb8c428a50e75/pantsbuild.pants-2.14.0a0-cp37-cp37m-manylinux2014_x86_64.whl/pants/version.py
$ ./explicit-root-req-normalized.pex -c 'import pants.version; print(pants.version.__file__)'
/home/jsirois/.pex/installed_wheels/7bd29822f3c7d8ef101cf687dd4bee6afed808fb143fd1b15dabb8c428a50e75/pantsbuild.pants-2.14.0a0-cp37-cp37m-manylinux2014_x86_64.whl/pants/version.py
As such, I think you should file a Pants bug here. Clearly it does not handle normalized versions (https://peps.python.org/pep-0440/#normalization) / use packaging in all the right places to handle this normalization.
👀 1
r
That's rather bizarre. Let me give pantsbuild.pants 2.14.0a0.a try. I'm using Pex built from source and can confirm it fails for this boto version on the lockfile. Making the patch to Pex referenced in this issue allows Pex to process the lockfile
Let me see if I can reproduce it. But looking at the code, it does seem like 
specifier.contains("2.49a1")
fails when the specifier is 
<SpecifierSet('===2.49.0a1')>
And 
str(version)
does generate the normalized form, which is "2.49a1"
e
One thing missing is expanding "fail" to a backtrace. That would be super helpful.
r
Is there a setting for that? The failure doesn't exactly have a backtrace
e
A full command line with full output would be good. You can always crank up logging with PEX_VERBOSE=9
Basically, the less words and more command lines / outputs the better for debugging!
So, one detail I was not faithful to. Did you actually mean to use 
===
vs 
==
? That has a very specific meaning which is - do not normalize - use literally!
r
Ahh, yes we do in fact use exact equality everywhere
Although if that's the case then isn't Pex arguably misbehaving here? We do have an exact match
e
For good reason?
There may be a bug here, yes.
r
Well... I wasn't involved in the decision-making there but I suppose it helps ensure some extra determinism in some edge cases
I'm fine with replacing it with soft equality to workaround this issue
e
Ok. Let me give a crack at using my pantsbuild.pants case to come up with a command line repro for the ticket. As it stands its way too vague.
👍 2
r
Let me add some clarifying details regarding arbitrary equality
e
I just updated with a repro.
Ok, I'll take a look. This shouldn't be too bad to fix. That said, for your forbears, 
==
is just as exact as 
===
, its just that you can't spell some exact versions with 
==
because they are illegal. For those, and only those, you must use 
===
if the tool supports it, which, as the PEP notes, it heavily discourages.
r
Yep I agree there. I would also push our internal maintainers to rename packages appropriately. I'll create a ticket for that.
e
Excellent. I'll see what I can do in the meantime.
🙌 1
r
Great, thanks. Let me verify the 
==
fix works
Yep, works. I'm now unblocked. Thanks @enough-analyst-54434
e
Ok, great.
Thanks for finding this @rough-vase-83553. It turns out resolving from a PEX repository handled this (as does a normal resolve) and only lock resolves handled this badly. The fix should be out by tomorrow in Pex 2.1.111: https://github.com/pantsbuild/pex/pull/1951 That will percolate through 2.15.x but if you need it sooner and have not done so before I can give you some pants.toml pointers to upgrade your Pants installation to that Pex version.
❤️ 2
r
I think 
==
works nicely enough for now. We're still deep in exploration / POC, so nothing urgent.
6 Views
Open in Slack
PreviousNext
Powered by