https://pantsbuild.org/ logo
#general
Title
# general
r

rough-vase-83553

10/17/2022, 7:58 PM
I'm having a weird issue with lockfile generation and verification w.r.t.
python_requirement
. We have some requirements on
boto===2.49.0a1
. I believe this is an internal boto fork that only exists in our private artifactory. However, lockfile generation turns this into
boto===2.49a1
which then fails all the
python_requirement
constraints. Changing the lockfile manually to
boto===2.49.0a1
doesn't fix it either. Is the version number ill-formed or something?
h

hundreds-father-404

10/17/2022, 8:00 PM
Hi, is this using the Poetry lockfile generator or Pex generator?
r

rough-vase-83553

10/17/2022, 8:00 PM
Pex
It seems like an issue with Pex itself. Is there a good way to debug Pex? I tried unzipping the sandbox ./pex and PDBing into it but it eventually fails with an import resolution failure.
Basically a way to run
Copy code
/Users/navneeth.jayendran/.pyenv/versions/3.7.13/bin/python3.7 .pex --tmpdir .tmp --jobs 2 --python-path $'/Users/navneeth.jayendran/.pyenv/versions/2.7.14/bin:/Users/navneeth.jayendran/.pyenv/versions/3.7.13/bin' --output-file requirements.pex --no-emit-warnings --manylinux manylinux2014 --python /Users/navneeth.jayendran/.pyenv/versions/3.7.13/bin/python3.7 $'--sources-directory=source_files' $'boto3>1.17.0' --layout packed
but using a built-from source Pex or something
Okay I started debugging PEX directly from source with
pex --lock mylockfile
. I've noticed that the normalization that PEX gives seems to be the root cause.
Copy code
LockedRequirement(pin=Pin(project_name=ProjectName(raw='boto', normalized='boto'), version=Version(raw='2.49.0a1', normalized='2.49a1')), ...
Ok it seems this package is not in canonical form according to the packaging module. That's unfortunate
@hundreds-father-404 Do you think it would be worth trying to handle this in Pex by normalizing discovered dependency versions before trying to match them to the normalized requirements?
Strangely enough, normalizing the version in the lockfile works with Pex directly, but not in Pants
e

enough-analyst-54434

10/18/2022, 7:16 PM
@rough-vase-83553 although I don't have access to boto 2.49.0a1, I do have access to pantsbuild.pants 2.14.0a0 which has the same normalization properties. I find Pex handles that completely for the entire lifecycle. Lock creation works:
Copy code
$ pex3 lock create --style universal --resolver-version pip-2020-resolver pantsbuild.pants==2.14.0a0 --indent 2 -o lock.json --interpreter-constraint "CPython>=3.7,<3.10" --target-system linux --target-system mac
$ jq .requirements lock.json
[
  "pantsbuild.pants==2.14.0a0"
]
$ jq -r '.locked_resolves[] | .locked_requirements[] | select(.project_name == "pantsbuild-pants") | .version' lock.json
2.14a0
Lock consumption works wether via implicit (resolve the whole lock) or explicit with version normalized or not:
Copy code
$ pex --lock lock.json -ofull.pex --interpreter-constraint "CPython>=3.7,<3.10"
$ pex --lock lock.json pantsbuild.pants==2.14.0a0 -oexplicit-root-req.pex --interpreter-constraint "CPython>=3.7,<3.10"
$ pex --lock lock.json pantsbuild.pants==2.14a0 -oexplicit-root-req-normalized.pex --interpreter-constraint "CPython>=3.7,<3.10"
$ ls -lrt *.pex
-rwxr-xr-x 1 jsirois jsirois 158380109 Oct 18 11:53 full.pex
-rwxr-xr-x 1 jsirois jsirois 158380109 Oct 18 11:54 explicit-root-req.pex
-rwxr-xr-x 1 jsirois jsirois 158380107 Oct 18 11:55 explicit-root-req-normalized.pex
$ diff -u <(pex-tools full.pex info -i2) <(pex-tools explicit-root-req.pex info -i2)
$ diff -u <(pex-tools explicit-root-req-normalized.pex info -i2) <(pex-tools explicit-root-req.pex info -i2)
--- /dev/fd/63  2022-10-18 12:06:41.181807816 -0700
+++ /dev/fd/62  2022-10-18 12:06:41.181807816 -0700
@@ -52,11 +52,11 @@
   "interpreter_constraints": [
     "CPython<3.10,>=3.7"
   ],
-  "pex_hash": "d2bcc8934d8b8eb3659333598c2878b1ad2ee50a",
+  "pex_hash": "66738a3a3a993b8c5165248e2017c8876d5dff93",
   "pex_path": "",
   "pex_paths": [],
   "requirements": [
-    "pantsbuild.pants==2.14a0"
+    "pantsbuild.pants==2.14.0a0"
   ],
   "strip_pex_env": true,
   "venv": false,
$ ./full.pex -c 'import pants.version; print(pants.version.__file__)'
/home/jsirois/.pex/installed_wheels/7bd29822f3c7d8ef101cf687dd4bee6afed808fb143fd1b15dabb8c428a50e75/pantsbuild.pants-2.14.0a0-cp37-cp37m-manylinux2014_x86_64.whl/pants/version.py
$ ./explicit-root-req.pex -c 'import pants.version; print(pants.version.__file__)'
/home/jsirois/.pex/installed_wheels/7bd29822f3c7d8ef101cf687dd4bee6afed808fb143fd1b15dabb8c428a50e75/pantsbuild.pants-2.14.0a0-cp37-cp37m-manylinux2014_x86_64.whl/pants/version.py
$ ./explicit-root-req-normalized.pex -c 'import pants.version; print(pants.version.__file__)'
/home/jsirois/.pex/installed_wheels/7bd29822f3c7d8ef101cf687dd4bee6afed808fb143fd1b15dabb8c428a50e75/pantsbuild.pants-2.14.0a0-cp37-cp37m-manylinux2014_x86_64.whl/pants/version.py
As such, I think you should file a Pants bug here. Clearly it does not handle normalized versions (https://peps.python.org/pep-0440/#normalization) / use packaging in all the right places to handle this normalization.
👀 1
r

rough-vase-83553

10/18/2022, 7:47 PM
That's rather bizarre. Let me give pantsbuild.pants 2.14.0a0.a try. I'm using Pex built from source and can confirm it fails for this boto version on the lockfile. Making the patch to Pex referenced in this issue allows Pex to process the lockfile
Let me see if I can reproduce it. But looking at the code, it does seem like
specifier.contains("2.49a1")
fails when the specifier is
<SpecifierSet('===2.49.0a1')>
And
str(version)
does generate the normalized form, which is "2.49a1"
e

enough-analyst-54434

10/18/2022, 7:51 PM
One thing missing is expanding "fail" to a backtrace. That would be super helpful.
r

rough-vase-83553

10/18/2022, 7:52 PM
Is there a setting for that? The failure doesn't exactly have a backtrace
e

enough-analyst-54434

10/18/2022, 7:53 PM
A full command line with full output would be good. You can always crank up logging with PEX_VERBOSE=9
Basically, the less words and more command lines / outputs the better for debugging!
So, one detail I was not faithful to. Did you actually mean to use
===
vs
==
? That has a very specific meaning which is - do not normalize - use literally!
r

rough-vase-83553

10/18/2022, 7:58 PM
Ahh, yes we do in fact use exact equality everywhere
Although if that's the case then isn't Pex arguably misbehaving here? We do have an exact match
e

enough-analyst-54434

10/18/2022, 7:58 PM
For good reason?
There may be a bug here, yes.
r

rough-vase-83553

10/18/2022, 7:59 PM
Well... I wasn't involved in the decision-making there but I suppose it helps ensure some extra determinism in some edge cases
I'm fine with replacing it with soft equality to workaround this issue
e

enough-analyst-54434

10/18/2022, 8:00 PM
Ok. Let me give a crack at using my pantsbuild.pants case to come up with a command line repro for the ticket. As it stands its way too vague.
👍 2
r

rough-vase-83553

10/18/2022, 8:01 PM
Let me add some clarifying details regarding arbitrary equality
e

enough-analyst-54434

10/18/2022, 8:04 PM
I just updated with a repro.
Ok, I'll take a look. This shouldn't be too bad to fix. That said, for your forbears,
==
is just as exact as
===
, its just that you can't spell some exact versions with
==
because they are illegal. For those, and only those, you must use
===
if the tool supports it, which, as the PEP notes, it heavily discourages.
r

rough-vase-83553

10/18/2022, 8:07 PM
Yep I agree there. I would also push our internal maintainers to rename packages appropriately. I'll create a ticket for that.
e

enough-analyst-54434

10/18/2022, 8:08 PM
Excellent. I'll see what I can do in the meantime.
🙌 1
r

rough-vase-83553

10/18/2022, 8:10 PM
Great, thanks. Let me verify the
==
fix works
Yep, works. I'm now unblocked. Thanks @enough-analyst-54434
e

enough-analyst-54434

10/18/2022, 8:30 PM
Ok, great.
Thanks for finding this @rough-vase-83553. It turns out resolving from a PEX repository handled this (as does a normal resolve) and only lock resolves handled this badly. The fix should be out by tomorrow in Pex 2.1.111: https://github.com/pantsbuild/pex/pull/1951 That will percolate through 2.15.x but if you need it sooner and have not done so before I can give you some pants.toml pointers to upgrade your Pants installation to that Pex version.
❤️ 2
r

rough-vase-83553

10/19/2022, 12:40 AM
I think
==
works nicely enough for now. We're still deep in exploration / POC, so nothing urgent.
6 Views