https://pantsbuild.org/ logo
Join Slack
Powered by
hello, there is a regression in `pants generate-lo...
# general
f
hello, there is a regression in 
pants generate-lockfiles
that prevents us from moving to versions higher than 2.18 (currently deprecated). as far as I can tell, because of this requirement specification in the 
jupyter-cadquery
package, 
pants generate-lockfiles
fails with this message:
Copy code
~/S/pants-219-lockfile-issue (main *) pants generate-lockfiles
Bootstrapping Pants 2.20.0
Installing pantsbuild.pants==2.20.0 into a virtual environment at /home/blue/.cache/nce/3d6643e46b53e4cc0b2a0d5c768866226ddce3de1f57f80c4a02d8d39800fa8e/bindings/venvs/2.20.0
Found existing installation: setuptools 58.1.0
Uninstalling setuptools-58.1.0:
  Successfully uninstalled setuptools-58.1.0
New virtual environment successfully created at /home/blue/.cache/nce/3d6643e46b53e4cc0b2a0d5c768866226ddce3de1f57f80c4a02d8d39800fa8e/bindings/venvs/2.20.0.
11:18:29.00 [INFO] Initializing scheduler...
11:18:32.08 [INFO] Scheduler initialized.
11:19:12.01 [INFO] Completed: Generate lockfile for python-default
11:19:12.01 [ERROR] 1 Exception encountered:

Engine traceback:
  in `generate-lockfiles` goal

ProcessExecutionFailure: Process 'Generate lockfile for python-default' failed with exit code 1.
stdout:

stderr:
Expected matching RIGHT_PARENTHESIS for LEFT_PARENTHESIS, after version specifier
    cadquery (==master) ; extra == 'prod'
             ~^
You can find a standalone reproducer here: https://github.com/eguiraud-pf/pants-219-lockfile-issue-repro . can anyone here suggest a solution or workaround (or generally tell what the underlying problem is)?
c
Is the 
cadquery==master
syntax valid? If I do 
$ pip install --dry-run jupyter-cadquery[prod]==3.5.2
I get
Copy code
ERROR: Could not find a version that satisfies the requirement cadquery==master; extra == "prod" (from jupyter-cadquery[prod]) (from versions: 0.1.0, 0.1.1, 0.1.2, 0.1.3, 0.1.4, 0.1.5, 0.1.7, 0.1.8, 0.2.0, 0.3.0, 0.4.0, 0.4.185, 0.4.186, 0.5.1, 0.5.2, 1.0.0, 1.1.0, 1.2.0, 1.2.1, 1.2.2, 1.2.4, 1.2.5, 2.2.0b0, 2.2.0b1, 2.2.0b2, 2.2.0, 2.3.0, 2.3.1, 2.4.0)
ERROR: No matching distribution found for cadquery==master; extra == "prod"
f
I can't attest the validity of the syntax, but that error is not about bad syntax. I think it's because there is no package on pypi with that version and the 
[prod]
feature. but also: in the 
requirements.txt
that 
pants generate-lockfiles
reads we don't mention 
[prod]
at all, and 
pip install --dry-run jupyter-cadquery==3.5.2
works fine
b
thanks for the minimal reproducer!
This looks to be a change in the pex tool, that was upgraded from v2.1.137 in pants 2.18.0 to v2.1.148 in pants 2.19.0 (now up to v2.2.1 in pants 2.19.2). If I add this to 
pants.toml
, to match 2.19.0's configuration, I get the same error.
Copy code
[pex-cli]
version = "v2.1.148"
known_versions = [
  "v2.1.148|macos_arm64|5b1dee5a89fff25747753e917f96b8707ea62eed404d037d5f8cf8f2e80a13b7|4197604",
  "v2.1.148|macos_x86_64|5b1dee5a89fff25747753e917f96b8707ea62eed404d037d5f8cf8f2e80a13b7|4197604",
  "v2.1.148|linux_x86_64|5b1dee5a89fff25747753e917f96b8707ea62eed404d037d5f8cf8f2e80a13b7|4197604",
  "v2.1.148|linux_arm64|5b1dee5a89fff25747753e917f96b8707ea62eed404d037d5f8cf8f2e80a13b7|4197604"
]
And indeed, using pex directly matches (this is a reduced form of the command in the 
__run.sh
script visible in the sandbox, when using `--keep-sandboxs=on_failure`):
Copy code
$ PEX_SCRIPT=pex3 pex-2.1.148 lock create jupyter-cadquery==3.5.2
Expected matching RIGHT_PARENTHESIS for LEFT_PARENTHESIS, after version specifier
    cadquery (==master) ; extra == 'prod'

$ PEX_SCRIPT=pex3 pex-2.1.137 lock create jupyter-cadquery==3.5.2
<json blob of the lockfile>
So, maybe these are next steps, in some order: 1. work out exactly which version of pex has the problem (e.g. bisect by downloading releases between 2.1.137 and 2.1.148 from https://github.com/pex-tool/pex/releases and running that command). This hopefully reveals: a. what caused the regression b. whether there's a workaround c. if it's a pex bug 2. for upgrading to pants 2.19, it looks like it allows pex 2.1.137, so you could pin to that version (or whatever the most recent working version is, from step 1) by copying the 
version
and 
known_versions
configuration from https://www.pantsbuild.org/2.18/reference/subsystems/pex-cli. (upgrading to pants 2.20 requires pex 2.1.148, so you'll need to solve the problem properly before doing that) 3. work with 
jupyer-cadquery
upstream to fix what seems to be a wonky dependency specification
f
Thank you for investigating @broad-processor-92400! As you said, pinning pex doesn't sound like a long-term solution, as eventually we'll be forced to a pex version that's not compatible with that package setup. And I am not confident pex will ever go back to be happy with it, so this has to be solved on the side of jupyter-cadquery. Unfortunately that package has not seen a commit in the last two years. So I just forked it, removed the offending line, and pointed pants at our fork đź« 
b
Pex generally doesn’t like regressing behaviour, and prefers to support what happens in practice even if it’s weird/non-standard. So if you wanted to put effort into understanding why pex changed, that may not be wasted. But fixing/forking the package and moving on is fine too.
29 Views
Open in Slack
PreviousNext
Powered by