https://pantsbuild.org/ logo
Join Slack
Powered by
Trying to lock a resolve that includes my `pants_r...
# general
f
Trying to lock a resolve that includes my 
pants_requirements
on 2.16.0.rc5 but it's unable to find `pants.testutil`:
Copy code
stderr:
pid 71103 -> /var/tmp/jreed-cache/pants/named_caches/pex_root/venvs/faa51f450f2258a6362960a2faecb9ea7e938193/73c7d7c618fac01449fa6e57a21210c0eb8a8f21/bin/python -sE /var/tmp/jreed-cache/pants/named_caches/pex_root/venvs/faa51f450f2258a6362960a2faecb9ea7e938193/73c7d7c618fac01449fa6e57a21210c0eb8a8f21/pex --disable-pip-version-check --no-python-version-warning --exists-action a --no-input --isolated -q --cache-dir /var/tmp/jreed-cache/pants/named_caches/pex_root/pip_cache --log /tmp/pants-sandbox-TuzYPS/.tmp/pex-pip-log.pctusz7z/pip.log download --dest /tmp/pants-sandbox-TuzYPS/.tmp/tmpax0xnw4h/usr.bin.python3.11 fastapi~=0.84.0 filelock~=3.7.1 httpx~=0.22.0 more-itertools~=8.12.0 packaging~=21.3 pantsbuild.pants.testutil<2.17,>=2.16.0a0 pantsbuild.pants<2.17,>=2.16.0a0 paramiko~=2.12.0 pydantic~=1.10.2 pytest~=7.1 uvicorn~=0.15.0 --index-url <https://pypi.org/simple/> --retries 5 --timeout 15 exited with 1 and STDERR:
ERROR: Could not find a version that satisfies the requirement pantsbuild.pants.testutil<2.17,>=2.16.0a0
ERROR: No matching distribution found for pantsbuild.pants.testutil<2.17,>=2.16.0a0
✅ 1
e
Here is your needle: 
--dest /tmp/pants-sandbox-TuzYPS/.tmp/tmpax0xnw4h/usr.bin.python3.11
Python 3.11 is a no-go for Pants.
You probably need a separate resolve for your pants plugin that has ICs like Pants of 
>=3.7,<3.10
.
f
Ah dammit
Resolves are hard
Would it work to widen the constraints on my other requirements?
e
I'm not following. This is all about interpreter constraints. You'd have to narrow those globally to be a subset of Pants'.
If you can't - I.E. non-pants-plugin parts of the repo should be using Python 3.11, then you must use two resolves.
Basically you hand Pex an IC of 
>=3.7,<3.12
it will ensure every bit of the lock is useable for every Python interpreter in that range or fail.
🤔 1
This is for "universal" style locks only, but that's all Pants uses. It gives you no option to use the other Pex lock styles.
Pants says 
pex3 lock create --style universal --target-system linux --target-system mac --interpreter-constraint <yours> ...
So Pex is told: The lock must work on Mac and Linux, for all interpreters in the IC range - that's what "universal" style means.
If you have a top-level requirement that is a direct input to the lock, you can qualify with en environment marker like: 
"pantsbuild.pants==X; python_version <= '3.9'"
It may be that `pants_requirements`is in your way here and does not do the right thing with env markers.
But its more likely you'd still have issues - say your Pants plugin code imported shared repo code that, in turn, depends on other 3rdparty things. Then 3.11 leaks in.
It's only if you're very narrowly luck that you could conceivably not need two resolves the minute your repo ICs diverge from Pants (plugin) ICs.
You could try hand-hacking that running Pants from sources in your repo. Should be a quick check.
f
okay I'll try that
e
Easier than that is just editing the Python source in your Pants install venv. That can be a trick to find though. Its under the ~/.cache/nce on Linux.
You could `rm -rf ~/Library/Caches/nce`(that's the Mac version of the path).
Pants will then re-install the venv printing it out.
Like so:
Copy code
...
New virtual environment successfully created at /home/jsirois/.cache/nce/c55ee58a557d20bd4b109870e5a01b264c0d501ce817cce29502b2552903834d/bindings/venvs/2.15.0.
It's that venv's 
lib/python3.9/site-packages/pants/...
you'd edit.
This is how I do it, way faster than using the Pants from sources thing if you are not hacking Rust.
You just have to remember the footgun that your Pants X.Y.Z is no longer Pants X.Y.Z and blow away the cache again when you're done so it gets re-pristined.
A smaller hammer to clean up is to just kill the locks dir:
Copy code
$ ls -la /home/jsirois/.cache/nce/c55ee58a557d20bd4b109870e5a01b264c0d501ce817cce29502b2552903834d/locks/
total 48
drwxr-xr-x 2 jsirois jsirois 4096 Jun  8 10:02 .
drwxr-xr-x 4 jsirois jsirois 4096 May 30 19:34 ..
-rw-r--r-- 1 jsirois jsirois  389 Jun  8 10:02 configure-59446adc946db9d1a0c4d72ea7f8b84454c568f1aa157d5021c7b6ef240cec6d
-rw-r--r-- 1 jsirois jsirois    0 Jun  8 10:02 configure-59446adc946db9d1a0c4d72ea7f8b84454c568f1aa157d5021c7b6ef240cec6d.lck
-rw-r--r-- 1 jsirois jsirois  389 Jun  6 18:32 configure-780aa8d5a58fadd979c2a39b6fea7efaa0f12c94122b18daa1a2c2efa462f249
-rw-r--r-- 1 jsirois jsirois    0 Jun  6 18:32 configure-780aa8d5a58fadd979c2a39b6fea7efaa0f12c94122b18daa1a2c2efa462f249.lck
-rw-r--r-- 1 jsirois jsirois  462 Jun  6 18:32 configure-7b99a7514da7dce498f4cb96001efda002e9caddeefb7fc15fd1890979892b99
-rw-r--r-- 1 jsirois jsirois    0 Jun  6 18:32 configure-7b99a7514da7dce498f4cb96001efda002e9caddeefb7fc15fd1890979892b99.lck
-rw-r--r-- 1 jsirois jsirois  389 Jun  2 15:04 configure-a5211a1d2ff92065904c469cfb5e75f28afd7fb9b5168a4184bbfbdc9e30c3e8
-rw-r--r-- 1 jsirois jsirois    0 Jun  2 15:04 configure-a5211a1d2ff92065904c469cfb5e75f28afd7fb9b5168a4184bbfbdc9e30c3e8.lck
-rw-r--r-- 1 jsirois jsirois    0 Jun  6 11:02 configure-cffb8e9d1514596f457494f660335454ecfec5e70d77b49a3d23377d83f76b4e.lck
-rw-r--r-- 1 jsirois jsirois  395 May 30 19:34 configure-e85c8a2f172af5d18c271e34913ce14eb4a4925eee82228e554d6e23de0c7c59
-rw-r--r-- 1 jsirois jsirois    0 May 30 19:34 configure-e85c8a2f172af5d18c271e34913ce14eb4a4925eee82228e554d6e23de0c7c59.lck
-rw-r--r-- 1 jsirois jsirois  402 Jun  8 10:02 install-1a8e7657fb3427ee1ef924949fceacc6cf9f83c5075ceec126d1013f57062551
-rw-r--r-- 1 jsirois jsirois    0 Jun  8 10:02 install-1a8e7657fb3427ee1ef924949fceacc6cf9f83c5075ceec126d1013f57062551.lck
-rw-r--r-- 1 jsirois jsirois  402 Jun  2 15:04 install-6c68f5fd6b345e29785bc14f6291bc37290637c5e091dd6f371abe6697bec5bb
-rw-r--r-- 1 jsirois jsirois    0 Jun  2 15:04 install-6c68f5fd6b345e29785bc14f6291bc37290637c5e091dd6f371abe6697bec5bb.lck
-rw-r--r-- 1 jsirois jsirois  402 Jun  6 18:32 install-729611cc37a4f52ebf282343876b3e9b197985598b596322e4f98fc92acce24f
-rw-r--r-- 1 jsirois jsirois    0 Jun  6 18:32 install-729611cc37a4f52ebf282343876b3e9b197985598b596322e4f98fc92acce24f.lck
-rw-r--r-- 1 jsirois jsirois  450 May 30 19:35 install-7d011b648994bc23f41271cd206cab7720af248440ec196eca9a7e42e812712d
-rw-r--r-- 1 jsirois jsirois    0 May 30 19:34 install-7d011b648994bc23f41271cd206cab7720af248440ec196eca9a7e42e812712d.lck
-rw-r--r-- 1 jsirois jsirois   14 Jun  6 11:03 scie-pants-info-04cd37e36e36ec00d672022bb5a37ee31b47042d206b6beeeca58aa4d4f76a69
-rw-r--r-- 1 jsirois jsirois    0 Jun  6 11:03 scie-pants-info-04cd37e36e36ec00d672022bb5a37ee31b47042d206b6beeeca58aa4d4f76a69.lck
That will do a more minimal re-install.
f
doesn't look like updating the requirement worked
although I think theres's a bug in 
PANTS_SEMVER.is_devrelease
or maybe the local cache is interferring
changing the code probably doesn't invalidate the node
e
I just 
pkill pantsd
- whatever your equivalent is for that. I don't muck around with being unsure.
And add a print! Those are always good.
f
okay now we're cooking with gas
Copy code
❯ pants --no-pantsd --no-local-cache generate-lockfiles --resolve=pip-f37
Copy code
13:21:35.08 [INFO] Completed: Generate lockfile for pip-f37
13:21:35.09 [INFO] Wrote lockfile for the resolve `pip-f37` to ci/python-tools/pants-requirements/pip-f37.lock
e
Since Python is python, this is always a much-easier go-to hack on a thing method. With Pants the only trick is finding the venv - not usually an issue for other projects - you built the venv most likely for those, and then the pantsd thing.
f
the lockfile didn't generate artifacts for pants though
or testutil
Copy code
❯ rg pantsbuild\.pants ci/python-tools/pants-requirements/pip-f37.lock
18://     "pantsbuild.pants.testutil<2.17,>=2.16.0a0; python_version <= \"3.9\"",
19://     "pantsbuild.pants<2.17,>=2.16.0a0; python_version <= \"3.9\"",
1211:    "pantsbuild.pants.testutil<2.17,>=2.16.0a0; python_version <= \"3.9\"",
1212:    "pantsbuild.pants<2.17,>=2.16.0a0; python_version <= \"3.9\"",
e
Ok then, yup - 2 resolves.
f
creating two resolves now, hitting the wall that pants-requirements doesn't support interpreter constraints
e
Ok, well at this point you should loop someone else in. I know Pants supports multiple resolves (multiple lock files), but as to how to plumb / debug, no clue.
f
I mean I think this is just an oversight in the pants_requirements code, i'll x-post to development
This has turned into "Can I create a separate resolve for Python 3.9 for my pants-plugin code and another one for my target Pyhon 3.11 prod code?" and the answer so far seems to be no. I don't seem to have a way of telling Pants to not use 3.11 for that resolve, even though all code that uses it has IC of 3.9
okay nvm answer seems to be `
Copy code
python.resolves_to_interpreter_constraints
all this stuff is in the docs if I read hard enough
2 Views
Open in Slack
PreviousNext
Powered by