https://pantsbuild.org/ logo
Join Slack
Powered by
There is not. You'd need to run Pip after creating...
# general
e
There is not. You'd need to run Pip after creating the venv to change versions of itself and other things.
p
Sad. Pip and setuptools are runtime dependencies of st2, so I've got to control what gets included. ie: st2 creates/manages virtualenvs at runtime (so it is running in one venv, but manages a number of other virtualenvs that it uses to run 3rd party actions (scripts essentially).
e
What's so bad about running a second command?
And have you actually tried depending on the versions you need? I'd guess PEX does the right thing?
p
Not bad. Just a bit more work, and it means the pants 
export
venv is not immediately ready for use because it uses 
--pip --collissions-ok
which means my locked version of setuptools gets replaced.
e
Do you have proof? All PEX deps get installed in the venv after the venv is created.
p
And depending on pip directly caused issues with locking iirc. I should try that again and see what happens.
e
Depending on what you depend on is the right thing to do here.
Definitely report a detailed bug if you can't for non mayoral reasons.
p
Using a series of export commands to export py 3.6, 3.7 and 3.8 like this one for 3.8:
Copy code
./pants export --resolve=st2 --py-resolve-format=mutable_virtualenv --python-bootstrap-search-path="['/usr/bin/python3.8']" --python-bootstrap-names="['python3.8']"
The setuptools versions are correct (the locked version) if I use 
symlink_immutable_virtualenv
, which I can't use since I need to modify the venv. (using pants 2.16.0a0) I get these different setuptools versions for the `mutable_virtualenv`:
Copy code
$ ls -1d dist/export/python/virtualenvs/st2/3.*/lib/python3*/site-packages/setuptools-*.dist-info
dist/export/python/virtualenvs/st2/3.6.15/lib/python3.6/site-packages/setuptools-59.6.0.dist-info
dist/export/python/virtualenvs/st2/3.7.13/lib/python3.7/site-packages/setuptools-59.6.0.dist-info
dist/export/python/virtualenvs/st2/3.8.16/lib/python3.8/site-packages/setuptools-67.2.0.dist-info
Oh hey cool. Adding pip to the lockfile didn't break. I guess I haven't tried that in awhile.
e
And you explicitly depend on the setuptools version you need?
p
But, the locked version of setuptools and pip does not match the locked version. No. I'm trying to loosen requirements as much as I can, and then use the lockfile to prevent drift - so I expect the locked version to end up in the venv.
e
Ok - so just let me know if the setuptools versions you observe are buggy or not with respect to your looseness.
Ok, I think I've paged this back in. Let me experiment.
Copy code
$ pex --python python3.8 pip==23.0.1 setuptools==67.2.0 -oexperiment.pex --include-tools
$ PEX_TOOLS=1 ./experiment.pex venv experiment.venv
$ experiment.venv/bin/pip -V
pip 23.0.1 from /home/jsirois/dev/pantsbuild/jsirois-pex/experiment.venv/lib/python3.8/site-packages/pip (python 3.8)
$ experiment.venv/bin/python -c 'import setuptools; print(setuptools.__file__); print(setuptools.__version__)'
/home/jsirois/dev/pantsbuild/jsirois-pex/experiment.venv/lib/python3.8/site-packages/setuptools/__init__.py
67.2.0
So ... I don't know.
Same if I blow away the venv then re-run 
PEX_TOOLS...
with 
--pip
.
And:
Copy code
$ pex --python python3.8 pip==20.3.4 setuptools==59.6.0 -oexperiment.pex --include-tools
$ rm -rf experiment.venv/
$ PEX_TOOLS=1 ./experiment.pex venv experiment.venv
$ experiment.venv/bin/pip -V
pip 20.3.4 from /home/jsirois/dev/pantsbuild/jsirois-pex/experiment.venv/lib/python3.8/site-packages/pip (python 3.8)
$ experiment.venv/bin/python -c 'import setuptools; print(setuptools.__file__); print(setuptools.__version__)'
/home/jsirois/dev/pantsbuild/jsirois-pex/experiment.venv/lib/python3.8/site-packages/setuptools/__init__.py
59.6.0
So this appears to work fine.
I think if you both use 
--pip
and depend on an older version of Pip explicitly, the Pip installed by 
--pip
will not downgrade. Maybe that's what you're seeing? If the venv has explicit deps on Pip and Setuptools though, there is no need for 
--pip
.
p
requirements: • 
setuptools
• 
pip
interpreter_constraints: 
CPython>=3.6,<3.9
locked versions: • setuptools-59.6.0 • pip-21.3.1 In my pants exported mutable venvs (ie with 
--pip --collissions-ok
): • python3.6 venv: ◦ setuptools-59.6.0 ◦ pip-21.3.1 • python3.7 venv: ◦ setuptools-59.6.0 ◦ pip-22.0.4 • python3.8 venv ◦ setuptools-67.2.0 ◦ pip-23.0.1 So, I don't think pex is replacing the default pip+setuptools (installed because of 
--pip
) with my locked versions. Maybe that's because of using 
--collissions-ok
?
e
How are you proving out those versions?
p
Copy code
ls -1d dist/export/python/virtualenvs/st2/3.*/lib/python3*/site-packages/setuptools-*.dist-info

ls -1d dist/export/python/virtualenvs/st2/3.*/lib/python3*/site-packages/pip-*.dist-info
Then look at the version in the path. I also looked in the lockfile to see which version was locked. (search for the distribution and see which version is listed)
e
I do repro pip being off iff 
--pip
, but not setuptools.
So, 1st off, no need for 
--pip
.
p
And I'm using 
./pants export
to create the venv, which uses the pex-tools command like this: https://github.com/pantsbuild/pants/blob/main/src/python/pants/backend/python/goals/export.py#L239-L253
e
Since you provide Pip.
Well, sure - you and I can both hack on Pants.
So, 2 avenues. Pants should probably be able to export and not use 
--pip
if you don't need it.
And Pex could maybe do ... I don't know what yet, if you both specify 
--pip
and depend on Pip.
p
Yeah. I'm looking at hacking pants to be more flexible here (I have a few other things to change of which, fixing pip+setuptools version is just one). But it would also be nice if pex could downgrade the version of pip+setuptools to match the locked version if 
--pip
is requested and pip is in the lockfile.
e
You see the problem though right? You asked for 2 things and the tie break is effectively arbitrary. Did you mean 
--pip
(the flag you specified) or did you mean 
pip>=20.3.5
(the requirement you specified).
We know what you want now in this situation. The question is if that makes universal sense. Because, really, if you ran that on the command line you'd just not use 
--pip
at all.
p
Hmm. Does pex-tools have access to the lockfile at this point (after reqs.pex is created)? Or just whatever happens to be included in the reqs.pex?
e
Just what is in the PEX.
Still the choice of which you meant is arbitrary afaict.
p
If the lockfile is available, then the request is unambiguous. Dang. No lockfile so yeah that makes it anmbiguous.
e
No, not at all.
PEX is a lock file with content. It has exactly 1 version.
Nonetheless, I still contend picking that version vs 
--pip
is arbitrary.
As a human you would not say both!
Pants is foisting you on a petard here.
p
yes. that is arbitrary here. It would only make sense if the lockfile (the json file, not the resulting pex file) itself had been passed somehow to define the requirements+constraints. That is what I thought was happening, so now I get to correct my mental model.
e
I think you're off with the lockfile business - it is not relevant. The PEX itself decalres a Pip version by dint of the Pip installed wheel it contains, which is necessarily exact.
How about 
--pip
+ PEX contains Pip nets you a warning "You are crazy", and overwrites 
--pip
with the PEX Pip?
p
yup. Thanks for helping me see that I was barking up the wrong tree. LOL. I would love an error message that said "you are crazy"
e
Ok. This seems reasonable to me. If you're willing to file a feature request I can make that so.
p
But yeah. I'd say the version in the pex is more exact then just saying "I want pip" (
--pip
). So, saying "I want pip" (
--pip
) and "use pip version x.y.z" are somewhat redundant but do not conflict, one is more precise than the other.
I'll file a feature request.
(I'll also hack on pants because ugh)
e
Yeah, reasoning is dime a dozen as you well know. But if the --help says this is how it works, that's fine.
👍 1
I think the existing 
--collisions-ok
can probably suffice to squelch the warning.
👍 1
p
I think I summarized my request.
e
Thanks. I should be able to finish this off and get out Pex 2.1.131 with support Wednesday morning: https://github.com/pantsbuild/pex/pull/2107
🎉 1
Alrighty - took longer than I expected to get the tests done. Now ready for review: https://github.com/pantsbuild/pex/pull/2107
❤️ 1
Ok, Pex 2.1.131 is released the Pants upgrade instructions are here: https://github.com/pantsbuild/pants/pull/18626#issuecomment-1491100039
p
sounds good. Thank you!
I have validated that with pex 2.1.131, the venv created with a pex generated via pants' 
pex_binary
has the locked versions of pip and setuptools as expected. Thanks @enough-analyst-54434!
e
Excellent, thanks for testing and reporting back.
p
I've also validated that this works via 
./pants export
.
Copy code
$ for y in 6 7 8; do ./pants export --resolve=st2 --py-resolve-format=mutable_virtualenv --python-bootstrap-search-path="['/usr/bin/python3.${y}']" --python-bootstrap-names="['python3.${y}']"; done
21:35:55.53 [INFO] waiting for pantsd to start...
21:35:56.73 [INFO] pantsd started
21:35:57.84 [INFO] Initializing scheduler...
21:36:00.25 [INFO] Scheduler initialized.
21:36:26.78 [INFO] Completed: Build pex for resolve `st2`
21:36:35.57 [INFO] Completed: Get interpreter version
/home/cognifloyd/.cache/pants/named_caches/pex_root/installed_wheels/68aebecfd30b9ad462edf73b41e6463402f0d1d1466f19cfaa883347451e1d7e/pex-2.1.131-py2.py3-none-any.whl/pex/tools/commands/venv.py:164: PEXWarning: You asked for --pip to be installed in the venv at /home/cognifloyd/p/st2sandbox/st2.git/dist/export/python/virtualenvs/st2/3.6.15,
but the PEX at /home/cognifloyd/p/st2sandbox/st2.git/dist/export/python/virtualenvs/st2/3.6.15/.8de694f37a724157ab709eadcd8a3d8c.tmp/st2.pex already contains:
pip 21.3.1
setuptools 59.6
Uninstalling venv versions and using versions from the PEX.
  message=message
Uninstalling pip-18.1:
  Successfully uninstalled pip-18.1
Uninstalling setuptools-40.6.2:
  Successfully uninstalled setuptools-40.6.2
Wrote mutable virtualenv for st2 (using Python 3.6.15) to dist/export/python/virtualenvs/st2/3.6.15
21:36:59.16 [INFO] Completed: Build pex for resolve `st2`
21:37:03.37 [INFO] Completed: Get interpreter version
/home/cognifloyd/.cache/pants/named_caches/pex_root/installed_wheels/68aebecfd30b9ad462edf73b41e6463402f0d1d1466f19cfaa883347451e1d7e/pex-2.1.131-py2.py3-none-any.whl/pex/tools/commands/venv.py:164: PEXWarning: You asked for --pip to be installed in the venv at /home/cognifloyd/p/st2sandbox/st2.git/dist/export/python/virtualenvs/st2/3.7.13,
but the PEX at /home/cognifloyd/p/st2sandbox/st2.git/dist/export/python/virtualenvs/st2/3.7.13/.89bb232d57154f73bcc0dfbd83f0d7b8.tmp/st2.pex already contains:
pip 21.3.1
setuptools 59.6
Uninstalling venv versions and using versions from the PEX.
  message=message
Found existing installation: pip 22.0.4
Uninstalling pip-22.0.4:
  Successfully uninstalled pip-22.0.4
Found existing installation: setuptools 47.1.0
Uninstalling setuptools-47.1.0:
  Successfully uninstalled setuptools-47.1.0
Wrote mutable virtualenv for st2 (using Python 3.7.13) to dist/export/python/virtualenvs/st2/3.7.13
21:37:30.20 [INFO] Completed: Build pex for resolve `st2`
21:37:34.30 [INFO] Completed: Get interpreter version
/home/cognifloyd/.cache/pants/named_caches/pex_root/installed_wheels/68aebecfd30b9ad462edf73b41e6463402f0d1d1466f19cfaa883347451e1d7e/pex-2.1.131-py2.py3-none-any.whl/pex/tools/commands/venv.py:162: PEXWarning: You asked for --pip to be installed in the venv at /home/cognifloyd/p/st2sandbox/st2.git/dist/export/python/virtualenvs/st2/3.8.16,
but the PEX at /home/cognifloyd/p/st2sandbox/st2.git/dist/export/python/virtualenvs/st2/3.8.16/.dd9e584efb314a6182e7391bb76bfd48.tmp/st2.pex already contains:
pip 21.3.1
setuptools 59.6
Uninstalling venv versions and using versions from the PEX.
  pex_warnings.warn(
Found existing installation: pip 23.0.1
Uninstalling pip-23.0.1:
  Successfully uninstalled pip-23.0.1
Found existing installation: setuptools 67.2.0
Uninstalling setuptools-67.2.0:
  Successfully uninstalled setuptools-67.2.0
Wrote mutable virtualenv for st2 (using Python 3.8.16) to dist/export/python/virtualenvs/st2/3.8.16


$ for y in 6 7 8; do dist/export/python/virtualenvs/st2/3.${y}.*/bin/pip list | grep pip; done
pip                        21.3.1
pip                        21.3.1
WARNING: You are using pip version 21.3.1; however, version 23.0.1 is available.
You should consider upgrading via the '/home/cognifloyd/p/st2sandbox/st2.git/dist/export/python/virtualenvs/st2/3.7.13/bin/python3.7m -m pip install --upgrade pip' command.
pip                        21.3.1
WARNING: You are using pip version 21.3.1; however, version 23.0.1 is available.
You should consider upgrading via the '/home/cognifloyd/p/st2sandbox/st2.git/dist/export/python/virtualenvs/st2/3.8.16/bin/python3.8 -m pip install --upgrade pip' command.


$ for y in 6 7 8; do dist/export/python/virtualenvs/st2/3.${y}.*/bin/pip list | grep setuptools; done
setuptools                 59.6.0
setuptools                 59.6.0
WARNING: You are using pip version 21.3.1; however, version 23.0.1 is available.
You should consider upgrading via the '/home/cognifloyd/p/st2sandbox/st2.git/dist/export/python/virtualenvs/st2/3.7.13/bin/python3.7m -m pip install --upgrade pip' command.
setuptools                 59.6.0
WARNING: You are using pip version 21.3.1; however, version 23.0.1 is available.
You should consider upgrading via the '/home/cognifloyd/p/st2sandbox/st2.git/dist/export/python/virtualenvs/st2/3.8.16/bin/python3.8 -m pip install --upgrade pip' command.


$ for y in 6 7 8; do dist/export/python/virtualenvs/st2/3.${y}.*/bin/pip --version ; done
pip 21.3.1 from /home/cognifloyd/p/st2sandbox/st2.git/dist/export/python/virtualenvs/st2/3.6.15/lib64/python3.6/site-packages/pip (python 3.6)
pip 21.3.1 from /home/cognifloyd/p/st2sandbox/st2.git/dist/export/python/virtualenvs/st2/3.7.13/lib/python3.7/site-packages/pip (python 3.7)
pip 21.3.1 from /home/cognifloyd/p/st2sandbox/st2.git/dist/export/python/virtualenvs/st2/3.8.16/lib/python3.8/site-packages/pip (python 3.8)
4 Views
Open in Slack
PreviousNext
Powered by