https://pantsbuild.org/ logo
e

enough-analyst-54434

03/27/2023, 6:05 PM
There is not. You'd need to run Pip after creating the venv to change versions of itself and other things.
p

proud-dentist-22844

03/27/2023, 6:08 PM
Sad. Pip and setuptools are runtime dependencies of st2, so I've got to control what gets included. ie: st2 creates/manages virtualenvs at runtime (so it is running in one venv, but manages a number of other virtualenvs that it uses to run 3rd party actions (scripts essentially).
e

enough-analyst-54434

03/27/2023, 6:11 PM
What's so bad about running a second command?
And have you actually tried depending on the versions you need? I'd guess PEX does the right thing?
p

proud-dentist-22844

03/27/2023, 6:13 PM
Not bad. Just a bit more work, and it means the pants
export
venv is not immediately ready for use because it uses
--pip --collissions-ok
which means my locked version of setuptools gets replaced.
e

enough-analyst-54434

03/27/2023, 6:14 PM
Do you have proof? All PEX deps get installed in the venv after the venv is created.
p

proud-dentist-22844

03/27/2023, 6:14 PM
And depending on pip directly caused issues with locking iirc. I should try that again and see what happens.
e

enough-analyst-54434

03/27/2023, 6:14 PM
Depending on what you depend on is the right thing to do here.
Definitely report a detailed bug if you can't for non mayoral reasons.
p

proud-dentist-22844

03/27/2023, 6:20 PM
Using a series of export commands to export py 3.6, 3.7 and 3.8 like this one for 3.8:
Copy code
./pants export --resolve=st2 --py-resolve-format=mutable_virtualenv --python-bootstrap-search-path="['/usr/bin/python3.8']" --python-bootstrap-names="['python3.8']"
The setuptools versions are correct (the locked version) if I use
symlink_immutable_virtualenv
, which I can't use since I need to modify the venv. (using pants 2.16.0a0) I get these different setuptools versions for the `mutable_virtualenv`:
Copy code
$ ls -1d dist/export/python/virtualenvs/st2/3.*/lib/python3*/site-packages/setuptools-*.dist-info
dist/export/python/virtualenvs/st2/3.6.15/lib/python3.6/site-packages/setuptools-59.6.0.dist-info
dist/export/python/virtualenvs/st2/3.7.13/lib/python3.7/site-packages/setuptools-59.6.0.dist-info
dist/export/python/virtualenvs/st2/3.8.16/lib/python3.8/site-packages/setuptools-67.2.0.dist-info
Oh hey cool. Adding pip to the lockfile didn't break. I guess I haven't tried that in awhile.
e

enough-analyst-54434

03/27/2023, 6:29 PM
And you explicitly depend on the setuptools version you need?
p

proud-dentist-22844

03/27/2023, 6:30 PM
But, the locked version of setuptools and pip does not match the locked version. No. I'm trying to loosen requirements as much as I can, and then use the lockfile to prevent drift - so I expect the locked version to end up in the venv.
e

enough-analyst-54434

03/27/2023, 6:31 PM
Ok - so just let me know if the setuptools versions you observe are buggy or not with respect to your looseness.
Ok, I think I've paged this back in. Let me experiment.
Copy code
$ pex --python python3.8 pip==23.0.1 setuptools==67.2.0 -oexperiment.pex --include-tools
$ PEX_TOOLS=1 ./experiment.pex venv experiment.venv
$ experiment.venv/bin/pip -V
pip 23.0.1 from /home/jsirois/dev/pantsbuild/jsirois-pex/experiment.venv/lib/python3.8/site-packages/pip (python 3.8)
$ experiment.venv/bin/python -c 'import setuptools; print(setuptools.__file__); print(setuptools.__version__)'
/home/jsirois/dev/pantsbuild/jsirois-pex/experiment.venv/lib/python3.8/site-packages/setuptools/__init__.py
67.2.0
So ... I don't know.
Same if I blow away the venv then re-run
PEX_TOOLS...
with
--pip
.
And:
Copy code
$ pex --python python3.8 pip==20.3.4 setuptools==59.6.0 -oexperiment.pex --include-tools
$ rm -rf experiment.venv/
$ PEX_TOOLS=1 ./experiment.pex venv experiment.venv
$ experiment.venv/bin/pip -V
pip 20.3.4 from /home/jsirois/dev/pantsbuild/jsirois-pex/experiment.venv/lib/python3.8/site-packages/pip (python 3.8)
$ experiment.venv/bin/python -c 'import setuptools; print(setuptools.__file__); print(setuptools.__version__)'
/home/jsirois/dev/pantsbuild/jsirois-pex/experiment.venv/lib/python3.8/site-packages/setuptools/__init__.py
59.6.0
So this appears to work fine.
I think if you both use
--pip
and depend on an older version of Pip explicitly, the Pip installed by
--pip
will not downgrade. Maybe that's what you're seeing? If the venv has explicit deps on Pip and Setuptools though, there is no need for
--pip
.
p

proud-dentist-22844

03/27/2023, 6:41 PM
requirements: •
setuptools
pip
interpreter_constraints:
CPython>=3.6,<3.9
locked versions: • setuptools-59.6.0 • pip-21.3.1 In my pants exported mutable venvs (ie with
--pip --collissions-ok
): • python3.6 venv: ◦ setuptools-59.6.0 ◦ pip-21.3.1 • python3.7 venv: ◦ setuptools-59.6.0 ◦ pip-22.0.4 • python3.8 venv ◦ setuptools-67.2.0 ◦ pip-23.0.1 So, I don't think pex is replacing the default pip+setuptools (installed because of
--pip
) with my locked versions. Maybe that's because of using
--collissions-ok
?
e

enough-analyst-54434

03/27/2023, 6:44 PM
How are you proving out those versions?
p

proud-dentist-22844

03/27/2023, 6:45 PM
Copy code
ls -1d dist/export/python/virtualenvs/st2/3.*/lib/python3*/site-packages/setuptools-*.dist-info

ls -1d dist/export/python/virtualenvs/st2/3.*/lib/python3*/site-packages/pip-*.dist-info
Then look at the version in the path. I also looked in the lockfile to see which version was locked. (search for the distribution and see which version is listed)
e

enough-analyst-54434

03/27/2023, 6:47 PM
I do repro pip being off iff
--pip
, but not setuptools.
So, 1st off, no need for
--pip
.
p

proud-dentist-22844

03/27/2023, 6:47 PM
And I'm using
./pants export
to create the venv, which uses the pex-tools command like this: https://github.com/pantsbuild/pants/blob/main/src/python/pants/backend/python/goals/export.py#L239-L253
e

enough-analyst-54434

03/27/2023, 6:47 PM
Since you provide Pip.
Well, sure - you and I can both hack on Pants.
So, 2 avenues. Pants should probably be able to export and not use
--pip
if you don't need it.
And Pex could maybe do ... I don't know what yet, if you both specify
--pip
and depend on Pip.
p

proud-dentist-22844

03/27/2023, 6:51 PM
Yeah. I'm looking at hacking pants to be more flexible here (I have a few other things to change of which, fixing pip+setuptools version is just one). But it would also be nice if pex could downgrade the version of pip+setuptools to match the locked version if
--pip
is requested and pip is in the lockfile.
e

enough-analyst-54434

03/27/2023, 6:52 PM
You see the problem though right? You asked for 2 things and the tie break is effectively arbitrary. Did you mean
--pip
(the flag you specified) or did you mean
pip>=20.3.5
(the requirement you specified).
We know what you want now in this situation. The question is if that makes universal sense. Because, really, if you ran that on the command line you'd just not use
--pip
at all.
p

proud-dentist-22844

03/27/2023, 6:55 PM
Hmm. Does pex-tools have access to the lockfile at this point (after reqs.pex is created)? Or just whatever happens to be included in the reqs.pex?
e

enough-analyst-54434

03/27/2023, 6:56 PM
Just what is in the PEX.
Still the choice of which you meant is arbitrary afaict.
p

proud-dentist-22844

03/27/2023, 6:56 PM
If the lockfile is available, then the request is unambiguous. Dang. No lockfile so yeah that makes it anmbiguous.
e

enough-analyst-54434

03/27/2023, 6:56 PM
No, not at all.
PEX is a lock file with content. It has exactly 1 version.
Nonetheless, I still contend picking that version vs
--pip
is arbitrary.
As a human you would not say both!
Pants is foisting you on a petard here.
p

proud-dentist-22844

03/27/2023, 6:58 PM
yes. that is arbitrary here. It would only make sense if the lockfile (the json file, not the resulting pex file) itself had been passed somehow to define the requirements+constraints. That is what I thought was happening, so now I get to correct my mental model.
e

enough-analyst-54434

03/27/2023, 6:59 PM
I think you're off with the lockfile business - it is not relevant. The PEX itself decalres a Pip version by dint of the Pip installed wheel it contains, which is necessarily exact.
How about
--pip
+ PEX contains Pip nets you a warning "You are crazy", and overwrites
--pip
with the PEX Pip?
p

proud-dentist-22844

03/27/2023, 7:00 PM
yup. Thanks for helping me see that I was barking up the wrong tree. LOL. I would love an error message that said "you are crazy"
e

enough-analyst-54434

03/27/2023, 7:01 PM
Ok. This seems reasonable to me. If you're willing to file a feature request I can make that so.
p

proud-dentist-22844

03/27/2023, 7:02 PM
But yeah. I'd say the version in the pex is more exact then just saying "I want pip" (
--pip
). So, saying "I want pip" (
--pip
) and "use pip version x.y.z" are somewhat redundant but do not conflict, one is more precise than the other.
I'll file a feature request.
(I'll also hack on pants because ugh)
e

enough-analyst-54434

03/27/2023, 7:02 PM
Yeah, reasoning is dime a dozen as you well know. But if the --help says this is how it works, that's fine.
👍 1
I think the existing
--collisions-ok
can probably suffice to squelch the warning.
👍 1
p

proud-dentist-22844

03/27/2023, 7:25 PM
I think I summarized my request.
e

enough-analyst-54434

03/28/2023, 1:18 PM
Thanks. I should be able to finish this off and get out Pex 2.1.131 with support Wednesday morning: https://github.com/pantsbuild/pex/pull/2107
🎉 1
Alrighty - took longer than I expected to get the tests done. Now ready for review: https://github.com/pantsbuild/pex/pull/2107
❤️ 1
Ok, Pex 2.1.131 is released the Pants upgrade instructions are here: https://github.com/pantsbuild/pants/pull/18626#issuecomment-1491100039
p

proud-dentist-22844

03/30/2023, 11:44 PM
sounds good. Thank you!
I have validated that with pex 2.1.131, the venv created with a pex generated via pants'
pex_binary
has the locked versions of pip and setuptools as expected. Thanks @enough-analyst-54434!
e

enough-analyst-54434

03/31/2023, 1:48 AM
Excellent, thanks for testing and reporting back.
p

proud-dentist-22844

03/31/2023, 2:40 AM
I've also validated that this works via
./pants export
.
Copy code
$ for y in 6 7 8; do ./pants export --resolve=st2 --py-resolve-format=mutable_virtualenv --python-bootstrap-search-path="['/usr/bin/python3.${y}']" --python-bootstrap-names="['python3.${y}']"; done
21:35:55.53 [INFO] waiting for pantsd to start...
21:35:56.73 [INFO] pantsd started
21:35:57.84 [INFO] Initializing scheduler...
21:36:00.25 [INFO] Scheduler initialized.
21:36:26.78 [INFO] Completed: Build pex for resolve `st2`
21:36:35.57 [INFO] Completed: Get interpreter version
/home/cognifloyd/.cache/pants/named_caches/pex_root/installed_wheels/68aebecfd30b9ad462edf73b41e6463402f0d1d1466f19cfaa883347451e1d7e/pex-2.1.131-py2.py3-none-any.whl/pex/tools/commands/venv.py:164: PEXWarning: You asked for --pip to be installed in the venv at /home/cognifloyd/p/st2sandbox/st2.git/dist/export/python/virtualenvs/st2/3.6.15,
but the PEX at /home/cognifloyd/p/st2sandbox/st2.git/dist/export/python/virtualenvs/st2/3.6.15/.8de694f37a724157ab709eadcd8a3d8c.tmp/st2.pex already contains:
pip 21.3.1
setuptools 59.6
Uninstalling venv versions and using versions from the PEX.
  message=message
Uninstalling pip-18.1:
  Successfully uninstalled pip-18.1
Uninstalling setuptools-40.6.2:
  Successfully uninstalled setuptools-40.6.2
Wrote mutable virtualenv for st2 (using Python 3.6.15) to dist/export/python/virtualenvs/st2/3.6.15
21:36:59.16 [INFO] Completed: Build pex for resolve `st2`
21:37:03.37 [INFO] Completed: Get interpreter version
/home/cognifloyd/.cache/pants/named_caches/pex_root/installed_wheels/68aebecfd30b9ad462edf73b41e6463402f0d1d1466f19cfaa883347451e1d7e/pex-2.1.131-py2.py3-none-any.whl/pex/tools/commands/venv.py:164: PEXWarning: You asked for --pip to be installed in the venv at /home/cognifloyd/p/st2sandbox/st2.git/dist/export/python/virtualenvs/st2/3.7.13,
but the PEX at /home/cognifloyd/p/st2sandbox/st2.git/dist/export/python/virtualenvs/st2/3.7.13/.89bb232d57154f73bcc0dfbd83f0d7b8.tmp/st2.pex already contains:
pip 21.3.1
setuptools 59.6
Uninstalling venv versions and using versions from the PEX.
  message=message
Found existing installation: pip 22.0.4
Uninstalling pip-22.0.4:
  Successfully uninstalled pip-22.0.4
Found existing installation: setuptools 47.1.0
Uninstalling setuptools-47.1.0:
  Successfully uninstalled setuptools-47.1.0
Wrote mutable virtualenv for st2 (using Python 3.7.13) to dist/export/python/virtualenvs/st2/3.7.13
21:37:30.20 [INFO] Completed: Build pex for resolve `st2`
21:37:34.30 [INFO] Completed: Get interpreter version
/home/cognifloyd/.cache/pants/named_caches/pex_root/installed_wheels/68aebecfd30b9ad462edf73b41e6463402f0d1d1466f19cfaa883347451e1d7e/pex-2.1.131-py2.py3-none-any.whl/pex/tools/commands/venv.py:162: PEXWarning: You asked for --pip to be installed in the venv at /home/cognifloyd/p/st2sandbox/st2.git/dist/export/python/virtualenvs/st2/3.8.16,
but the PEX at /home/cognifloyd/p/st2sandbox/st2.git/dist/export/python/virtualenvs/st2/3.8.16/.dd9e584efb314a6182e7391bb76bfd48.tmp/st2.pex already contains:
pip 21.3.1
setuptools 59.6
Uninstalling venv versions and using versions from the PEX.
  pex_warnings.warn(
Found existing installation: pip 23.0.1
Uninstalling pip-23.0.1:
  Successfully uninstalled pip-23.0.1
Found existing installation: setuptools 67.2.0
Uninstalling setuptools-67.2.0:
  Successfully uninstalled setuptools-67.2.0
Wrote mutable virtualenv for st2 (using Python 3.8.16) to dist/export/python/virtualenvs/st2/3.8.16


$ for y in 6 7 8; do dist/export/python/virtualenvs/st2/3.${y}.*/bin/pip list | grep pip; done
pip                        21.3.1
pip                        21.3.1
WARNING: You are using pip version 21.3.1; however, version 23.0.1 is available.
You should consider upgrading via the '/home/cognifloyd/p/st2sandbox/st2.git/dist/export/python/virtualenvs/st2/3.7.13/bin/python3.7m -m pip install --upgrade pip' command.
pip                        21.3.1
WARNING: You are using pip version 21.3.1; however, version 23.0.1 is available.
You should consider upgrading via the '/home/cognifloyd/p/st2sandbox/st2.git/dist/export/python/virtualenvs/st2/3.8.16/bin/python3.8 -m pip install --upgrade pip' command.


$ for y in 6 7 8; do dist/export/python/virtualenvs/st2/3.${y}.*/bin/pip list | grep setuptools; done
setuptools                 59.6.0
setuptools                 59.6.0
WARNING: You are using pip version 21.3.1; however, version 23.0.1 is available.
You should consider upgrading via the '/home/cognifloyd/p/st2sandbox/st2.git/dist/export/python/virtualenvs/st2/3.7.13/bin/python3.7m -m pip install --upgrade pip' command.
setuptools                 59.6.0
WARNING: You are using pip version 21.3.1; however, version 23.0.1 is available.
You should consider upgrading via the '/home/cognifloyd/p/st2sandbox/st2.git/dist/export/python/virtualenvs/st2/3.8.16/bin/python3.8 -m pip install --upgrade pip' command.


$ for y in 6 7 8; do dist/export/python/virtualenvs/st2/3.${y}.*/bin/pip --version ; done
pip 21.3.1 from /home/cognifloyd/p/st2sandbox/st2.git/dist/export/python/virtualenvs/st2/3.6.15/lib64/python3.6/site-packages/pip (python 3.6)
pip 21.3.1 from /home/cognifloyd/p/st2sandbox/st2.git/dist/export/python/virtualenvs/st2/3.7.13/lib/python3.7/site-packages/pip (python 3.7)
pip 21.3.1 from /home/cognifloyd/p/st2sandbox/st2.git/dist/export/python/virtualenvs/st2/3.8.16/lib/python3.8/site-packages/pip (python 3.8)
4 Views