enough-analyst-54434
03/27/2023, 6:05 PMproud-dentist-22844
03/27/2023, 6:08 PMenough-analyst-54434
03/27/2023, 6:11 PMproud-dentist-22844
03/27/2023, 6:13 PMexport
venv is not immediately ready for use because it uses --pip --collissions-ok
which means my locked version of setuptools gets replaced.enough-analyst-54434
03/27/2023, 6:14 PMproud-dentist-22844
03/27/2023, 6:14 PMenough-analyst-54434
03/27/2023, 6:14 PMproud-dentist-22844
03/27/2023, 6:20 PM./pants export --resolve=st2 --py-resolve-format=mutable_virtualenv --python-bootstrap-search-path="['/usr/bin/python3.8']" --python-bootstrap-names="['python3.8']"
The setuptools versions are correct (the locked version) if I use symlink_immutable_virtualenv
, which I can't use since I need to modify the venv.
(using pants 2.16.0a0) I get these different setuptools versions for the `mutable_virtualenv`:
$ ls -1d dist/export/python/virtualenvs/st2/3.*/lib/python3*/site-packages/setuptools-*.dist-info
dist/export/python/virtualenvs/st2/3.6.15/lib/python3.6/site-packages/setuptools-59.6.0.dist-info
dist/export/python/virtualenvs/st2/3.7.13/lib/python3.7/site-packages/setuptools-59.6.0.dist-info
dist/export/python/virtualenvs/st2/3.8.16/lib/python3.8/site-packages/setuptools-67.2.0.dist-info
enough-analyst-54434
03/27/2023, 6:29 PMproud-dentist-22844
03/27/2023, 6:30 PMenough-analyst-54434
03/27/2023, 6:31 PM$ pex --python python3.8 pip==23.0.1 setuptools==67.2.0 -oexperiment.pex --include-tools
$ PEX_TOOLS=1 ./experiment.pex venv experiment.venv
$ experiment.venv/bin/pip -V
pip 23.0.1 from /home/jsirois/dev/pantsbuild/jsirois-pex/experiment.venv/lib/python3.8/site-packages/pip (python 3.8)
$ experiment.venv/bin/python -c 'import setuptools; print(setuptools.__file__); print(setuptools.__version__)'
/home/jsirois/dev/pantsbuild/jsirois-pex/experiment.venv/lib/python3.8/site-packages/setuptools/__init__.py
67.2.0
PEX_TOOLS...
with --pip
.$ pex --python python3.8 pip==20.3.4 setuptools==59.6.0 -oexperiment.pex --include-tools
$ rm -rf experiment.venv/
$ PEX_TOOLS=1 ./experiment.pex venv experiment.venv
$ experiment.venv/bin/pip -V
pip 20.3.4 from /home/jsirois/dev/pantsbuild/jsirois-pex/experiment.venv/lib/python3.8/site-packages/pip (python 3.8)
$ experiment.venv/bin/python -c 'import setuptools; print(setuptools.__file__); print(setuptools.__version__)'
/home/jsirois/dev/pantsbuild/jsirois-pex/experiment.venv/lib/python3.8/site-packages/setuptools/__init__.py
59.6.0
So this appears to work fine.--pip
and depend on an older version of Pip explicitly, the Pip installed by --pip
will not downgrade. Maybe that's what you're seeing? If the venv has explicit deps on Pip and Setuptools though, there is no need for --pip
.proud-dentist-22844
03/27/2023, 6:41 PMsetuptools
• pip
interpreter_constraints: CPython>=3.6,<3.9
locked versions:
• setuptools-59.6.0
• pip-21.3.1
In my pants exported mutable venvs (ie with --pip --collissions-ok
):
• python3.6 venv:
◦ setuptools-59.6.0
◦ pip-21.3.1
• python3.7 venv:
◦ setuptools-59.6.0
◦ pip-22.0.4
• python3.8 venv
◦ setuptools-67.2.0
◦ pip-23.0.1
So, I don't think pex is replacing the default pip+setuptools (installed because of --pip
) with my locked versions. Maybe that's because of using --collissions-ok
?enough-analyst-54434
03/27/2023, 6:44 PMproud-dentist-22844
03/27/2023, 6:45 PMls -1d dist/export/python/virtualenvs/st2/3.*/lib/python3*/site-packages/setuptools-*.dist-info
ls -1d dist/export/python/virtualenvs/st2/3.*/lib/python3*/site-packages/pip-*.dist-info
Then look at the version in the path.
I also looked in the lockfile to see which version was locked. (search for the distribution and see which version is listed)enough-analyst-54434
03/27/2023, 6:47 PM--pip
, but not setuptools.--pip
.proud-dentist-22844
03/27/2023, 6:47 PM./pants export
to create the venv, which uses the pex-tools command like this:
https://github.com/pantsbuild/pants/blob/main/src/python/pants/backend/python/goals/export.py#L239-L253enough-analyst-54434
03/27/2023, 6:47 PM--pip
if you don't need it.--pip
and depend on Pip.proud-dentist-22844
03/27/2023, 6:51 PM--pip
is requested and pip is in the lockfile.enough-analyst-54434
03/27/2023, 6:52 PM--pip
(the flag you specified) or did you mean pip>=20.3.5
(the requirement you specified).--pip
at all.proud-dentist-22844
03/27/2023, 6:55 PMenough-analyst-54434
03/27/2023, 6:56 PMproud-dentist-22844
03/27/2023, 6:56 PMenough-analyst-54434
03/27/2023, 6:56 PM--pip
is arbitrary.proud-dentist-22844
03/27/2023, 6:58 PMenough-analyst-54434
03/27/2023, 6:59 PM--pip
+ PEX contains Pip nets you a warning "You are crazy", and overwrites --pip
with the PEX Pip?proud-dentist-22844
03/27/2023, 7:00 PMenough-analyst-54434
03/27/2023, 7:01 PMproud-dentist-22844
03/27/2023, 7:02 PM--pip
). So, saying "I want pip" (--pip
) and "use pip version x.y.z" are somewhat redundant but do not conflict, one is more precise than the other.enough-analyst-54434
03/27/2023, 7:02 PM--collisions-ok
can probably suffice to squelch the warning.proud-dentist-22844
03/27/2023, 7:25 PMenough-analyst-54434
03/28/2023, 1:18 PMproud-dentist-22844
03/30/2023, 11:44 PMpex_binary
has the locked versions of pip and setuptools as expected. Thanks @enough-analyst-54434!enough-analyst-54434
03/31/2023, 1:48 AMproud-dentist-22844
03/31/2023, 2:40 AM./pants export
.
$ for y in 6 7 8; do ./pants export --resolve=st2 --py-resolve-format=mutable_virtualenv --python-bootstrap-search-path="['/usr/bin/python3.${y}']" --python-bootstrap-names="['python3.${y}']"; done
21:35:55.53 [INFO] waiting for pantsd to start...
21:35:56.73 [INFO] pantsd started
21:35:57.84 [INFO] Initializing scheduler...
21:36:00.25 [INFO] Scheduler initialized.
21:36:26.78 [INFO] Completed: Build pex for resolve `st2`
21:36:35.57 [INFO] Completed: Get interpreter version
/home/cognifloyd/.cache/pants/named_caches/pex_root/installed_wheels/68aebecfd30b9ad462edf73b41e6463402f0d1d1466f19cfaa883347451e1d7e/pex-2.1.131-py2.py3-none-any.whl/pex/tools/commands/venv.py:164: PEXWarning: You asked for --pip to be installed in the venv at /home/cognifloyd/p/st2sandbox/st2.git/dist/export/python/virtualenvs/st2/3.6.15,
but the PEX at /home/cognifloyd/p/st2sandbox/st2.git/dist/export/python/virtualenvs/st2/3.6.15/.8de694f37a724157ab709eadcd8a3d8c.tmp/st2.pex already contains:
pip 21.3.1
setuptools 59.6
Uninstalling venv versions and using versions from the PEX.
message=message
Uninstalling pip-18.1:
Successfully uninstalled pip-18.1
Uninstalling setuptools-40.6.2:
Successfully uninstalled setuptools-40.6.2
Wrote mutable virtualenv for st2 (using Python 3.6.15) to dist/export/python/virtualenvs/st2/3.6.15
21:36:59.16 [INFO] Completed: Build pex for resolve `st2`
21:37:03.37 [INFO] Completed: Get interpreter version
/home/cognifloyd/.cache/pants/named_caches/pex_root/installed_wheels/68aebecfd30b9ad462edf73b41e6463402f0d1d1466f19cfaa883347451e1d7e/pex-2.1.131-py2.py3-none-any.whl/pex/tools/commands/venv.py:164: PEXWarning: You asked for --pip to be installed in the venv at /home/cognifloyd/p/st2sandbox/st2.git/dist/export/python/virtualenvs/st2/3.7.13,
but the PEX at /home/cognifloyd/p/st2sandbox/st2.git/dist/export/python/virtualenvs/st2/3.7.13/.89bb232d57154f73bcc0dfbd83f0d7b8.tmp/st2.pex already contains:
pip 21.3.1
setuptools 59.6
Uninstalling venv versions and using versions from the PEX.
message=message
Found existing installation: pip 22.0.4
Uninstalling pip-22.0.4:
Successfully uninstalled pip-22.0.4
Found existing installation: setuptools 47.1.0
Uninstalling setuptools-47.1.0:
Successfully uninstalled setuptools-47.1.0
Wrote mutable virtualenv for st2 (using Python 3.7.13) to dist/export/python/virtualenvs/st2/3.7.13
21:37:30.20 [INFO] Completed: Build pex for resolve `st2`
21:37:34.30 [INFO] Completed: Get interpreter version
/home/cognifloyd/.cache/pants/named_caches/pex_root/installed_wheels/68aebecfd30b9ad462edf73b41e6463402f0d1d1466f19cfaa883347451e1d7e/pex-2.1.131-py2.py3-none-any.whl/pex/tools/commands/venv.py:162: PEXWarning: You asked for --pip to be installed in the venv at /home/cognifloyd/p/st2sandbox/st2.git/dist/export/python/virtualenvs/st2/3.8.16,
but the PEX at /home/cognifloyd/p/st2sandbox/st2.git/dist/export/python/virtualenvs/st2/3.8.16/.dd9e584efb314a6182e7391bb76bfd48.tmp/st2.pex already contains:
pip 21.3.1
setuptools 59.6
Uninstalling venv versions and using versions from the PEX.
pex_warnings.warn(
Found existing installation: pip 23.0.1
Uninstalling pip-23.0.1:
Successfully uninstalled pip-23.0.1
Found existing installation: setuptools 67.2.0
Uninstalling setuptools-67.2.0:
Successfully uninstalled setuptools-67.2.0
Wrote mutable virtualenv for st2 (using Python 3.8.16) to dist/export/python/virtualenvs/st2/3.8.16
$ for y in 6 7 8; do dist/export/python/virtualenvs/st2/3.${y}.*/bin/pip list | grep pip; done
pip 21.3.1
pip 21.3.1
WARNING: You are using pip version 21.3.1; however, version 23.0.1 is available.
You should consider upgrading via the '/home/cognifloyd/p/st2sandbox/st2.git/dist/export/python/virtualenvs/st2/3.7.13/bin/python3.7m -m pip install --upgrade pip' command.
pip 21.3.1
WARNING: You are using pip version 21.3.1; however, version 23.0.1 is available.
You should consider upgrading via the '/home/cognifloyd/p/st2sandbox/st2.git/dist/export/python/virtualenvs/st2/3.8.16/bin/python3.8 -m pip install --upgrade pip' command.
$ for y in 6 7 8; do dist/export/python/virtualenvs/st2/3.${y}.*/bin/pip list | grep setuptools; done
setuptools 59.6.0
setuptools 59.6.0
WARNING: You are using pip version 21.3.1; however, version 23.0.1 is available.
You should consider upgrading via the '/home/cognifloyd/p/st2sandbox/st2.git/dist/export/python/virtualenvs/st2/3.7.13/bin/python3.7m -m pip install --upgrade pip' command.
setuptools 59.6.0
WARNING: You are using pip version 21.3.1; however, version 23.0.1 is available.
You should consider upgrading via the '/home/cognifloyd/p/st2sandbox/st2.git/dist/export/python/virtualenvs/st2/3.8.16/bin/python3.8 -m pip install --upgrade pip' command.
$ for y in 6 7 8; do dist/export/python/virtualenvs/st2/3.${y}.*/bin/pip --version ; done
pip 21.3.1 from /home/cognifloyd/p/st2sandbox/st2.git/dist/export/python/virtualenvs/st2/3.6.15/lib64/python3.6/site-packages/pip (python 3.6)
pip 21.3.1 from /home/cognifloyd/p/st2sandbox/st2.git/dist/export/python/virtualenvs/st2/3.7.13/lib/python3.7/site-packages/pip (python 3.7)
pip 21.3.1 from /home/cognifloyd/p/st2sandbox/st2.git/dist/export/python/virtualenvs/st2/3.8.16/lib/python3.8/site-packages/pip (python 3.8)