There is not You d need to run Pip after creating the venv t

Question

There is not You d need to run Pip after creating the venv to change versions of itself and other things

proud-dentist-22844 · Answer

Sad Pip and setuptools are runtime dependencies of st2 so I ve got to control what gets included ie st2 creates manages virtualenvs at runtime so it is running in one venv but manages a number of other virtualenvs that it uses to run 3rd party actions scripts essentially

enough-analyst-54434 · Answer

What s so bad about running a second command

enough-analyst-54434 · Answer

And have you actually tried depending on the versions you need I d guess PEX does the right thing

proud-dentist-22844 · Answer

Not bad Just a bit more work and it means the pants `export` venv is not immediately ready for use because it uses ` pip collissions ok` which means my locked version of setuptools gets replaced

enough-analyst-54434 · Answer

Do you have proof All PEX deps get installed in the venv after the venv is created

proud-dentist-22844 · Answer

And depending on pip directly caused issues with locking iirc I should try that again and see what happens

enough-analyst-54434 · Answer

Depending on what you depend on is the right thing to do here

enough-analyst-54434 · Answer

Definitely report a detailed bug if you can t for non mayoral reasons

proud-dentist-22844 · Answer

Using a series of export commands to export py 3 6 3 7 and 3 8 like this one for 3 8 ``` pants export resolve=st2 py resolve format=mutable virtualenv python bootstrap search path= usr bin python3 8 python bootstrap names= python3 8 ``` The setuptools versions are correct the locked version if I use `symlink immutable virtualenv` which I can t use since I need to modify the venv using pants 2 16 0a0 I get these different setuptools versions for the `mutable virtualenv` ```$ ls 1d dist export python virtualenvs st2 3 lib python3 site packages setuptools dist info dist export python virtualenvs st2 3 6 15 lib python3 6 site packages setuptools 59 6 0 dist info dist export python virtualenvs st2 3 7 13 lib python3 7 site packages setuptools 59 6 0 dist info dist export python virtualenvs st2 3 8 16 lib python3 8 site packages setuptools 67 2 0 dist info```

proud-dentist-22844 · Answer

Oh hey cool Adding pip to the lockfile didn t break I guess I haven t tried that in awhile

enough-analyst-54434 · Answer

And you explicitly depend on the setuptools version you need

proud-dentist-22844 · Answer

But the locked version of setuptools and pip does not match the locked version No I m trying to loosen requirements as much as I can and then use the lockfile to prevent drift so I expect the locked version to end up in the venv

enough-analyst-54434 · Answer

Ok so just let me know if the setuptools versions you observe are buggy or not with respect to your looseness

enough-analyst-54434 · Answer

Ok I think I ve paged this back in Let me experiment

enough-analyst-54434 · Answer

```$ pex python python3 8 pip==23 0 1 setuptools==67 2 0 oexperiment pex include tools $ PEX TOOLS=1 experiment pex venv experiment venv $ experiment venv bin pip V pip 23 0 1 from home jsirois dev pantsbuild jsirois pex experiment venv lib python3 8 site packages pip python 3 8 $ experiment venv bin python c import setuptools print setuptools file print setuptools version home jsirois dev pantsbuild jsirois pex experiment venv lib python3 8 site packages setuptools init py 67 2 0```

enough-analyst-54434 · Answer

So I don t know

enough-analyst-54434 · Answer

Same if I blow away the venv then re run `PEX TOOLS ` with ` pip`

enough-analyst-54434 · Answer

And ```$ pex python python3 8 pip==20 3 4 setuptools==59 6 0 oexperiment pex include tools $ rm rf experiment venv $ PEX TOOLS=1 experiment pex venv experiment venv $ experiment venv bin pip V pip 20 3 4 from home jsirois dev pantsbuild jsirois pex experiment venv lib python3 8 site packages pip python 3 8 $ experiment venv bin python c import setuptools print setuptools file print setuptools version home jsirois dev pantsbuild jsirois pex experiment venv lib python3 8 site packages setuptools init py 59 6 0``` So this appears to work fine

enough-analyst-54434 · Answer

I think if you both use ` pip` and depend on an older version of Pip explicitly the Pip installed by ` pip` will not downgrade Maybe that s what you re seeing If the venv has explicit deps on Pip and Setuptools though there is no need for ` pip`

proud-dentist-22844 · Answer

requirements `setuptools` `pip` interpreter constraints `CPython gt =3 6 lt 3 9` locked versions setuptools 59 6 0 pip 21 3 1 In my pants exported mutable venvs ie with ` pip collissions ok` python3 6 venv setuptools 59 6 0 pip 21 3 1 python3 7 venv setuptools 59 6 0 pip 22 0 4 python3 8 venv setuptools 67 2 0 pip 23 0 1 So I don t think pex is replacing the default pip+setuptools installed because of ` pip` with my locked versions Maybe that s because of using ` collissions ok`

enough-analyst-54434 · Answer

How are you proving out those versions

proud-dentist-22844 · Answer

```ls 1d dist export python virtualenvs st2 3 lib python3 site packages setuptools dist info ls 1d dist export python virtualenvs st2 3 lib python3 site packages pip dist info``` Then look at the version in the path I also looked in the lockfile to see which version was locked search for the distribution and see which version is listed

enough-analyst-54434 · Answer

I do repro pip being off iff ` pip` but not setuptools

enough-analyst-54434 · Answer

So 1st off no need for ` pip`

proud-dentist-22844 · Answer

And I m using ` pants export` to create the venv which uses the pex tools command like this <https github com pantsbuild pants blob main src python pants backend python goals export py L239 L253>

enough-analyst-54434 · Answer

Since you provide Pip

enough-analyst-54434 · Answer

Well sure you and I can both hack on Pants

enough-analyst-54434 · Answer

So 2 avenues Pants should probably be able to export and not use ` pip` if you don t need it

enough-analyst-54434 · Answer

And Pex could maybe do I don t know what yet if you both specify ` pip` and depend on Pip

proud-dentist-22844 · Answer

Yeah I m looking at hacking pants to be more flexible here I have a few other things to change of which fixing pip+setuptools version is just one But it would also be nice if pex could downgrade the version of pip+setuptools to match the locked version if ` pip` is requested and pip is in the lockfile

enough-analyst-54434 · Answer

You see the problem though right You asked for 2 things and the tie break is effectively arbitrary Did you mean ` pip` the flag you specified or did you mean `pip gt =20 3 5` the requirement you specified

enough-analyst-54434 · Answer

We know what you want now in this situation The question is if that makes universal sense Because really if you ran that on the command line you d just not use ` pip` at all

proud-dentist-22844 · Answer

Hmm Does pex tools have access to the lockfile at this point after reqs pex is created Or just whatever happens to be included in the reqs pex

enough-analyst-54434 · Answer

Just what is in the PEX

enough-analyst-54434 · Answer

Still the choice of which you meant is arbitrary afaict

proud-dentist-22844 · Answer

If the lockfile is available then the request is unambiguous Dang No lockfile so yeah that makes it anmbiguous

enough-analyst-54434 · Answer

No not at all

enough-analyst-54434 · Answer

PEX is a lock file with content It has exactly 1 version

enough-analyst-54434 · Answer

Nonetheless I still contend picking that version vs ` pip` is arbitrary

enough-analyst-54434 · Answer

As a human you would not say both

enough-analyst-54434 · Answer

Pants is foisting you on a petard here

proud-dentist-22844 · Answer

yes that is arbitrary here It would only make sense if the lockfile the json file not the resulting pex file itself had been passed somehow to define the requirements+constraints That is what I thought was happening so now I get to correct my mental model

enough-analyst-54434 · Answer

I think you re off with the lockfile business it is not relevant The PEX itself decalres a Pip version by dint of the Pip installed wheel it contains which is necessarily exact

enough-analyst-54434 · Answer

How about ` pip` + PEX contains Pip nets you a warning You are crazy and overwrites ` pip` with the PEX Pip

proud-dentist-22844 · Answer

yup Thanks for helping me see that I was barking up the wrong tree LOL I would love an error message that said you are crazy

enough-analyst-54434 · Answer

Ok This seems reasonable to me If you re willing to file a feature request I can make that so

proud-dentist-22844 · Answer

But yeah I d say the version in the pex is more exact then just saying I want pip ` pip` So saying I want pip ` pip` and use pip version x y z are somewhat redundant but do not conflict one is more precise than the other

proud-dentist-22844 · Answer

I ll file a feature request

proud-dentist-22844 · Answer

I ll also hack on pants because ugh

enough-analyst-54434 · Answer

Yeah reasoning is dime a dozen as you well know But if the help says this is how it works that s fine

enough-analyst-54434 · Answer

I think the existing ` collisions ok` can probably suffice to squelch the warning

proud-dentist-22844 · Answer

proud-dentist-22844 · Answer

I think I summarized my request

enough-analyst-54434 · Answer

Thanks I should be able to finish this off and get out Pex 2 1 131 with support Wednesday morning <https github com pantsbuild pex pull 2107>

enough-analyst-54434 · Answer

Alrighty took longer than I expected to get the tests done Now ready for review <https github com pantsbuild pex pull 2107>

enough-analyst-54434 · Answer

Ok Pex 2 1 131 is released the Pants upgrade instructions are here <https github com pantsbuild pants pull 18626 issuecomment 1491100039>

proud-dentist-22844 · Answer

sounds good Thank you

proud-dentist-22844 · Answer

I have validated that with pex 2 1 131 the venv created with a pex generated via pants `pex binary` has the locked versions of pip and setuptools as expected Thanks < enough analyst 54434>

enough-analyst-54434 · Answer

Excellent thanks for testing and reporting back

proud-dentist-22844 · Answer

I ve also validated that this works via ` pants export` ```$ for y in 6 7 8 do pants export resolve=st2 py resolve format=mutable virtualenv python bootstrap search path= usr bin python3 $ y python bootstrap names= python3 $ y done 21 35 55 53 INFO waiting for pantsd to start 21 35 56 73 INFO pantsd started 21 35 57 84 INFO Initializing scheduler 21 36 00 25 INFO Scheduler initialized 21 36 26 78 INFO Completed Build pex for resolve `st2` 21 36 35 57 INFO Completed Get interpreter version home cognifloyd cache pants named caches pex root installed wheels 68aebecfd30b9ad462edf73b41e6463402f0d1d1466f19cfaa883347451e1d7e pex 2 1 131 py2 py3 none any whl pex tools commands venv py 164 PEXWarning You asked for pip to be installed in the venv at home cognifloyd p st2sandbox st2 git dist export python virtualenvs st2 3 6 15 but the PEX at home cognifloyd p st2sandbox st2 git dist export python virtualenvs st2 3 6 15 8de694f37a724157ab709eadcd8a3d8c tmp st2 pex already contains pip 21 3 1 setuptools 59 6 Uninstalling venv versions and using versions from the PEX message=message Uninstalling pip 18 1 Successfully uninstalled pip 18 1 Uninstalling setuptools 40 6 2 Successfully uninstalled setuptools 40 6 2 Wrote mutable virtualenv for st2 using Python 3 6 15 to dist export python virtualenvs st2 3 6 15 21 36 59 16 INFO Completed Build pex for resolve `st2` 21 37 03 37 INFO Completed Get interpreter version home cognifloyd cache pants named caches pex root installed wheels 68aebecfd30b9ad462edf73b41e6463402f0d1d1466f19cfaa883347451e1d7e pex 2 1 131 py2 py3 none any whl pex tools commands venv py 164 PEXWarning You asked for pip to be installed in the venv at home cognifloyd p st2sandbox st2 git dist export python virtualenvs st2 3 7 13 but the PEX at home cognifloyd p st2sandbox st2 git dist export python virtualenvs st2 3 7 13 89bb232d57154f73bcc0dfbd83f0d7b8 tmp st2 pex already contains pip 21 3 1 setuptools 59 6 Uninstalling venv versions and using versions from the PEX message=message Found existing installation pip 22 0 4 Uninstalling pip 22 0 4 Successfully uninstalled pip 22 0 4 Found existing installation setuptools 47 1 0 Uninstalling setuptools 47 1 0 Successfully uninstalled setuptools 47 1 0 Wrote mutable virtualenv for st2 using Python 3 7 13 to dist export python virtualenvs st2 3 7 13 21 37 30 20 INFO Completed Build pex for resolve `st2` 21 37 34 30 INFO Completed Get interpreter version home cognifloyd cache pants named caches pex root installed wheels 68aebecfd30b9ad462edf73b41e6463402f0d1d1466f19cfaa883347451e1d7e pex 2 1 131 py2 py3 none any whl pex tools commands venv py 162 PEXWarning You asked for pip to be installed in the venv at home cognifloyd p st2sandbox st2 git dist export python virtualenvs st2 3 8 16 but the PEX at home cognifloyd p st2sandbox st2 git dist export python virtualenvs st2 3 8 16 dd9e584efb314a6182e7391bb76bfd48 tmp st2 pex already contains pip 21 3 1 setuptools 59 6 Uninstalling venv versions and using versions from the PEX pex warnings warn Found existing installation pip 23 0 1 Uninstalling pip 23 0 1 Successfully uninstalled pip 23 0 1 Found existing installation setuptools 67 2 0 Uninstalling setuptools 67 2 0 Successfully uninstalled setuptools 67 2 0 Wrote mutable virtualenv for st2 using Python 3 8 16 to dist export python virtualenvs st2 3 8 16 $ for y in 6 7 8 do dist export python virtualenvs st2 3 $ y bin pip list | grep pip done pip 21 3 1 pip 21 3 1 WARNING You are using pip version 21 3 1 however version 23 0 1 is available You should consider upgrading via the home cognifloyd p st2sandbox st2 git dist export python virtualenvs st2 3 7 13 bin python3 7m m pip install upgrade pip command pip 21 3 1 WARNING You are using pip version 21 3 1 however version 23 0 1 is available You should consider upgrading via the home cognifloyd p st2sandbox st2 git dist export python virtualenvs st2 3 8 16 bin python3 8 m pip install upgrade pip command $ for y in 6 7 8 do dist export python virtualenvs st2 3 $ y bin pip list | grep setuptools done setuptools 59 6 0 setuptools 59 6 0 WARNING You are using pip version 21 3 1 however version 23 0 1 is available You should consider upgrading via the home cognifloyd p st2sandbox st2 git dist export python virtualenvs st2 3 7 13 bin python3 7m m pip install upgrade pip command setuptools 59 6 0 WARNING You are using pip version 21 3 1 however version 23 0 1 is available You should consider upgrading via the home cognifloyd p st2sandbox st2 git dist export python virtualenvs st2 3 8 16 bin python3 8 m pip install upgrade pip command $ for y in 6 7 8 do dist export python virtualenvs st2 3 $ y bin pip version done pip 21 3 1 from home cognifloyd p st2sandbox st2 git dist export python virtualenvs st2 3 6 15 lib64 python3 6 site packages pip python 3 6 pip 21 3 1 from home cognifloyd p st2sandbox st2 git dist export python virtualenvs st2 3 7 13 lib python3 7 site packages pip python 3 7 pip 21 3 1 from home cognifloyd p st2sandbox st2 git dist export python virtualenvs st2 3 8 16 lib python3 8 site packages pip python 3 8 ```