Getting the following when running ` pants` with version set

Question

Getting the following when running ` pants` with version set to 2 8 0 dev4 2 8 0 dev3 works fine ``` ERROR Command errored out with exit status 1 command Users gordon cache pants setup bootstrap Darwin x86 64 pants 2H81Or install bin python c import io os sys setuptools tokenize sys argv 0 = private var folders g4 8ps3 1d52wl2h7g4120w1yy40000gn T pip install tgl3794i pystache 31f7891a3aad43f38c9c7f0b3c203d9d setup py file = private var folders g4 8ps3 1d52wl2h7g4120w1yy40000gn T pip install tgl3794i pystache 31f7891a3aad43f38c9c7f0b3c203d9d setup py f = getattr tokenize open open file if os path exists file else io StringIO from setuptools import setup setup code = f read replace r n n f close exec compile code file exec egg info egg base private var folders g4 8ps3 1d52wl2h7g4120w1yy40000gn T pip pip egg info 5bh87jeb cwd private var folders g4 8ps3 1d52wl2h7g4120w1yy40000gn T pip install tgl3794i pystache 31f7891a3aad43f38c9c7f0b3c203d9d Complete output 3 lines pystache using version 58 0 4 of lt module setuptools from Users gordon cache pants setup bootstrap Darwin x86 64 pants 2H81Or install lib python3 9 site packages setuptools init py gt Warning classifiers should be a list got type tuple error in pystache setup command use 2to3 is invalid ```

witty-crayon-22786 · Answer

so i have seen this recently and i think that it is because `setuptools` can float in virtualenvs

witty-crayon-22786 · Answer

one sec

witty-crayon-22786 · Answer

hm i don t repro with 3 9

witty-crayon-22786 · Answer

< clean city 64472> how old is your `pants` script this sounds a bit like something that might have been addressed there

clean-city-64472 · Answer

brand new today

witty-crayon-22786 · Answer

oof alrighty then

clean-city-64472 · Answer

i changed `pants version` to dev3 and rerun the exact same command and it all works

clean-city-64472 · Answer

switched back to dev4 and i get the error again

witty-crayon-22786 · Answer

the script creates virtualenvs in `~ cache pants setup` if you move them aside do you see the same error for both dev3 and dev4

clean-city-64472 · Answer

``` pants version = 2 8 0 dev3 rm rf ~ cache pants setup pants Works pants version = 2 8 0 dev4 rm rf ~ cache pants setup pants Error```

clean-city-64472 · Answer

`pystache` doesn t show up in the output for dev3

witty-crayon-22786 · Answer

ok can you include more of the output from the error run there should be some more references to `setuptools`

clean-city-64472 · Answer

```Bootstrapping Pants using usr local bin python3 9 Creating the virtualenv PEX Downloading the Pex PEX Total Received Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 609 100 609 0 0 1371 0 1371 100 3529k 100 3529k 0 0 3550k 0 9794k SHA256 fingerprint of verified Installing pantsbuild pants==2 8 0 dev4 into a virtual environment at Users gordon cache pants setup bootstrap Darwin x86 64 2 8 0 dev4 py39 created virtual environment CPython3 9 1 final 0 64 in 258ms creator CPython3Posix dest= Users gordon cache pants setup bootstrap Darwin x86 64 pants 7Os7rG install clear=False no vcs ignore=False global=False seeder FromAppData download=False pip=bundle setuptools=bundle wheel=bundle via=copy app data dir= Users gordon Library Application Support virtualenv added seed packages pip==21 2 4 setuptools==58 0 4 wheel==0 37 0 activators BashActivator CShellActivator FishActivator PowerShellActivator PythonActivator XonshActivator Requirement already satisfied pip in Users gordon cache pants setup bootstrap Darwin x86 64 pants 7Os7rG install lib python3 9 site packages 21 2 4 Collecting pip Using cached pip 21 3 py3 none any whl 1 7 MB Installing collected packages pip Attempting uninstall pip Found existing installation pip 21 2 4 Uninstalling pip 21 2 4 Successfully uninstalled pip 21 2 4 Successfully installed pip 21 3 Collecting pantsbuild pants==2 8 0 dev4 Using cached pantsbuild pants 2 8 0 dev4 cp39 cp39 macosx 10 15 x86 64 whl 9 4 MB Collecting pex==2 1 51 Using cached pex 2 1 51 py2 py3 none any whl 2 6 MB Collecting setproctitle==1 2 2 Using cached setproctitle 1 2 2 cp39 cp39 macosx 10 15 x86 64 whl Collecting ijson==3 1 4 Using cached ijson 3 1 4 cp39 cp39 macosx 10 9 x86 64 whl 52 kB Collecting types PyYAML==5 4 3 Using cached types PyYAML 5 4 3 py2 py3 none any whl 12 kB Collecting types setuptools==57 0 0 Using cached types setuptools 57 0 0 py3 none any whl 4 8 kB Collecting psutil==5 8 0 Using cached psutil 5 8 0 cp39 cp39 macosx 10 9 x86 64 whl 236 kB Collecting types toml==0 1 3 Using cached types toml 0 1 3 py2 py3 none any whl 2 1 kB Collecting pystache==0 5 4 Using cached pystache 0 5 4 tar gz 75 kB Preparing metadata setup py error ERROR Command errored out with exit status 1 command Users gordon cache pants setup bootstrap Darwin x86 64 pants 7Os7rG install bin python c import io os sys setuptools tokenize sys argv 0 = private var folders g4 8ps3 1d52wl2h7g4120w1yy40000gn T pip install r83h41d7 pystache afeb63400dc04fa6945ef47635299500 setup py file = private var folders g4 8ps3 1d52wl2h7g4120w1yy40000gn T pip install r83h41d7 pystache afeb63400dc04fa6945ef47635299500 setup py f = getattr tokenize open open file if os path exists file else io StringIO from setuptools import setup setup code = f read replace r n n f close exec compile code file exec egg info egg base private var folders g4 8ps3 1d52wl2h7g4120w1yy40000gn T pip pip egg info 6stlpqok cwd private var folders g4 8ps3 1d52wl2h7g4120w1yy40000gn T pip install r83h41d7 pystache afeb63400dc04fa6945ef47635299500 Complete output 3 lines pystache using version 58 0 4 of lt module setuptools from Users gordon cache pants setup bootstrap Darwin x86 64 pants 7Os7rG install lib python3 9 site packages setuptools init py gt Warning classifiers should be a list got type tuple error in pystache setup command use 2to3 is invalid WARNING Discarding from Command errored out with exit status 1 python setup py egg info Check the logs for full command output ERROR Could not find a version that satisfies the requirement pystache==0 5 4 from pantsbuild pants from versions 0 1 0 0 2 0 0 3 0 0 3 1 0 4 0 0 4 1 0 5 0 0 5 1 0 5 2 0 5 3 0 5 4 ERROR No matching distribution found for pystache==0 5 4```

witty-crayon-22786 · Answer

oh you re right that is a very important difference

clean-city-64472 · Answer

Interesting in dev3 it looks like it uninstalls setuptools 58 and reinstalls 57 That happens near the end after other deps are installed

witty-crayon-22786 · Answer

i think that that combines with `pystache` not being compatible with some `setuptools` versions after `58` and your machine has `58 0 4` in the initial venv

witty-crayon-22786 · Answer

yea i learned that the other day <https pantsbuild slack com archives C0D7TNJHL p1633563470103600 thread ts=1633555505 099200 amp cid=C0D7TNJHL>

clean-city-64472 · Answer

Ic anyway dev3 gives me everything I need so I m not blocked or anything

witty-crayon-22786 · Answer

so two things 1 we should pin the setuptools we use inside the venv before installing deps or switch to distributing a PEX because sheesh 2 we should stop depending on `pystache`

fast-nail-55400 · Answer

unclear if relevant we just started using pystache in dev4 for the Go plugin to produce the main function for test binaries

hundreds-father-404 · Answer

Oh yeah I think that was meant as a build support only type of dep < fast nail 55400> I didn t realize that in code review

witty-crayon-22786 · Answer

yea it s ancient and unsupported unfortunately =

hundreds-father-404 · Answer

As I think we know Pants definitely needs a visibility feature to avoid issues like this

fast-nail-55400 · Answer

so we need to switch off of pystache

witty-crayon-22786 · Answer

Before 2 8 0rc0 we need to do one of these two yea <https pantsbuild slack com archives C0D7TNJHL p1634168353330300 thread ts=1634167077 325800 amp cid=C0D7TNJHL|https pantsbuild slack com archives C0D7TNJHL p1634168353330300 thread ts=1634167077 325800 amp cid=C0D7TNJHL>

fast-nail-55400 · Answer

I can switch us off of pystache easily enough Any suggestions for template libraries We could also just generate the main function in Go using Go s template library since we already run a Go wrapper to analyze the test sources

hundreds-father-404 · Answer

that s in its std lib Probably that approach Whenever you add a new dep to Pants it slows down Pants install times which hits users CI

fast-nail-55400 · Answer

yes it is a part of Go s stdlib

fast-nail-55400 · Answer

gt Whenever you add a new dep to Pants it slows down Pants install times which hits users CI not the case this time I chose `pystache` because it was already in the venv

hundreds-father-404 · Answer

in our internal venv not pantsbuild pants dist s requirements It s deceptive that not everything in our requirements txt actually gets included in the dist

fast-nail-55400 · Answer

we should make those contexts clearer

fast-nail-55400 · Answer

or have a CI test that fails if you use an internal only venv module or something

fast-nail-55400 · Answer

or a lint

fast-nail-55400 · Answer

in any event I ll switch the Go plugin to use Go s template lib

hundreds-father-404 · Answer

Yes agreed Which is the visibility feature I m talking about We shouldn t just fix this for Pants itself but give a new feature for all users This is a generally useful thing make sure that a dev doesn t unintentionally add a dep the org doesn t want to add like adding NumPy to something

witty-crayon-22786 · Answer

and I know what < clean city 64472> is thinking if this had been in a `BUILD` file we would have seen it but yea this is a case for some sort of visibility enforcement probably because if you don t see it in imports I doubt you d see it in the BUILD file and transitive still gets you

clean-city-64472 · Answer

I m getting warmer on dependency inference slightly smiling face

hundreds-father-404 · Answer

I was thinking the same thing at first but I don t think it would have saved us here I knew we were adding a new dep to Go code just didn t realize it wasn t in the dist already Transitive deps were the culprit not direct explicit BUILD files only help with direct deps And indeed that s a fundamental promise of Pants you shouldn t have to reason about transitive deps Let the tool do it for you

fast-nail-55400 · Answer

curved-television-6568 · Answer

We got into a similar situation with the `dockerfile parser` just my +1