Is there a way to check what `pants toml` options are in eff

Question

Is there a way to check what `pants toml` options are in effect during a particular run of Pants e g if I am running ` pants package` I would like to see what `python repos` options are active

enough-analyst-54434 · Answer

The most sure footed way I can think of is adding ` ldebug` to the Pants command line This prints out the full set of environment variables and argv for every process Pants invokes

fast-nail-55400 · Answer

You can also run ` pants help python repos` and the configured value should be listed next to every option

bored-glass-58755 · Answer

Thanks < enough analyst 54434> and < fast nail 55400> I am going to try both of these

enough-analyst-54434 · Answer

Tom s method is what you want to normally do If it turns out that method doesn t give an answer that corresponds to behavior you re observing the ldebug will help sort that out We ve had some churn recently in options affecting repos and I m not confident in the status of the plumbing to various resolves The thrash may have all been in ideas though with pen never actually hitting paper

bored-glass-58755 · Answer

Ok so here s what ` pants help python repos` outputs ````python repos` subsystem options External Python code repositories such as PyPI These options may be used to point to custom cheeseshops when resolving requirements Activated by pants core Config section python repos None available Advanced options available You can list them by running pants help advanced python repos ``` It doesn t show me anything even though I have the env variable `PANTS PYTHON REPOS INDEXES` set in my CI along with the option in `pants toml`

enough-analyst-54434 · Answer

It s probably an advanced option although that classification is odd Although its probably advanced in a global sense the minute you re in the scope of the `python repos` subsystem those options are bread and butter Try ` pants help advanced python repos`

bored-glass-58755 · Answer

Ok so here s the output ````python repos` subsystem advanced options python repos repos= lt str gt lt str gt PANTS PYTHON REPOS REPOS repos default current value URLs of code repositories to look for requirements In Pip and Pex this option corresponds to the ` find links` option python repos indexes= lt str gt lt str gt PANTS PYTHON REPOS INDEXES indexes default current value from pants toml from env var PANTS PYTHON REPOS INDEXES URLs of code repository indexes to look for requirements If set to an empty list then Pex will use no indices meaning it will not use PyPI The values should be compliant with PEP 503 `python repos` subsystem deprecated options None available ``` The default is still set to pypi could that be a problem

enough-analyst-54434 · Answer

No the default will always be that

enough-analyst-54434 · Answer

Rather yes it could be a problem but only if Pants has bugs The ldebug info will help get to the bottom of that if `current value` looks good to you but behavior of Pants indicates that s not what is being used

bored-glass-58755 · Answer

The behavior suggests ```pex result ResultError There were 2 errors downloading required artifacts 1 setuptools 57 5 from lt urlopen error Errno 97 Address family not supported by protocol gt 2 wheel 0 37 1 from lt urlopen error Errno 97 Address family not supported by protocol gt ```

bored-glass-58755 · Answer

Here s the stack trace ```Engine traceback in select in pants core goals package package asset in pants backend python goals setup py package python dist client ts client in pants backend python util rules dists run pep517 build in pants backend python util rules pex create venv pex build backend pex in pants backend python util rules pex build pex build backend pex in pants engine process fallible to exec result or raise Traceback most recent call last File $XDG CACHE HOME pants setup bootstrap Linux x86 64 2 12 0 py37 lib python3 7 site packages pants engine process py line 277 in fallible to exec result or raise process cleanup=process cleanup val pants engine process ProcessExecutionFailure Process Building build backend pex from setuptools default lock failed with exit code 1 ``` And a part snapshot redacted of the debug ```09 40 31 51 DEBUG Running Building build backend pex from setuptools default lock under semaphore with concurrency id 2 and concurrency 8 09 40 31 51 INFO Starting Building build backend pex from setuptools default lock 09 40 31 51 DEBUG Starting setup sandbox 09 40 31 53 DEBUG Completed setup sandbox 09 40 31 57 DEBUG spawned local process as Some 122689 for Process argv usr local python 3 7 6 bin python3 7 pex tmpdir tmp jobs 8 python path protoc 3 20 1 bin redis 6 2 5 bin bazel 4 0 0 bin golang 1 15 bin venv bin nodejs v12 bin usr local java openjdk 8u302 x64 bin root usr bin usr local ext cmake 3 17 1 bin maven 3 6 3 bin usr local ext git 2 24 0 bin usr local ext git 2 24 0 libexec git core usr local perl 5 26 1 bin sonar wrapper 6 32 bin sonar scanner 4 7 0 bin helm 3 6 1 dotnet 3 1 usr local sybase bin etc bin etc bin usr bin bin output file build backend pex no emit warnings manylinux manylinux2014 venv seed verbose venv site packages copies python usr local python 3 7 6 bin python3 7 sources directory=source files lock setuptools default lock no pypi index= index= layout packed env CPPFLAGS LANG en US UTF 8 LC ALL en US UTF 8 LDFLAGS PATH protoc 3 20 1 bin redis 6 2 5 bin bazel 4 0 0 bin golang 1 15 bin venv bin nodejs v12 bin usr local java openjdk 8u302 x64 bin root usr bin usr local ext cmake 3 17 1 bin maven 3 6 3 bin usr local ext git 2 24 0 bin usr local ext git 2 24 0 libexec git core usr local perl 5 26 1 bin sonar wrapper 6 32 bin sonar scanner 4 7 0 bin helm 3 6 1 dotnet 3 1 usr local sybase bin etc bin etc bin usr bin bin PEX IGNORE RCFILES true PEX ROOT cache pex root working directory None input digests InputDigests complete DirectoryDigest digest Digest hash Fingerprint lt 0fe2393a7ac8d77f0c987b7edd9962b1c7c9e23264ca982596b13a0c7f446830 gt size bytes 340 tree Some nailgun DirectoryDigest digest Digest hash Fingerprint lt e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 gt size bytes 0 tree Some input files DirectoryDigest digest Digest hash Fingerprint lt 0fe2393a7ac8d77f0c987b7edd9962b1c7c9e23264ca982596b13a0c7f446830 gt size bytes 340 tree Some immutable inputs use nailgun output files output directories RelativePath build backend pex timeout None execution slot variable None concurrency available 10 description Building build backend pex from setuptools default lock level Info append only caches CacheName pex root RelativePath cache pex root jdk home None platform constraint Some Linux x86 64 cache scope Successful 09 40 33 88 INFO Completed Building build backend pex from setuptools default lock 09 40 33 88 DEBUG Completed Scheduling Building build backend pex from setuptools default lock 09 40 33 90 DEBUG Completed pants backend python util rules pex build pex 09 40 33 90 DEBUG Completed `package` goal 09 40 33 90 DEBUG computed 1 nodes in 25 237837 seconds there are 945 total nodes 09 40 33 90 ERROR 1 Exception encountered ProcessExecutionFailure Process Building build backend pex from setuptools default lock failed with exit code 1 ``` One thing that concerns me in debug output is why `PEX ROOT` is set as ` cache pex root` where as I set it as `$XDG CACHE HOME pants named caches pex root` at the beginning of the CI Also the 4th line from last in debug output suggests that package goal is complete whereas according to the stack trace its not

enough-analyst-54434 · Answer

That `PEX ROOT` is a relative path in the sandbox to a symlink that points to `$XDG CACHE HOME pants named caches pex root` No matter how you configure `PEX ROOT`externally Pants will always use that local sandbox path with a symlink to where you set it to

enough-analyst-54434 · Answer

Ok so 1st line ```Running Building build backend pex from setuptools default lock ``` That means you ran into this <https github com pantsbuild pants issues 16328> In short you ll need to customize the default lock file location and re generate the lock for every tool Pants uses Pants ships with open source lock files for all the tools it uses When you don t have access to open source repos you need to regenerate all those locks

enough-analyst-54434 · Answer

This is currently painful and non obvious to be sure

enough-analyst-54434 · Answer

So follow along with that issue which has detailed steps I took to solve this issue in the Pants example python repo Your steps should be similar

bored-glass-58755 · Answer

Thanks a lot for the lead I ll let you know of the progress I make

bored-glass-58755 · Answer

The `tools` directory lockfile location in your example `tools lockfile` needs to be in repo root

enough-analyst-54434 · Answer

Yes It can be anywhere under the repo root however you see fit

bored-glass-58755 · Answer

Ok so I executed the command `PIP INDEX URL=<http private repo url simple> pants version` and it ended as following ```Installing pantsbuild pants==2 12 0 into a virtual environment at $XDG CACHE HOME pants setup bootstrap Linux x86 64 2 12 0 py37 WARNING The repository located at private repo url is not a trusted or secure host and is being ignored If this repository is available via HTTPS we recommend you use HTTPS instead otherwise you may silence this warning and allow it anyway with trusted host private repo url WARNING The repository located at private repo url is not a trusted or secure host and is being ignored If this repository is available via HTTPS we recommend you use HTTPS instead otherwise you may silence this warning and allow it anyway with trusted host private repo url ERROR Could not find a version that satisfies the requirement setuptools lt 58 from versions none ERROR No matching distribution found for setuptools lt 58 WARNING The repository located at private repo url is not a trusted or secure host and is being ignored If this repository is available via HTTPS we recommend you use HTTPS instead otherwise you may silence this warning and allow it anyway with trusted host private repo url error Bash exited with code 1 ``` If lines 2 amp 3 are just warnings and if it did not use `private repo url` why could not it download setuptools just how it was doing it before passing in the `PIP INDEX URL` Also the only change I have made is in `pants toml` by introducing the ` setuptools ` subsystem with `lockfile = tools setuptools lock` I haven t added any other subsystems and also haven t made any change to pants script as it shows in the GitHub link you provided in function `bootstrap virtualenv `

bored-glass-58755 · Answer

Retried by making following change to the script waiting for it to complete ``` $ staging dir install bin pip install trusted host private repo url quiet U pip setuptools lt 58 amp amp $ staging dir install bin pip install $ maybe find links trusted host private repo url quiet progress bar off $ pants requirement ```

bored-glass-58755 · Answer

Now it failed with ```Installing pantsbuild pants==2 12 0 into a virtual environment at $XDG CACHE HOME pants setup bootstrap Linux x86 64 2 12 0 py37 ERROR Could not find a version that satisfies the requirement pantsbuild pants==2 12 0 from versions none ERROR No matching distribution found for pantsbuild pants==2 12 0 error Bash exited with code 1 ```

enough-analyst-54434 · Answer

Perhaps add PIP VERBOSE=3 to see more

bored-glass-58755 · Answer

Ok but the changes I described you above look fine

enough-analyst-54434 · Answer

Probably A diff would be easier to validate That said you need not have edited the script for the experiment you could have stuck with `PIP lt option gt `

bored-glass-58755 · Answer

Yeah I wish I could share the diff Regarding the script so I need not even make this change which I haven t done anyway but just wanted to confirm ``` 270 7 +270 9 function bootstrap virtualenv staging dir=$ tempdir $ PANTS BOOTSTRAP cd $ staging dir echo $ VIRTUALENV REQUIREMENTS gt requirements txt $ python $ pex path r requirements txt c virtualenv o virtualenv pex + $ python $ pex path r requirements txt c virtualenv o virtualenv pex + no pypi + index <https pextest jfrog io artifactory api pypi default pypi simple >```

enough-analyst-54434 · Answer

Correct If you re just experimenting you can use the `PIP `env vars If you want to write it all down you might then want to edit the script unless you have a mechanism like direnv to write down env vars to activate whenever using Pants

enough-analyst-54434 · Answer

In my example on the issue I was writing it all down so I showed final modification of the script

bored-glass-58755 · Answer

Ok

bored-glass-58755 · Answer

I executed `PIP INDEX URL=<http private repo url simple> PIP TRUSTED HOST=private repo url PIP VERBOSE=3 pants version` and it couldn t `find pantsbuild pants==2 12 0` in there which is expected Is there a way that I bootstrap first and then use the `PIP INDEX URL` to `generate lockfiles` Or do I have to bootstrap using this private repo URL as `PIP INDEX URL` for it to work If yes then am I expected to onboard pantsbuild pants onto the private repo The logs are quite long but I can provide them if you don t mind going thru the long logs

enough-analyst-54434 · Answer

gt it couldn t `find pantsbuild pants==2 12 0` in there which is expected Does this mean <http private repo url simple> is not a mirror of PyPI aka it is missing some distributions that are otherwise on PyPI

bored-glass-58755 · Answer

Yes it s not a mirror

enough-analyst-54434 · Answer

Oh wow this is going to be painful

enough-analyst-54434 · Answer

Um let me think

enough-analyst-54434 · Answer

Yeah so the initial Pants bootstrap which is just ~ `pip install pantsbuild pants== lt pants version gt ` is the only part of the process that does not have a lock file right now In order to figure out what distributions you need to mirror to make that work you ll need to run do the following on every platform you target dev machines and CI ``` python we use python mvenv figure out what pants needs figure out what pants needs bin pip wheel wheel dir=wheels pants needs pantsbuild pants== lt pants version gt ``` You d then need to add the contents of the `wheels pants needs ` directory from each platform you target to your private index You ll need to do this for every Pants upgrade so it will behoove you to script it up after getting it all working Pants will need to address all this awkwardness at some point but until then

enough-analyst-54434 · Answer

This just handles the bootstrap though Once you have that handled and you re migrating Pants tool lockfiles you ll undoubtedly run into more missing wheels For those cases though you can review the default tool lock files to see what you re missing For example for pytest it s here as indicated in those docs

bored-glass-58755 · Answer

Ok I completely understood the first part related to bootstrapping pants with only using the private repo I am trying to look at the second part regarding lockfiles Regarding lockfiles do we need to generate them every time CI runs or we can generate them once and then have them checked in thru git If its later then maybe I can generate them once and check it in and then bootstrap pants normally

enough-analyst-54434 · Answer

The latter But to even re generate them the repo at <http private repo url simple> will need to have all the required distributions already available If not the lock re gen will fail As such I think you proetty much need to read through the default lock file checked into the Pants project and grab the full list of wheels and use that to pre populate your private index

bored-glass-58755 · Answer

Ok I ll check the lock file in Pants project But if its the latter then why do I need to re generate them as long as I am using the same pants version

enough-analyst-54434 · Answer

Because the lock files have URLs in them and they are the URLs of Open source artifacts

enough-analyst-54434 · Answer

Presumably the whole point here is your dev CI machines can t use those URLs

enough-analyst-54434 · Answer

IOW the Pants lock files lock URL + fingerprint and not just project name version + fingerprint

bored-glass-58755 · Answer

Ok what I am thinking is 1 Getting a list of wheels for pantsbuild pants on my local dev machine and check which ones need to be onboarded to the private repo 2 Once I have them onboarded I can bootstrap pants on my local using the PIP INDEX URL 3 The third step would be to generate lockfiles and check them to the code repo And then when CI runs it would use the lockfiles I generated and checked in would it after I bootstrap pants again in CI Of course I need to repeat this exercise for every platform I am expected to work on The best would be to do this exercise only once on an environment that resembles the CI s environment and have it run

enough-analyst-54434 · Answer

Step 3 is a one time thing The generated lock files are universal and will have listed all wheels and sdists needed for any platform As such you cannot generate a proper lockfile until your private index has all needed wheels and sdists Concretely say your CI is locked down to 1 Linux distribution CentOS 7 and 1 Python interpreter Python 3 8 10 and then you have devs working purely on Mac but some intel and some arm Lets further say all devs also use Python 3 8 10 That means you have 3 platforms to account for If so then your steps are in order 1 For each platform `pip wheel pantsbuild pants` and gather wheels and put them in your private index 2 For each platform and for each Python tool Pants uses Find the default tool lockfile checked into the Pants repo and gather all the wheels and sdists listed that apply to your platforms and put them in your private index 3 Now bootstrap Pants using the private index and run ` pants generate lockfiles` then check in all the lock files At this point Panst can run on dev machines and in CI

enough-analyst-54434 · Answer

This is all definitely a huge pain and totally ad hoc right now Ideally there would be a Pants tool that could look at your pants toml and allow you to say something like `pants shrinkwrap download dir here pants toml`And that would gather all the artifacts Pants needed under `here `and maybe also generate some form of manifest that could then be used by another tool or command to re configure Pants once you moved all the downloaded files where they needed to go namely re gen lockfiles `pants shinkwrap migrate manifest manifest json pants toml`

enough-analyst-54434 · Answer

Of course you d need to run `pants shrinkwrap download` on a machine with public internet access but then the rest could be done in a locked down environment

bored-glass-58755 · Answer

Ok so to check if I understand it well let s assume that the only wheels that it currently needs to `package` and tries to download from are `setuptools` and `wheel` as demonstrated in the starting message of this thread I assume those wheels are present in the default `setuptools lock` file in the Pants repo and are also present in my private repo If I download that file and replace the URL with my private repo url along with versions and hashes that it needs and check this modified `setuptools lock` file in to my code repo under `tools ` directory Thereafter I have my `pants toml` modified to have ` setuptools ` subsystem with line `lockfile = tools setuptools lock ` After all this I assume if I understand it all correctly that my `package` goal would run successfully

enough-analyst-54434 · Answer

That is correct but hand editing the lock json is not a good idea Instead you download all the wheels by hand edit pants toml to have a custom `lockfile = ` and then run ` pants generate lockfiles`

enough-analyst-54434 · Answer

As long as pants toml also has the ` python repos `config that points at your private repo it will do the right thing and use urls to your private repo in the locks it generates

enough-analyst-54434 · Answer

The only issue is all the right wheels must be uploaded to your private repo 1st

bored-glass-58755 · Answer

Ok I am going to hand edit json for now for just those two packages

bored-glass-58755 · Answer

As long as I have the wheels required by pantsbuild pants in my private repo I can run generate lockfiles and it will re generate files based on the subsystems present in `pants toml` or it will re generate all lockfiles The config in `pants toml` is only to provide a lockfile location in the absence of which it will generate files in the default location

enough-analyst-54434 · Answer

` pants generate lockfiles` will generate lockfiles for every subsystem in `pants toml` that has an explicit non default `lockfile`defined So by default it will do nothing It only does things once you have one or more non default lockfile defined This is additional reason why this is all so paintful right now There is no obvious list of which subsystems you use and thus which ones you ll need to define a custom `lockfile`location for

enough-analyst-54434 · Answer

If you don t have a custom `lockfile` for a subsystem that needs one you currently find out with the style of error you led with in the original post

bored-glass-58755 · Answer

Yeah I got that The first thing to do now is to ensure that the private repo has `wheels that pants needs`

bored-glass-58755 · Answer

and then lockfiles can be generated and then private repo must have those wheels too

bored-glass-58755 · Answer

exploding head

enough-analyst-54434 · Answer

One thing that might help at a high level is to ignore Pants here For any tool that relies on downloading things from public places as its default you ll need to 1st gather all the things it needs to download add them to private places configure the tool to look at the private places then run the tool using the private places configuration

bored-glass-58755 · Answer

Yeah

enough-analyst-54434 · Answer

The unique problems Pants presents in this scenario are 1 It provides no listing of the things it needs to download 2 It is configured in 3 places The bootstrap script pants toml and lockfiles

bored-glass-58755 · Answer

Bootstrap script for Pex download

enough-analyst-54434 · Answer

`pants` script That bootstraps pants and has configuration for the tools it needs to do the job One of which is download a Pex PEX

bored-glass-58755 · Answer

Yeah but I thought bootstrap is only to download Pex Oh wait there are those VIRTUALENV REQUIREMENTS also and not only Pex

enough-analyst-54434 · Answer

The `pants`script has two roles 1 install Pants if not already 2 run the installed Pants To do 1 it needs the Pex PEX which then downloads virtualenv and its depenedncies and then it creates a virtualenv and then installs Pants in that virtualenv using Pip

enough-analyst-54434 · Answer

gt To do 1 it needs the Pex PEX which then downloads virtualenv and its depenedncies and then it creates a virtualenv and then installs Pants in that virtualenv using Pip That is the bootstrap

bored-glass-58755 · Answer

Yeah ok

bored-glass-58755 · Answer

I have one more question My CI is bootstrapping pants just fine without the use of `PIP INDEX URL` not sure how The only place where I am stuck at is because it is unable to download anything from <http pythonhosted org|pythonhosted org> and that s where my `package` goal is failing in CI which is a linux platform What if I just do following on a linux dev box 1 Bootstrap pants which it is able to do 2 Modify `pants toml` for setuptools to generate a lockfile with PIP INDEX URL 3 Verify the generated lockfile against the private repo to ensure that I have the required wheels 4 Check this in to code repo and run CI

enough-analyst-54434 · Answer

gt My CI is bootstrapping pants just fine without the use of `PIP INDEX URL` not sure how I think you need to fully understand this If you don t the rest will be a muddled jumbled chain of hacks around this central key missing fact

bored-glass-58755 · Answer

Ok

enough-analyst-54434 · Answer

I m definitely confused at any rate Are you saying dev machines can access the internet but CI machines can t or at least partially can t I m familiar with locked down environments but not familiar with partially locked down ones dev can see internet CI cannot

bored-glass-58755 · Answer

I am new at my company and trying to figure out this too I ll come back with answers

enough-analyst-54434 · Answer

Aha Ok great Yeah knowing the ground rules would be super useful here

bored-glass-58755 · Answer

Ok so first of all I am really sorry for causing a lot of confusion It seems like CI has access to PyPI and thru that I could see it was bootstrapping pants using that source itself and was failing in `package` goal because the subprocess doesn t have access to env proxy variables I was able to go past that error by setting proxy variables `env add` under ` subprocess environment ` subsystem in `pants toml` but now `package` is throwing below ```File $XDG CACHE HOME pants named caches pex root installed wheels 6242b902db69f59e1092b406655c0fb1634486c47ce563f5fd27277cf4561822 pex 2 1 90 py2 py3 none any whl pex jobs py line 396 in handle spawn error raise self raise type self spawn error message item exception pex resolve resolvers Untranslatable Failed to spawn a job for InstallRequest target=LocalInterpreter id= usr local python 3 7 6 bin python3 7 platform=Platform platform= manylinux 2 17 x86 64 impl= cp version= 3 7 6 version info= 3 7 6 abi= cp37m marker environment=MarkerEnvironment implementation name= cpython implementation version= 3 7 6 os name= posix platform machine= x86 64 platform python implementation= CPython platform release= 3 10 0 1160 71 1 el7 x86 64 platform system= Linux platform version= 1 SMP Wed Jun 15 08 55 08 UTC 2022 python full version= 3 7 6 python version= 3 7 sys platform= linux interpreter=PythonInterpreter usr local python 3 7 6 bin python3 7 PythonIdentity usr local python 3 7 6 bin python3 7 cp37 cp37m manylinux 2 17 x86 64 3 7 6 wheel path= $XDG CACHE HOME pants named caches pex root downloads 4bdcd7d840138086126cd09254dc6195fb4fc6f01c050a1d7236f2630db1d22a wheel 0 37 1 py2 py3 none any whl fingerprint= 4bdcd7d840138086126cd09254dc6195fb4fc6f01c050a1d7236f2630db1d22a Errno 2 No such file or directory $XDG CACHE HOME pants named caches pex root venvs 5632a685cd3903cb1ab183bd3e3fb9767d9d2e42 15d3ad9788dc0da4a814dd5bb1a36eed666cd279 pex $XDG CACHE HOME pants named caches pex root venvs 5632a685cd3903cb1ab183bd3e3fb9767d9d2e42 15d3ad9788dc0da4a814dd5bb1a36eed666cd279 pex ``` Could you please help me with that

enough-analyst-54434 · Answer

No worries but it looks like you have more to work out still ```wheel path= $XDG CACHE HOME pants named caches pex root downloads 4bdcd7d840138086126cd09254dc6195fb4fc6f01c050a1d7236f2630db1d22a wheel 0 37 1 py2 py3 none any whl ```

enough-analyst-54434 · Answer

Unless you re hand editing out the real value of `$XDG CACHE HOME` in your posts it appears the env var is not getting set up correctly

bored-glass-58755 · Answer

I am just redacting the prefix essentially the path before ` pants named caches `

enough-analyst-54434 · Answer

Hrm are you positive That style error is not making sense

bored-glass-58755 · Answer

Yes I double checked Probably following might help ```$XDG CACHE HOME is some prefix work 36334 xdg cache File some prefix work 36334 xdg cache pants named caches pex root installed wheels 6242b902db69f59e1092b406655c0fb1634486c47ce563f5fd27277cf4561822 pex 2 1 90 py2 py3 none any whl pex jobs py line 396 in handle spawn error raise self raise type self spawn error message item exception pex resolve resolvers Untranslatable Failed to spawn a job for InstallRequest target=LocalInterpreter id= usr local python 3 7 6 bin python3 7 platform=Platform platform= manylinux 2 17 x86 64 impl= cp version= 3 7 6 version info= 3 7 6 abi= cp37m marker environment=MarkerEnvironment implementation name= cpython implementation version= 3 7 6 os name= posix platform machine= x86 64 platform python implementation= CPython platform release= 3 10 0 1160 71 1 el7 x86 64 platform system= Linux platform version= 1 SMP Wed Jun 15 08 55 08 UTC 2022 python full version= 3 7 6 python version= 3 7 sys platform= linux interpreter=PythonInterpreter usr local python 3 7 6 bin python3 7 PythonIdentity usr local python 3 7 6 bin python3 7 cp37 cp37m manylinux 2 17 x86 64 3 7 6 wheel path= some prefix work 36334 xdg cache pants named caches pex root downloads 4bdcd7d840138086126cd09254dc6195fb4fc6f01c050a1d7236f2630db1d22a wheel 0 37 1 py2 py3 none any whl fingerprint= 4bdcd7d840138086126cd09254dc6195fb4fc6f01c050a1d7236f2630db1d22a Errno 2 No such file or directory some prefix work 36334 xdg cache pants named caches pex root venvs 5632a685cd3903cb1ab183bd3e3fb9767d9d2e42 15d3ad9788dc0da4a814dd5bb1a36eed666cd279 pex some prefix work 36334 xdg cache pants named caches pex root venvs 5632a685cd3903cb1ab183bd3e3fb9767d9d2e42 15d3ad9788dc0da4a814dd5bb1a36eed666cd279 pex ```

bored-glass-58755 · Answer

Also I get this warning at the time of bootstrap where if the PEX ROOT is more than 128 characters Could that be an issue

enough-analyst-54434 · Answer

I doubt it Here is what would be most fruitful at this point since its hard to track where we are for me with the revelation that custom indexes aren t needed apparently Can you please do 3 things 1 Change 36334 to 36335 or anything else to effectively wipe the PEX ROOT cache in CI 2 Re run 3 Describe your complete setup and failing CI command with as full a log as you can provide This should help reset my internal debugger Right now I feel like I have no firm footing

bored-glass-58755 · Answer

I can t change that 36334 to 36335 because that s my CI s workspace directory that gets generated allocated randomly What I can do to wipe PEX ROOT cache is to not set PEX ROOT at all only set XDG CACHE HOME

bored-glass-58755 · Answer

In fact the entire path until ` xdg cache` directory is generated by CI I just set XDG CACHE HOME variable to that path and then make additional directories `pants setup` and `pants named caches pex root` to make paths `$XDG CACHE HOME pants setup` and `$XDG CACHE HOME pants named caches pex root`

enough-analyst-54434 · Answer

Ok well use your creativity but ensure the next run has a new final XDG CACHE HOME so we have a clean slate

bored-glass-58755 · Answer

Ok I can do that But do you want me to not set PEX ROOT at all

bored-glass-58755 · Answer

Hi please find attached the redacted logs for pants bootstrap and `package` goal I have also echoed the values for `$XDG CACHE HOME` amp `$PEX ROOT` I only set the former in this run and so the later is empty

enough-analyst-54434 · Answer

Ok thanks I m staring at this now Remind me of the current state of your world It looks to me like public internet access works For example see ```09 19 02 39 DEBUG Completed Downloading DownloadFile url= expected digest=FileDigest 2781255baf77c2a8fdc85c5e830f7191a6048fd91d2e20b5c7a20e5a0b7beb66 3755345 1 characters truncated ``` That s public and it worked Elsewhere there is evidence this may work because a proxy is configured ``` http proxy https proxy no proxy internal url1 internal url2 internal url3 tools that we use url ``` However elsewhere still there is still use of a private PyPI ``` no pypi index= ``` That private PyPI will be used only to create new lockfiles say for your own code s 3rdparty deps but it will not be used for existing lock files that ship with Pants for all its tools

bored-glass-58755 · Answer

Yes I have kept this private repo as part of my config but I don t think that is being used until the point I am at currently

bored-glass-58755 · Answer

I think I figured what s causing the issue However I am unable to resolve it This is going to be a long message but I will try to be as elaborate as possible Following is what I did in my CI 1 Set `$XDG CACHE HOME` to ` path to CI workspace directory work 33042 xdg cache` basically ` work 33042 ` is also a part of CI workspace path but I thought I need not redact it The directory where CI checks out the code and which becomes the repo root is ` path to CI workspace directory work 33042 s` 2 Create dirs `$XDG CACHE HOME pants setup` and `$XDG CACHE HOME pants named caches pex root` 3 Set `$PANTS SETUP CACHE` to `$XDG CACHE HOME pants setup` and `$PEX ROOT` to `$XDG CACHE HOME pants named caches pex root` 4 Bootstrap Pants 5 Do a `ls ltra path to CI workspace directory work 33042 xdg cache pants named caches pex root downloads ` because this is what the wheel path is in the failing step s output Also do a `ls ltra path to CI workspace directory work 33042 xdg cache pants named caches pex root venvs ` because this is the path to directory or file that the step is unable to find 6 Do pants tailor check 7 Repeat step 5 again Output of this step is shared below 8 Run pants ldebug package ```Starting LS step BEFORE Package Generating script ========================== Starting Command Output =========================== Today s date Thu Sep 1 13 30 36 EDT 2022 gt ls ltra path to CI workspace directory work 23715 xdg cache pants named caches pex root downloads total 0 drwxrwsr x 12 owner group 229 Sep 1 13 29 drwxrwsr x 2 owner group 10 Sep 1 13 30 gt ls ltra path to CI workspace directory work 23715 xdg cache pants named caches pex root venvs total 0 drwxrwsr x 4 owner group 75 Sep 1 13 29 drwxrwsr x 3 owner group 72 Sep 1 13 29 s drwxrwsr x 3 owner group 136 Sep 1 13 29 15158fe84406f2733a98f687f98d59238ad4210d drwxrwsr x 12 owner group 229 Sep 1 13 29 gt ls ltra path to CI workspace directory work 23715 xdg cache pants named caches pex root venvs 15158fe84406f2733a98f687f98d59238ad4210d total 0 rwxrwxr x 1 owner group 0 Sep 1 13 29 15d3ad9788dc0da4a814dd5bb1a36eed666cd279 atomic directory lck drwxrwsr x 4 owner group 75 Sep 1 13 29 drwxrwsr x 5 owner group 156 Sep 1 13 29 15d3ad9788dc0da4a814dd5bb1a36eed666cd279 drwxrwsr x 3 owner group 136 Sep 1 13 29 Finishing LS step BEFORE Package``` The `package` is failing with following missing file or directory ``` Errno 2 No such file or directory path to CI workspace directory work 4052 xdg cache pants named caches pex root venvs 5632a685cd3903cb1ab183bd3e3fb9767d9d2e42 15d3ad9788dc0da4a814dd5bb1a36eed666cd279 pex path to CI workspace directory work 4052 xdg cache pants named caches pex root venvs 5632a685cd3903cb1ab183bd3e3fb9767d9d2e42 15d3ad9788dc0da4a814dd5bb1a36eed666cd279 pex ``` The path getting generated in my CI is ` path to CI workspace directory work 4052 xdg cache pants named caches pex root venvs 15158fe84406f2733a98f687f98d59238ad4210d 15d3ad9788dc0da4a814dd5bb1a36eed666cd279 pex` Basically the directory inside the `pex root venvs ` has different name `5632a685cd3903cb1ab183bd3e3fb9767d9d2e42` that the `package` goal expects and the directory name actually present is `15158fe84406f2733a98f687f98d59238ad4210d` As a fix I tried adding following steps before running package in the above sequence of steps 7a A step `mv actual dir name present in the path expected dir name` 7b Again `ls ltra` over all the directories that I was doing before and included the directories `actual dir name present in the path` and `expected dir name` This 7b step showed that there is no `actual dir name present in the path` anymore which means that mv was successful But the package command failed even after that with the same error that directory or file is not found

bored-glass-58755 · Answer

Oh so apart from the wrong path whether in CI or the failing module the entire path is also missing the last bit of the entire path which is ` pex` Not sure if this is a directory or a file Does it make sense

enough-analyst-54434 · Answer

Is ` path to CI workspace directory work 4052 xdg cache ` known to be empty or non empty before the Pants run in CI I d like it to be empty for the purposes of debugging from a clean slate

bored-glass-58755 · Answer

I just checked following paths before running bootstrap and these are empty ` path to CI workspace directory work 33042 xdg cache pants named caches pex root downloads ` ` path to CI workspace directory work 33042 xdg cache pants named caches pex root venvs ` ` path to CI workspace directory work 33042 xdg cache pants named caches pex root venvs s` ` path to CI workspace directory work 4052 xdg cache pants named caches pex root venvs 15158fe84406f2733a98f687f98d59238ad4210d` ` path to CI workspace directory work 4052 xdg cache pants named caches pex root venvs 5632a685cd3903cb1ab183bd3e3fb9767d9d2e42` I will just check ` path to CI workspace directory work 33042 xdg cache ` now

bored-glass-58755 · Answer

Just ignore the number after `work ` in above I copied those paths from my previous long chat which had outputs from two different pipeline runs one before fix and one after fix

enough-analyst-54434 · Answer

gt Create dirs `$XDG CACHE HOME pants setup` and `$XDG CACHE HOME pants named caches pex root` When you say this you mean you re manually making these directories You should not need to do this Pants automatically makes the directories it needs if they do not exist Avoiding manually creating these directories avoids potential permission issues which it does not look like you re hitting and reduces the surface area of debugging suspects

bored-glass-58755 · Answer

Ok I started creating these directories because Pants was creating them but not in the structure `$XDG CACHE HOME pants setup` `$XDG CACHE HOME pants named caches pex root` I learnt this structure from my local Mac s bootstrapped Pants

enough-analyst-54434 · Answer

Yeah you don t want to create the directories you want to configure Pants correctly such that Pants creates those directories

enough-analyst-54434 · Answer

< bored glass 58755> I m almost completely lost at this point This is a huge thread If I had my hands on your infra I would be debugging by starting over Eliminating all customizations to scripts and pants toml and seeing what fails from a clean slate 1st with no custom env vars etc

bored-glass-58755 · Answer

Ok so how would you suggest I start over Just setting $XDG CACHE HOME and $PEX ROOT and nothing else I understood that I should not create the directories Also following is the output of directory contents of $XDG CACHE HOME before bootstrap ```Starting LS $XDG CACHE HOME before bootstrap Generating script ========================== Starting Command Output =========================== gt ls ltra path to CI workspace directory work 19378 xdg cache total 12 drwxrwxr x 7 owner group 4096 Sep 1 17 26 drwxrwxr x 3 owner group 4096 Sep 1 17 26 drwxrwxr x 4 owner group 4096 Sep 1 17 26 pants Finishing LS $XDG CACHE HOME before bootstrap```

enough-analyst-54434 · Answer

I would start from a completely clean slate Have CI try to run ` pants` using what s checked in to the repo No edits to the `pants` script and no edits `pants toml` and no custom environment variables My problem is I have lost track of what you may have modified to try to get CI working So for me starting with a completely blank slate is the only way to build back up a confident picture of the variables in play And having a confident picture of those variables is critical to debugging something 5x so when remotely debugging blind

bored-glass-58755 · Answer

Ok I tried doing that and following is the progress Let me know if you have any questions and I would try to answer them 1 Ran pants without any env variables and ` pants version` failed because the CI doesn t have perms to create `$HOME cache pants` 2 Set `$XDG CACHE HOME` to ` path to CI workspace directory work lt num gt xdg cache` and re ran ` pants version` It gave some PEX ROOT related warnings that `$HOME pex` is un writable and that it fell back to a temp directory and warned me that it will hurt performance But apart from this warning it failed this time for env vars `$LC ALL` and `$LANG` saying that they were set to `ISO 8859 1` but Pants wants `UTF 8` 3 Set `$LC ALL` and `$LANG` to `en US UTF 8` along with `$XDG CACHE HOME` as in step 2 and reran ` pants version` It bootstrapped fine but with the PEX ROOT warnings that `$HOME pex` is un writable and that it fell back to a temp directory and warned me that it will hurt performance After that the CI continued to run ` pants tailor check` which completed successfully The third step in CI was to `package` a `python distribution ` and it failed because it could not download `setuptools` and `wheel` from `<http files pythonhosted org|files pythonhosted org>` It needs proxies here because the subprocess that runs `package` doesn t have access to the default proxies that CI uses from the environment because of which it was able to bootstrap Pants in the first place 4 Set proxies under ` subprocess environment ` in `pants toml` and re ran CI This time it failed to download `setuptools` and `wheel` because of `SSL certificate` error 5 I found out the certificate path and set the variable `$PANTS CA CERTS PATH` with that path in CI and re ran This time the `package` step failed with the same error that it can t find the file or directory The directory names under the directory ` pex root venvs ` are exactly the same as before and so pasting the error here from earlier chats so please ignore the ` lt num gt ` under ` work ` so that its easier to have a look at it at a single place ``` Errno 2 No such file or directory path to CI workspace directory work 4052 xdg cache pants named caches pex root venvs 5632a685cd3903cb1ab183bd3e3fb9767d9d2e42 15d3ad9788dc0da4a814dd5bb1a36eed666cd279 pex path to CI workspace directory work 4052 xdg cache pants named caches pex root venvs 5632a685cd3903cb1ab183bd3e3fb9767d9d2e42 15d3ad9788dc0da4a814dd5bb1a36eed666cd279 pex ``` Apart from this following is `pants toml` ``` GLOBAL pants version = 2 12 0 backend packages add = pants backend python pants backend python typecheck mypy pants backend python lint flake8 pants backend experimental python lint autoflake pants backend plugin development pants backend codegen protobuf python pants backend experimental python source root patterns = test module1 module2 module3 module4 module5 subprocess environment env vars add = http proxy https proxy no proxy ``` Please let me know if anything is not clear and I can try to answer that

enough-analyst-54434 · Answer

Ok great That is very helpful Each modification has a known reason explained Thinking through what could be going on As I do that though can you do 6 `rm rf path to CI workspace directory work 4052 xdg cache pants named caches pex root` and then re run ` pants package ` Modifying that path to reflect whatever it really is There should be a bunch of deletes Perhaps use `rm rfv `to get a flood of messages from `rm` proving it is removing things

bored-glass-58755 · Answer

I introduced following step just before running the `package` goal `rm rfv $ XDG CACHE HOME pants named caches pex root` and there are two cases that happened Case 1 It seemed like that in this case the CI ran in a workspace where it had already ran sometime earlier and so it did not have to bootstrap Pants Pants was already there See the output for the ` pants version` ignore the warning ```Starting Pants bootstrap step Generating script Script contents pants version ========================== Starting Command Output =========================== 22 26 21 55 INFO Initializing scheduler 22 26 21 86 INFO Scheduler initialized 22 26 21 91 WARN Please either set `enabled = true` in the anonymous telemetry section of pants toml to enable sending anonymous stats to the Pants project to aid development or set `enabled = false` to disable it No telemetry sent for this run An explicit setting will get rid of this message See <https www pantsbuild org v2 12 docs anonymous telemetry> for details 2 12 0 Finishing Pants bootstrap step``` And please find attached the output of the delete step Case 2 This was more of a normal run where Pants bootstrapped I could see the green lines that said Bootstrapping pants Downloading PEX SHA256 Verified Installing pantsbuild pants etc I could also see the PEX ROOT un writable warning but there were no directories to delete in this run in the delete step that was introduced ```Starting Deleting $ XDG CACHE HOME pants named caches pex root Generating script ========================== Starting Command Output =========================== gt rm rfv path to CI workspace directory work 34538 xdg cache pants named caches pex root Finishing Deleting $ XDG CACHE HOME pants named caches pex root``` The `package` goal ended with the same error as before in both the cases

bored-glass-58755 · Answer

Hi I think I misunderstood your last message I think what you meant was run package then delete and then run package again If that was the case please ignore my last message Here s the output of package delete and then package again in the same CI First package failed ```File path to CI workspace directory work 13694 xdg cache pants named caches pex root installed wheels 6242b902db69f59e1092b406655c0fb1634486c47ce563f5fd27277cf4561822 pex 2 1 90 py2 py3 none any whl pex jobs py line 396 in handle spawn error raise self raise type self spawn error message item exception pex resolve resolvers Untranslatable Failed to spawn a job for InstallRequest target=LocalInterpreter id= usr local python 3 7 6 bin python3 7 platform=Platform platform= manylinux 2 17 x86 64 impl= cp version= 3 7 6 version info= 3 7 6 abi= cp37m marker environment=MarkerEnvironment implementation name= cpython implementation version= 3 7 6 os name= posix platform machine= x86 64 platform python implementation= CPython platform release= 3 10 0 1160 71 1 el7 x86 64 platform system= Linux platform version= 1 SMP Wed Jun 15 08 55 08 UTC 2022 python full version= 3 7 6 python version= 3 7 sys platform= linux interpreter=PythonInterpreter usr local python 3 7 6 bin python3 7 PythonIdentity usr local python 3 7 6 bin python3 7 cp37 cp37m manylinux 2 17 x86 64 3 7 6 wheel path= path to CI workspace directory work 13694 xdg cache pants named caches pex root downloads 4bdcd7d840138086126cd09254dc6195fb4fc6f01c050a1d7236f2630db1d22a wheel 0 37 1 py2 py3 none any whl fingerprint= 4bdcd7d840138086126cd09254dc6195fb4fc6f01c050a1d7236f2630db1d22a Errno 2 No such file or directory path to CI workspace directory work 13694 xdg cache pants named caches pex root venvs 7590ee51593acd29997ca5259a9f2ec093c0559a 15d3ad9788dc0da4a814dd5bb1a36eed666cd279 pex path to CI workspace directory work 13694 xdg cache pants named caches pex root venvs 7590ee51593acd29997ca5259a9f2ec093c0559a 15d3ad9788dc0da4a814dd5bb1a36eed666cd279 pex ``` The delete step s output is attached in a ` txt` file And lastly here s the output of the re run of the `package` goal ```File path to CI workspace directory work 13694 xdg cache pants named caches pex root installed wheels 6242b902db69f59e1092b406655c0fb1634486c47ce563f5fd27277cf4561822 pex 2 1 90 py2 py3 none any whl pex jobs py line 396 in handle spawn error raise self raise type self spawn error message item exception pex resolve resolvers Untranslatable Failed to spawn a job for InstallRequest target=LocalInterpreter id= usr local python 3 7 6 bin python3 7 platform=Platform platform= manylinux 2 17 x86 64 impl= cp version= 3 7 6 version info= 3 7 6 abi= cp37m marker environment=MarkerEnvironment implementation name= cpython implementation version= 3 7 6 os name= posix platform machine= x86 64 platform python implementation= CPython platform release= 3 10 0 1160 71 1 el7 x86 64 platform system= Linux platform version= 1 SMP Wed Jun 15 08 55 08 UTC 2022 python full version= 3 7 6 python version= 3 7 sys platform= linux interpreter=PythonInterpreter usr local python 3 7 6 bin python3 7 PythonIdentity usr local python 3 7 6 bin python3 7 cp37 cp37m manylinux 2 17 x86 64 3 7 6 wheel path= path to CI workspace directory work 13694 xdg cache pants named caches pex root downloads 4bdcd7d840138086126cd09254dc6195fb4fc6f01c050a1d7236f2630db1d22a wheel 0 37 1 py2 py3 none any whl fingerprint= 4bdcd7d840138086126cd09254dc6195fb4fc6f01c050a1d7236f2630db1d22a Errno 2 No such file or directory path to CI workspace directory work 13694 xdg cache pants named caches pex root venvs 7590ee51593acd29997ca5259a9f2ec093c0559a 15d3ad9788dc0da4a814dd5bb1a36eed666cd279 pex path to CI workspace directory work 13694 xdg cache pants named caches pex root venvs 7590ee51593acd29997ca5259a9f2ec093c0559a 15d3ad9788dc0da4a814dd5bb1a36eed666cd279 pex ``` It s weird that it did not find the directory while running the `package` goal first but then deleted the same directory in the very next step and then again failed for the same directory not found

bored-glass-58755 · Answer

Hi < enough analyst 54434> trust you are doing well May I please request your attention to these findings and help to proceed

enough-analyst-54434 · Answer

Thanks for the data At this point you ll need to roll up your sleeves I m stumped If I understand correctly this error only happens on CI On your workstation with the same configuration minus the bits you had to change above for the reasons we isolated things work This tells me as a debugger to look for what is different between the two settings We know the list of things you had to change are different but none of those seem material to this sort of issue Is there anything further different Are any of the paths in question in CI on a network mount for example

enough-analyst-54434 · Answer

Another thing you might try as a debugger is varying one thing Maybe try different releases 2 12 1 was just released and 2 13 is close to stable you might try 2 13 0rc3

bored-glass-58755 · Answer

Ok let me try to find that out with the help of infra team Yes `package` and `publish` work on my local with the exception that `publish` only works for docker targets as I don t have access to push a wheel to private repo using my user ID It has to be done via a CI which is a separate thing and not a Pants issue Let me also try to run the existing pipeline with 2 3 0rc3

bored-glass-58755 · Answer

I d like to thank you once again for all the help you provided I will try to understand how things are working here and will get back if needed

bored-glass-58755 · Answer

Do you think if this could be a bug in reporting the error message As in the real error could be something else Just like this bug here <https pantsbuild slack com archives C046T6T9U p1660918888184319>

bored-glass-58755 · Answer

Is it possible that Pants downloads a PEX version other than what is mentioned in the pants script Look at this error line where it is referring to `pex 2 1 90` but I am using the latest script that has `2 1 103` ```File path to CI workspace directory work 13694 xdg cache pants named caches pex root installed wheels 6242b902db69f59e1092b406655c0fb1634486c47ce563f5fd27277cf4561822 pex 2 1 90 py2 py3 none any whl pex jobs py line 396 in handle spawn error raise self raise type self spawn error message item exception ```

enough-analyst-54434 · Answer

No the Pex version used in the bootstrap script the pants script to install Pants bears no relation to the Pex version used by Pants once it is installed

enough-analyst-54434 · Answer

These are simply two separate pieces of software The bootstrap pants script just installs Pants one time nothing more

bored-glass-58755 · Answer

Ok

bored-glass-58755 · Answer

Are these operations async As in could this be a possibility that the job is trying to refer the directory before it gets fully created or is in the process of creation

enough-analyst-54434 · Answer

They are sync but they are racy in other words there can be 8 jobs all trying to synchronously build the same venv in parallel That though has been the case for gt 3 years and Pex has long since had in its internals an atomic directory abstraction that uses a POSIX file lock to ensure just one racing job can create a given venv directory at a time basically all the jobs view for ` path to venvs lt hash gt lck` the lock winner creates ` path to venvs lt hash gt lt uuid gt ` and populates it and then as its last 2 steps does an atomic `os rename path to venvs lt hash gt lt uuid gt path to venvs lt hash gt ` and then releases the file lock Keep in mind as you debug this the issue only happens for you in CI Presumably both your machine and CI have multiple cores It may be the case that your machine has more cores than the CI job in which case your machine is more likely to race than CI So the things to focus on are CI differences from your machine One is the POSIX file lock and that s why I asked if any CI paths were on a network mount for example Its well known that older NFS v3 or older does not support POSIX file locks

bored-glass-58755 · Answer

Ok I can see the lock file inside the venv but whether or not that is working is based on my CI infra