Is there a way to check what

pants.toml

options are in effect during a particular run of Pants e.g. if I am running

./pants package

, I would like to see what

python-repos

options are active.

The most sure footed way I can think of is adding

-ldebug

to the Pants command line. This prints out the full set of environment variables and argv for every process Pants invokes.

You can also run

./pants help python-repos

and the configured value should be listed next to every option.

Thanks @enough-analyst-54434 and @fast-nail-55400. I am going to try both of these.

Tom's method is what you want to normally do. If it turns out that method doesn't give an answer that corresponds to behavior you're observing, the -ldebug will help sort that out. We've had some churn recently in options affecting repos and I'm not confident in the status of the plumbing to various resolves. The thrash may have all been in ideas though with pen never actually hitting paper.

Ok, so here's what

./pants help python-repos

outputs:

`python-repos` subsystem options
--------------------------------

External Python code repositories, such as PyPI.

These options may be used to point to custom cheeseshops when resolving requirements.
 
Activated by pants.core
Config section: [python-repos]
 
None available.
Advanced options available. You can list them by running ./pants help-advanced python-repos.

It doesn't show me anything even though I have the env variable

PANTS_PYTHON_REPOS_INDEXES

set in my CI along with the option in

pants.toml

.

It's probably an "advanced" option, although that classification is odd. Although its probably advanced in a global sense, the minute you're in the scope of the

python-repos

subsystem, those options are bread and butter. Try

./pants help-advanced python-repos

Ok, so here's the output:

`python-repos` subsystem advanced options
-----------------------------------------
 
  --python-repos-repos="['<str>', '<str>', ...]"
  PANTS_PYTHON_REPOS_REPOS
  repos
      default: []
      current value: []
      URLs of code repositories to look for requirements. In Pip and Pex, this option
      corresponds to the `--find-links` option.

  --python-repos-indexes="['<str>', '<str>', ...]"
  PANTS_PYTHON_REPOS_INDEXES
  indexes
      default: [
          "<https://pypi.org/simple/>"
      ]
      current value: [
          "<http://private.repo.url/simple>",
          "<http://private.repo.url/simple>"
      ] (from pants.toml, from env var PANTS_PYTHON_REPOS_INDEXES)
      URLs of code repository indexes to look for requirements. If set to an empty list, then
      Pex will use no indices (meaning it will not use PyPI). The values should be compliant
      with PEP 503.


`python-repos` subsystem deprecated options
-------------------------------------------
 
None available.

The default is still set to pypi, could that be a problem?

No, the default will always be that.

The behavior suggests:

pex.result.ResultError: There were 2 errors downloading required artifacts:
1. setuptools 57.5 from <https://files.pythonhosted.org/packages/4b/b9/71687c5d2034c863db1db2e0704f5e27581ff3cb44d7f293968c5e08ceb3/setuptools-57.5.0-py3-none-any.whl>
    <urlopen error [Errno 97] Address family not supported by protocol>
2. wheel 0.37.1 from <https://files.pythonhosted.org/packages/27/d6/003e593296a85fd6ed616ed962795b2f87709c3eee2bca4f6d0fe55c6d00/wheel-0.37.1-py2.py3-none-any.whl>
    <urlopen error [Errno 97] Address family not supported by protocol>

That

PEX_ROOT

is a relative path in the sandbox to a symlink that points to

$XDG_CACHE_HOME/pants/named_caches/pex_root

. No matter how you configure `PEX_ROOT`externally, Pants will always use that local sandbox path with a symlink to where you set it to.

Thanks a lot for the lead, I'll let you know of the progress I make.

Yes. It can be anywhere under the repo root however you see fit.

Ok, so I executed the command

PIP_INDEX_URL=<http://private.repo.url/simple> ./pants --version

and it ended as following:

Installing pantsbuild.pants==2.12.0 into a virtual environment at /$XDG_CACHE_HOME/pants/setup/bootstrap-Linux-x86_64/2.12.0_py37
WARNING: The repository located at private.repo.url is not a trusted or secure host and is being ignored. If this repository is available via HTTPS we recommend you use HTTPS instead, otherwise you may silence this warning and allow it anyway with '--trusted-host private.repo.url'.
WARNING: The repository located at private.repo.url is not a trusted or secure host and is being ignored. If this repository is available via HTTPS we recommend you use HTTPS instead, otherwise you may silence this warning and allow it anyway with '--trusted-host private.repo.url'.
ERROR: Could not find a version that satisfies the requirement setuptools<58 (from versions: none)
ERROR: No matching distribution found for setuptools<58
WARNING: The repository located at private.repo.url is not a trusted or secure host and is being ignored. If this repository is available via HTTPS we recommend you use HTTPS instead, otherwise you may silence this warning and allow it anyway with '--trusted-host private.repo.url'.
##[error]Bash exited with code '1'.

If lines #2 & #3 are just warnings and if it did not use

private.repo.url

, why could not it download setuptools just how it was doing it before passing in the

PIP_INDEX_URL

? Also, the only change I have made is in

pants.toml

by introducing the

[setuptools]

subsystem with

lockfile = tools/setuptools.lock

. I haven't added any other subsystems and also haven't made any change to pants script as it shows in the GitHub link you provided (in function

bootstrap_virtualenv()

).

Perhaps add PIP_VERBOSE=3 to see more.

Ok, but the changes I described you above look fine?

Probably. A diff would be easier to validate. That said - you need not have edited the script for the experiment you could have stuck with

PIP_<option>

Yeah, I wish I could share the diff. Regarding the script, so I need not even make this change (which I haven't done anyway, but just wanted to confirm):

@@ -270,7 +270,9 @@ function bootstrap_virtualenv {
       staging_dir=$(tempdir "${PANTS_BOOTSTRAP}")
       cd "${staging_dir}"
       echo "${VIRTUALENV_REQUIREMENTS}" > requirements.txt
-      "${python}" "${pex_path}" -r requirements.txt -c virtualenv -o virtualenv.pex
+      "${python}" "${pex_path}" -r requirements.txt -c virtualenv -o virtualenv.pex \
+        --no-pypi \
+        --index <https://pextest.jfrog.io/artifactory/api/pypi/default-pypi/simple/>

Correct. If you're just experimenting you can use the `PIP_*`env vars. If you want to write it all down, you might then want to edit the script unless you have a mechanism like direnv to write down env vars to activate whenever using Pants.

Ok

it couldn't

find pantsbuild.pants==2.12.0

in there which is expected.

Does this mean http://private.repo.url/simple is not a mirror of PyPI, aka it is missing some distributions that are otherwise on PyPI?

Yes, it's not a mirror.

Oh wow, this is going to be painful.

Ok, I completely understood the first part related to bootstrapping pants with only using the private repo. I am trying to look at the second part regarding lockfiles. Regarding lockfiles, do we need to generate them every time CI runs or we can generate them once and then have them checked in thru git? If its later, then maybe I can generate them once and check it in and then bootstrap pants normally.

The latter. But to even re-generate them, the repo at http://private.repo.url/simple will need to have all the required distributions already available! If not, the lock re-gen will fail. As such, I think you proetty much need to read through the default lock file checked into the Pants project and grab the full list of wheels and use that to pre-populate your private index.

Ok, I'll check the lock file in Pants project. But if its the latter. then why do I need to re-generate them as long as I am using the same pants version?

Because the lock files have URLs in them and they are the URLs of Open source artifacts.

Ok, what I am thinking is: 1. Getting a list of wheels for pantsbuild.pants on my local dev machine and check which ones need to be onboarded to the private repo. 2. Once I have them onboarded, I can bootstrap pants on my local using the PIP_INDEX_URL. 3. The third step would be to generate-lockfiles and check them to the code repo. And then when CI runs, it would use the lockfiles I generated and checked in (would it after I bootstrap pants again in CI?) Of course, I need to repeat this exercise for every platform I am expected to work on. The best would be to do this exercise only once on an environment that resembles the CI's environment and have it run.

Step 3 is a one time thing. The generated lock files are universal and will have listed all wheels and sdists needed for any platform. As such, you cannot generate a proper lockfile until your private index has all needed wheels and sdists! Concretely, say your CI is locked down to 1 Linux distribution - CentOS 7, and 1 Python interpreter, Python 3.8.10 and then you have devs working purely on Mac, but some intel and some arm. Lets further say all devs also use Python 3.8.10. That means you have 3 platforms to account for. If so then your steps are, in order: 1. For each platform:

pip wheel pantsbuild.pants

and gather wheels and put them in your private index. 2. For each platform and for each (Python) tool Pants uses: Find the default tool lockfile checked into the Pants repo and gather all the wheels and sdists listed that apply to your platforms and put them in your private index. 3. Now bootstrap Pants using the private index and run

./pants generate-lockfiles

then check in all the lock files. At this point Panst can run on dev machines and in CI.

Ok, so to check if I understand it well - let’s assume that the only wheels that it currently needs to

package

and tries to download from pythonhosted.org are

setuptools

and

wheel

(as demonstrated in the starting message of this thread). I assume those wheels are present in the default

setuptools.lock

file in the Pants repo and are also present in my private repo. If I download that file and replace the pythonhosted.org URL with my private.repo.url along with versions and hashes that it needs and check this modified

setuptools.lock

file in to my code repo under

tools/

directory. Thereafter, I have my

pants.toml

modified to have

[setuptools]

subsystem with line

lockfile = "tools/setuptools.lock"

. After all this, I assume - if I understand it all correctly - that my

package

goal would run successfully.

That is correct, but hand-editing the lock json is not a good idea. Instead, you download all the wheels by hand, edit pants.toml to have a custom

lockfile = ...

and then run

./pants generate-lockfiles

Ok, I am going to hand edit json for now for just those two packages.

./pants generate-lockfiles

will generate lockfiles for every subsystem in

pants.toml

that has an explicit non-default `lockfile`defined. So, by default it will do nothing. It only does things once you have one or more non-default lockfile defined. This is additional reason why this is all so paintful right now. There is no obvious list of which subsystems you use and thus which ones you'll need to define a custom `lockfile`location for.

Yeah, I got that. The first thing to do now is to ensure that the private repo has

wheels-that-pants-needs

One thing that might help at a high level is to ignore Pants here. For any tool that relies on downloading things from public places as its default, you'll need to 1st gather all the things it needs to download, add them to private places, configure the tool to look at the private places, then run the tool using the private places configuration.

Yeah

The unique problems Pants presents in this scenario are: 1. It provides no listing of the things it needs to download. 2. It is configured in 3 places: The bootstrap script, pants.toml and lockfiles.

Bootstrap script for Pex download?

pants

script. That bootstraps pants and has configuration for the tools it needs to do the job. One of which is download a Pex PEX.

Yeah, but I thought bootstrap is only to download Pex. Oh wait, there are those VIRTUALENV_REQUIREMENTS also and not only Pex.

The `pants`script has two roles, 1: install Pants if not already, 2: run the installed Pants. To do 1 it needs the Pex PEX which then downloads virtualenv and its depenedncies and then it creates a virtualenv and then installs Pants in that virtualenv using Pip.

Yeah, ok!

My CI is bootstrapping pants just fine without the use of

PIP_INDEX_URL

(not sure how).

I think you need to fully understand this. If you don't the rest will be a muddled jumbled chain of hacks around this central key missing fact.

Ok

I'm definitely confused at any rate. Are you saying dev machines can access the internet but CI machines can't - or at least partially can't? I'm familiar with locked down environments but not familiar with partially locked down ones - dev can see internet, CI cannot.

I am new at my company and trying to figure out this too. I'll come back with answers.

Aha! Ok, great. Yeah, knowing the ground rules would be super useful here.

Ok, so first of all I am really sorry for causing a lot of confusion. It seems like CI has access to PyPI (and files.pythonhosted.org thru that). I could see it was bootstrapping pants using that source itself and was failing in

package

goal because the subprocess doesn't have access to env proxy variables: https://github.com/pantsbuild/pants/issues/15675. I was able to go past that error by setting proxy variables

env.add

under

[subprocess-environment]

subsystem in

pants.toml

but now

package

is throwing below:

File "/$XDG_CACHE_HOME/pants/named_caches/pex_root/installed_wheels/6242b902db69f59e1092b406655c0fb1634486c47ce563f5fd27277cf4561822/pex-2.1.90-py2.py3-none-any.whl/pex/jobs.py", line 396, in handle_spawn_error
    raise self._raise_type(self.spawn_error_message(item, exception))
pex.resolve.resolvers.Untranslatable: Failed to spawn a job for InstallRequest(target=LocalInterpreter(id='usr.local.python.3.7.6.bin.python3.7', platform=Platform(platform='manylinux_2_17_x86_64', impl='cp', version='3.7.6', version_info=(3, 7, 6), abi='cp37m'), marker_environment=MarkerEnvironment(implementation_name='cpython', implementation_version='3.7.6', os_name='posix', platform_machine='x86_64', platform_python_implementation='CPython', platform_release='3.10.0-1160.71.1.el7.x86_64', platform_system='Linux', platform_version='#1 SMP Wed Jun 15 08:55:08 UTC 2022', python_full_version='3.7.6', python_version='3.7', sys_platform='linux'), interpreter=PythonInterpreter('/usr/local/python/3.7.6/bin/python3.7', PythonIdentity('/usr/local/python/3.7.6/bin/python3.7', 'cp37', 'cp37m', 'manylinux_2_17_x86_64', (3, 7, 6)))), wheel_path='/$XDG_CACHE_HOME/pants/named_caches/pex_root/downloads/4bdcd7d840138086126cd09254dc6195fb4fc6f01c050a1d7236f2630db1d22a/wheel-0.37.1-py2.py3-none-any.whl', fingerprint='4bdcd7d840138086126cd09254dc6195fb4fc6f01c050a1d7236f2630db1d22a'): [Errno 2] No such file or directory: '/$XDG_CACHE_HOME/pants/named_caches/pex_root/venvs/5632a685cd3903cb1ab183bd3e3fb9767d9d2e42/15d3ad9788dc0da4a814dd5bb1a36eed666cd279/pex': '/$XDG_CACHE_HOME/pants/named_caches/pex_root/venvs/5632a685cd3903cb1ab183bd3e3fb9767d9d2e42/15d3ad9788dc0da4a814dd5bb1a36eed666cd279/pex'

Could you please help me with that?

No worries, but it looks like you have more to work out still:

wheel_path='/$XDG_CACHE_HOME/pants/named_caches/pex_root/downloads/4bdcd7d840138086126cd09254dc6195fb4fc6f01c050a1d7236f2630db1d22a/wheel-0.37.1-py2.py3-none-any.whl'

I am just redacting the prefix - essentially the path before

/pants/named_caches/...

Hrm, are you positive? That style error is not making sense.

Yes, I double checked. Probably following might help?

$XDG_CACHE_HOME is /some/prefix/_work/36334/.xdg-cache


File "/some/prefix/_work/36334/.xdg-cache/pants/named_caches/pex_root/installed_wheels/6242b902db69f59e1092b406655c0fb1634486c47ce563f5fd27277cf4561822/pex-2.1.90-py2.py3-none-any.whl/pex/jobs.py", line 396, in handle_spawn_error
    raise self._raise_type(self.spawn_error_message(item, exception))
pex.resolve.resolvers.Untranslatable: Failed to spawn a job for InstallRequest(target=LocalInterpreter(id='usr.local.python.3.7.6.bin.python3.7', platform=Platform(platform='manylinux_2_17_x86_64', impl='cp', version='3.7.6', version_info=(3, 7, 6), abi='cp37m'), marker_environment=MarkerEnvironment(implementation_name='cpython', implementation_version='3.7.6', os_name='posix', platform_machine='x86_64', platform_python_implementation='CPython', platform_release='3.10.0-1160.71.1.el7.x86_64', platform_system='Linux', platform_version='#1 SMP Wed Jun 15 08:55:08 UTC 2022', python_full_version='3.7.6', python_version='3.7', sys_platform='linux'), interpreter=PythonInterpreter('/usr/local/python/3.7.6/bin/python3.7', PythonIdentity('/usr/local/python/3.7.6/bin/python3.7', 'cp37', 'cp37m', 'manylinux_2_17_x86_64', (3, 7, 6)))), wheel_path='/some/prefix/_work/36334/.xdg-cache/pants/named_caches/pex_root/downloads/4bdcd7d840138086126cd09254dc6195fb4fc6f01c050a1d7236f2630db1d22a/wheel-0.37.1-py2.py3-none-any.whl', fingerprint='4bdcd7d840138086126cd09254dc6195fb4fc6f01c050a1d7236f2630db1d22a'): [Errno 2] No such file or directory: '/some/prefix/_work/36334/.xdg-cache/pants/named_caches/pex_root/venvs/5632a685cd3903cb1ab183bd3e3fb9767d9d2e42/15d3ad9788dc0da4a814dd5bb1a36eed666cd279/pex': '/some/prefix/_work/36334/.xdg-cache/pants/named_caches/pex_root/venvs/5632a685cd3903cb1ab183bd3e3fb9767d9d2e42/15d3ad9788dc0da4a814dd5bb1a36eed666cd279/pex'

I doubt it. Here is what would be most fruitful at this point since its hard to track where we are (for me) with the revelation that custom indexes aren't needed apparently. Can you please do 3 things: 1. Change 36334 to 36335 or anything else to effectively wipe the PEX_ROOT cache in CI. 2. Re-run. 3. Describe your complete setup and failing CI command with as full a log as you can provide. This should help reset my internal debugger. Right now I feel like I have no firm footing.

I can't change that 36334 to 36335 because that's my CI's workspace directory that gets generated/allocated randomly. What I can do to wipe PEX_ROOT cache is to not set PEX_ROOT at all (only set XDG_CACHE_HOME).

Ok, well use your creativity but ensure the next run has a new final XDG_CACHE_HOME so we have a clean slate.

Ok, I can do that. But do you want me to not set PEX_ROOT at all?

Ok, thanks. I'm staring at this now. Remind me of the current state of your world? It looks to me like public internet access works. For example see:

09:19:02.39 [DEBUG] Completed: Downloading: DownloadFile(url='<https://github.com/pantsbuild/pex/releases/download/v2.1.90/pex>', expected_digest=FileDigest('2781255baf77c2a8fdc85c5e830f7191a6048fd91d2e20b5c7a20e5a0b7beb66', 3755345)... (1 characters truncated)

That's public and it worked. Elsewhere there is evidence this may work because a proxy is configured:

"http_proxy": "<http://corp.proxy.com:8080>", "https_proxy": "<http://corp.proxy.com:8080>", "no_proxy": "internal.url1,internal.url2,internal.url3,tools-that-we-use.url"

However, elsewhere still there is still use of a private PyPI:

"--no-pypi", "--index=<http://private.repo.url/simple>"

That private PyPI will be used only to create new lockfiles - say for your own code's 3rdparty deps - but it will not be used for existing lock files that ship with Pants for all its tools.

Yes, I have kept this private repo as part of my config but I don't think that is being used until the point I am at currently.

Is

/path/to/CI/workspace/directory/_work/4052/.xdg-cache/

known to be empty or non-empty before the Pants run in CI? I'd like it to be empty for the purposes of debugging from a clean slate.

I just checked following paths before running bootstrap and these are empty:

/path/to/CI/workspace/directory/_work/33042/.xdg-cache/pants/named_caches/pex_root/downloads/

/path/to/CI/workspace/directory/_work/33042/.xdg-cache/pants/named_caches/pex_root/venvs/

/path/to/CI/workspace/directory/_work/33042/.xdg-cache/pants/named_caches/pex_root/venvs/s

/path/to/CI/workspace/directory/_work/4052/.xdg-cache/pants/named_caches/pex_root/venvs/15158fe84406f2733a98f687f98d59238ad4210d

/path/to/CI/workspace/directory/_work/4052/.xdg-cache/pants/named_caches/pex_root/venvs/5632a685cd3903cb1ab183bd3e3fb9767d9d2e42

I will just check

/path/to/CI/workspace/directory/_work/33042/.xdg-cache/

now

Create dirs

$XDG_CACHE_HOME/pants/setup

and

$XDG_CACHE_HOME/pants/named_caches/pex_root

.

When you say this, you mean you're manually making these directories? You should not need to do this. Pants automatically makes the directories it needs if they do not exist. Avoiding manually creating these directories avoids potential permission issues (which it does not look like you're hitting) and reduces the surface area of debugging suspects.

Ok, I started creating these directories because Pants was creating them but not in the structure:

$XDG_CACHE_HOME/pants/setup

$XDG_CACHE_HOME/pants/named_caches/pex_root

I learnt this structure from my local Mac's bootstrapped Pants.

Yeah, you don't want to create the directories, you want to configure Pants correctly such that Pants creates those directories.

Ok, so how would you suggest I start over? Just setting $XDG_CACHE_HOME and $PEX_ROOT and nothing else? I understood that I should not create the directories. Also, following is the output of directory contents of $XDG_CACHE_HOME before bootstrap:

Starting: LS $XDG_CACHE_HOME before bootstrap
Generating script.
========================== Starting Command Output ===========================
> ls -ltra /path/to/CI/workspace/directory/_work/19378/.xdg-cache
total 12
drwxrwxr-x. 7 owner group 4096 Sep  1 17:26 ..
drwxrwxr-x. 3 owner group 4096 Sep  1 17:26 .
drwxrwxr-x. 4 owner group 4096 Sep  1 17:26 pants
Finishing: LS $XDG_CACHE_HOME before bootstrap

I would start from a completely clean slate. Have CI try to run

./pants

using what's checked in to the repo. No edits to the

pants

script and no edits

pants.toml

and no custom environment variables. My problem is I have lost track of what you may have modified to try to get CI working. So, for me, starting with a completely blank slate is the only way to build back up a confident picture of the variables in play. And having a confident picture of those variables is critical to debugging something, 5x so when remotely debugging blind.

Ok, I tried doing that and following is the progress. Let me know if you have any questions and I would try to answer them: 1. Ran pants without any env variables and

./pants --version

failed because the CI doesn't have perms to create

$HOME/.cache/pants

. 2. Set

$XDG_CACHE_HOME

to

/path/to/CI/workspace/directory/work/<num>/_._xdg_cache

and re-ran

./pants --version

. It gave some PEX_ROOT related warnings that

$HOME/.pex

is un-writable and that it fell back to a temp directory and warned me that it will hurt performance. But apart from this warning, it failed this time for env vars

$LC_ALL

and

$LANG

saying that they were set to

ISO-8859-1

but Pants wants

UTF-8

. 3. Set

$LC_ALL

and

$LANG

to

en_US.UTF-8

along with

$XDG_CACHE_HOME

as in step #2 and reran

./pants --version

. It bootstrapped fine but with the PEX_ROOT warnings that

$HOME/.pex

is un-writable and that it fell back to a temp directory and warned me that it will hurt performance. After that the CI continued to run

./pants tailor --check

which completed successfully. The third step in CI was to

package

a

python_distribution()

and it failed because it could not download

setuptools

and

wheel

from

<http://files.pythonhosted.org|files.pythonhosted.org>

. It needs proxies here because the subprocess that runs

package

doesn't have access to the default proxies that CI uses from the environment (because of which it was able to bootstrap Pants in the first place). 4. Set proxies under

[subprocess-environment]

in

pants.toml

and re-ran CI. This time it failed to download

setuptools

and

wheel

because of

SSL certificate

error. 5. I found out the certificate path and set the variable

$PANTS_CA_CERTS_PATH

with that path in CI and re-ran. This time the

package

step failed with the same error that it can't find the file or directory. The directory names under the directory

/pex_root/venvs/

are exactly the same as before and so, pasting the error here from earlier chats (so please ignore the

<num>

under

_work/

) so that its easier to have a look at it at a single place:

[Errno 2] No such file or directory: '/path/to/CI/workspace/directory/_work/4052/.xdg-cache/pants/named_caches/pex_root/venvs/5632a685cd3903cb1ab183bd3e3fb9767d9d2e42/15d3ad9788dc0da4a814dd5bb1a36eed666cd279/pex': '/path/to/CI/workspace/directory/_work/4052/.xdg-cache/pants/named_caches/pex_root/venvs/5632a685cd3903cb1ab183bd3e3fb9767d9d2e42/15d3ad9788dc0da4a814dd5bb1a36eed666cd279/pex'

Apart from this, following is `pants.toml`:

[GLOBAL]
pants_version = "2.12.0"

backend_packages.add = [
  "pants.backend.python",
  "pants.backend.python.typecheck.mypy",
  "pants.backend.python.lint.flake8",
  "pants.backend.experimental.python.lint.autoflake",
  "pants.backend.plugin_development",
  "pants.backend.codegen.protobuf.python",
  "pants.backend.experimental.python"
]

[source]
root_patterns = [
  "/",
  "/test",
  "/module1",
  "/module2",
  "/module3",
  "/module4",
  "/module5"
]

[subprocess-environment]
env_vars.add = ["http_proxy", "https_proxy", "no_proxy"]

Please let me know if anything is not clear and I can try to answer that.

Ok, great. That is very helpful. Each modification has a known reason explained. Thinking through what could be going on. As I do that though, can you do? 6.

rm -rf /path/to/CI/workspace/directory/_work/4052/.xdg-cache/pants/named_caches/pex_root

and then re-run

./pants package ...

(Modifying that path to reflect whatever it really is. There should be a bunch of deletes. Perhaps use `rm -rfv ...`to get a flood of messages from

rm

proving it is removing things.

I introduced following step just before running the

package

goal:

rm -rfv ${XDG_CACHE_HOME}/pants/named_caches/pex_root

and there are two cases that happened: Case 1: It seemed like that in this case, the CI ran in a workspace where it had already ran (sometime) earlier and so, it did not have to "bootstrap" Pants. Pants was already there. See the output for the

./pants --version

(ignore the warning):

Starting: Pants bootstrap step
Generating script.
Script contents:
./pants --version
========================== Starting Command Output ===========================
22:26:21.55 [INFO] Initializing scheduler...
22:26:21.86 [INFO] Scheduler initialized.
22:26:21.91 [WARN] Please either set `enabled = true` in the [anonymous-telemetry] section of pants.toml to enable sending anonymous stats to the Pants project to aid development, or set `enabled = false` to disable it. No telemetry sent for this run. An explicit setting will get rid of this message. See <https://www.pantsbuild.org/v2.12/docs/anonymous-telemetry> for details.
2.12.0
Finishing: Pants bootstrap step

And please find attached the output of the delete step. Case 2: This was more of a normal run where Pants bootstrapped (I could see the green lines that said Bootstrapping pants, Downloading PEX, SHA256 Verified, Installing pantsbuild.pants, etc. I could also see the PEX_ROOT un-writable warning) but there were no directories to delete in this run in the delete step that was introduced:

Starting: Deleting ${XDG_CACHE_HOME}/pants/named_caches/pex_root
Generating script.
========================== Starting Command Output ===========================
> rm -rfv /path/to/CI/workspace/directory/_work/34538/.xdg-cache/pants/named_caches/pex_root
Finishing: Deleting ${XDG_CACHE_HOME}/pants/named_caches/pex_root

The

package

goal ended with the same error as before in both the cases.

Thanks for the data. At this point you'll need to roll up your sleeves - I'm stumped. If I understand correctly, this error only happens on CI. On your workstation with the same configuration (minus the bits you had to change above for the reasons we isolated) things work. This tells me, as a debugger, to look for what is different between the two settings. We know the list of things you had to change are different, but none of those seem material to this sort of issue. Is there anything further different? Are any of the paths in question in CI on a network mount for example?

Ok, let me try to find that out with the help of infra team. Yes,

package

and

publish

work on my local with the exception that

publish

only works for docker targets as I don't have access to push a wheel to private repo using my user ID. It has to be done via a CI which is a separate thing and not a Pants issue. Let me also try to run the existing pipeline with 2.3.0rc3.

No, the Pex version used in the bootstrap script (the pants script) to install Pants bears no relation to the Pex version used by Pants once it is installed.

Ok

They are sync, but they are racy - in other words there can be 8 jobs all trying to synchronously build the same venv in parallel. That, though, has been the case for >3 years and Pex has long since had in its internals an atomic directory abstraction that uses a POSIX file lock to ensure just one racing job can create a given venv directory at a time, basically all the jobs view for

/path/to/venvs/.<hash>.lck

, the lock winner creates

/path/to/venvs/<hash>.<uuid>

and populates it, and then as its last 2 steps, does an atomic

os.rename("/path/to/venvs/<hash>.<uuid>", "/path/to/venvs/<hash>")

and then releases the file lock. Keep in mind as you debug this - the issue only happens for you in CI. Presumably both your machine and CI have multiple cores. It may be the case that your machine has more cores than the CI job, in which case your machine is more likely to race than CI! So the things to focus on are CI differences from your machine. One is the POSIX file lock and that's why I asked if any CI paths were on a network mount for example. Its "well known" that older NFS (v3 or older) does not support POSIX file locks.

Ok, I can see the lock file inside the venv but whether or not that is working is based on my CI infra.