Has anyone seen this error? I'm getting it in a CI environment (ADO). It happens randomly / very intermittently.

$ pants publish ci/pants-test:docker
Bootstrapping Pants 2.21.0.dev4
Installing pantsbuild.pants==2.21.0.dev4 into a virtual environment at /nvme/home/agent-2/.cache/nce/60b513559c7b53eb2acecbd7b8aaaeb942686f3997d07fa77377b51324f58fda/bindings/venvs/2.21.0.dev4
Failed to create Pants virtual environment.
Error: Command '['/nvme/home/agent-2/.cache/nce/60b513559c7b53eb2acecbd7b8aaaeb942686f3997d07fa77377b51324f58fda/bindings/pex_root/venvs/5914543a6b7dd2ca81816f310ee1a60ef388f6b7/8821ef26a0029f66e0d477397bfa4d7b49254cbe/bin/python', '/nvme/tmp/agent-2/tmp4_ljhung.pex', 'venv', '--prompt', 'Pants 2.21.0.dev4', '--compile', '--pip', '--collisions-ok', '--no-emit-warnings', '--disable-cache', '/nvme/home/agent-2/.cache/nce/60b513559c7b53eb2acecbd7b8aaaeb942686f3997d07fa77377b51324f58fda/bindings/venvs/2.21.0.dev4']' returned non-zero exit status 1., output:
-----
b"[Errno 17] File exists: '__main__.py' -> '/nvme/home/agent-2/.cache/nce/60b513559c7b53eb2acecbd7b8aaaeb942686f3997d07fa77377b51324f58fda/bindings/venvs/2.21.0.dev4/pex'\n"
-----

Error: Failed to establish atomic directory /nvme/home/agent-2/.cache/nce/60b513559c7b53eb2acecbd7b8aaaeb942686f3997d07fa77377b51324f58fda/locks/install-a052fb005b23b988dfa6132b8aafb42f3420a14b954467537c3535faf0e70ccb. Population of work directory failed: Boot binding command failed: exit status: 1

Isolates your Pants from the elements.

Please select from the following boot commands:

<default> (when SCIE_BOOT is not set in the environment)  Detects the current Pants installation and launches it.
bootstrap-tools                                           Introspection tools for the Pants bootstrap process.
update                                                    Update scie-pants.

You can select a boot command by setting the SCIE_BOOT environment variable.

Do you know if the machines run many jobs concurrently? Or if your CI is doing that? Just guessing from the error that there may be multiple processes running concurrently.

The VMs run multiple concurrently, but never with the same user. In other words, each process has a dedicated user and each user does not have concurrent pants processes running.

.. and they don't share SCIE_BASE either? (ah, looking at the logged path, seems

agent-2

is a user here, which I guess is unique then..)

oh, yeah, we also have this in CI. I think we didn't have it in 2.21.0rc1, but we got it after we upgraded to 2.21.0. I'm also interested in solution

I am seeing the similar problem. @gentle-flower-25372 @square-psychiatrist-19087 did you manage to figure out the problem or solutioin?

I did "solve it" ...

# pipeline/templates/install-pants.yml

steps:
  - script: |
      set -x

      export PATH="$HOME/.local/bin:$PATH"

      which pants || curl --proto '=https' --tlsv1.2 -fsSL <https://static.pantsbuild.org/setup/get-pants.sh> | bash
      pants --version
      if [[ $? -ne 0 ]]; then
          rm -rf ~/.cache/nce
          pants --version
      fi
    displayName: install pants
    env:
      PANTS_CONFIG_FILES: $(PANTS_CONFIG_FILES)

ah ok. I am getting a similar error more consistently. thanks for sharing.

Ah that's a pain. Sorry for the trouble. If you can see it more reliably, maybe we can use that to try to debug it. 1. Can you set

RUST_LOG=debug

in the environment? That should result in a bunch of log lines like

[DEBUG TimerFinished] lift::assemble(), Elapsed=326.084µs

2. Can you run

tree .../.cache/nce

before the jobs run and see if there's a difference between the successful and unsuccessful jobs?

For sure. I will run those and provide information back.

Hm, that looks like a slightly different error

"No module named 'pex'"

. Still a problem, though... Is that the full set of logging? There's nothing before

Downloading ...

?

I just remove the nce and re-run the 2.20.

$ rm -rf ~/.cache/nce
$ ./pants --version
[DEBUG TimerFinished] lift::assemble(), Elapsed=15.967µs
[DEBUG TimerFinished] fingerprint::digest(), Elapsed=81.123µs
[DEBUG TimerFinished] lift::load_scie(), Elapsed=428.906µs
[DEBUG jump] No .env files found for invocation of /XXXX/repos/pants from cwd of Ok("/XXX/repos/")
[DEBUG TimerFinished] jump::load_dotenv, Elapsed=1.867069ms
[DEBUG TimerFinished] Context::new(), Elapsed=259.571µs
[DEBUG TimerFinished] fingerprint::digest_reader(), Elapsed=9.422835ms
[DEBUG jump::installer] The blob destination /XXX/.cache/nce/f74aecd85566838e8d82ae83a5be2f5f7e03a49385ec913b687c3f566249e024/scie-pants.bin of size 759960 had expected hash
[DEBUG TimerFinished] installer::unpack_blob(), Elapsed=28.223019ms
[DEBUG TimerFinished] Installer::install(), Elapsed=28.25975ms
[DEBUG TimerFinished] jump::prepare_boot(), Elapsed=31.545411ms
[DEBUG TimerFinished] BuildRoot::find(), Elapsed=450.57µs
[DEBUG TimerFinished] PantsConfig::parse(), Elapsed=849.52µs
[INFO  scie_pants] Found Pants build root at Some("/data/user/qj3h339/repos/atom")
[INFO  scie_pants] The required Pants version is Some("2.20.0")
[DEBUG TimerFinished] scie-pants::get_pants_process(), Elapsed=1.738471ms
[DEBUG TimerFinished] lift::assemble(), Elapsed=13.133µs
[DEBUG TimerFinished] fingerprint::digest(), Elapsed=81.911µs
[DEBUG TimerFinished] lift::load_scie(), Elapsed=404.086µs
[DEBUG jump] No .env files found for invocation of /XXX/repos/pants from cwd of Ok("/XXX/repos/")
[DEBUG TimerFinished] jump::load_dotenv, Elapsed=49.557µs
[DEBUG TimerFinished] Context::new(), Elapsed=236.033µs
[DEBUG TimerFinished] Process::fingerprint(), Elapsed=174.698µs
[DEBUG TimerFinished] fingerprint::digest_reader(), Elapsed=48.291709ms
[DEBUG jump::installer] The blob destination /XXX/.cache/nce/22652cf60b12e7a187ea0cf21d7bc9d8234bbb630fa94ca6f9c28655bd6a81fb/ptex-linux-x86_64 of size 3904904 had expected hash
[DEBUG TimerFinished] installer::unpack_blob(), Elapsed=89.642884ms
[DEBUG TimerFinished] fingerprint::digest_reader(), Elapsed=10.609293ms
[DEBUG jump::installer] The blob destination /XXX/.cache/nce/c73caed20fd04cdfc52f3148f8f556fec77d9bb12aa49c816dae14dfdfbadfd5/tools.pex of size 847798 had expected hash
[DEBUG TimerFinished] installer::unpack_blob(), Elapsed=25.616962ms
[INFO  jump::installer] Loading cpython-3.9.18+20240107-x86_64-unknown-linux-gnu-install_only.tar.gz via "/XXX/.cache/nce/22652cf60b12e7a187ea0cf21d7bc9d8234bbb630fa94ca6f9c28655bd6a81fb/ptex-linux-x86_64"...
[DEBUG TimerFinished] fingerprint::digest_reader(), Elapsed=322.949547ms
[DEBUG jump::installer] The tar.gz destination /XXX/.cache/nce/67912efc04f9156d8f5b48a0348983defb964de043b8c13ddc6cc8a002f8e691/cpython-3.9.18+20240107-x86_64-unknown-linux-gnu-install_only.tar.gz of size 25920461 had expected hash
[DEBUG TimerFinished] installer::unpack_tar(), Elapsed=8.386253288s
[DEBUG TimerFinished] installer::unpack_archive(), Elapsed=11.176649221s
[DEBUG TimerFinished] Installer::install(), Elapsed=11.292038684s
[DEBUG TimerFinished] Process::fingerprint(), Elapsed=192.15µs
[DEBUG TimerFinished] Installer::install(), Elapsed=565ns
Bootstrapping Pants 2.20.0
Installing pantsbuild.pants==2.20.0 into a virtual environment at /XXX/.cache/nce/f60c448851c3b3112b179a56f168ee90d8597297cfc45d3ca7178c6715f6cba8/bindings/venvs/2.20.0
Traceback (most recent call last):
  File "/XXX/.cache/nce/67912efc04f9156d8f5b48a0348983defb964de043b8c13ddc6cc8a002f8e691/cpython-3.9.18+20240107-x86_64-unknown-linux-gnu-install_only.tar.gz/python/lib/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/XXX/.cache/nce/67912efc04f9156d8f5b48a0348983defb964de043b8c13ddc6cc8a002f8e691/cpython-3.9.18+20240107-x86_64-unknown-linux-gnu-install_only.tar.gz/python/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/XXX/.pex/unzipped_pexes/9c188bd7041127af5913d880aac8c417079dfb56/__main__.py", line 105, in <module>
    from pex.pex_bootstrapper import bootstrap_pex
ModuleNotFoundError: No module named 'pex'
Install failed: Command '['/XXX/.cache/nce/f60c448851c3b3112b179a56f168ee90d8597297cfc45d3ca7178c6715f6cba8/bindings/pex_root/venvs/b9a9cccc22b667334dfe5c969fd5448be1098f18/64a0d12156ca69cfbdafc968a6fcac5ef2dec80d/bin/python', '/tmp/tmplqun2l87.pex', 'venv', '--prompt', 'Pants 2.20.0', '--compile', '--pip', '--collisions-ok', '--no-emit-warnings', '--disable-cache', '/XXX/.cache/nce/f60c448851c3b3112b179a56f168ee90d8597297cfc45d3ca7178c6715f6cba8/bindings/venvs/2.20.0']' returned non-zero exit status 1.
More information can be found in the log at: /XXX/.cache/nce/f60c448851c3b3112b179a56f168ee90d8597297cfc45d3ca7178c6715f6cba8/bindings/logs/install.log

[DEBUG TimerFinished] jump::prepare_boot(), Elapsed=21.494126561s
Error: Failed to establish atomic directory /XXX/.cache/nce/f60c448851c3b3112b179a56f168ee90d8597297cfc45d3ca7178c6715f6cba8/locks/install-2ee8683cd8fc4a4e90b8c26e5ea73f77a1fc800868b4b72c4e57e384cd7ae0ae. Population of work directory failed: Boot binding command failed: exit status: 1

Isolates your Pants from the elements.

Please select from the following boot commands:

<default> (when SCIE_BOOT is not set in the environment)  Detects the current Pants installation and launches it.
bootstrap-tools                                           Introspection tools for the Pants bootstrap process.
update                                                    Update scie-pants.You can select a boot command by setting the SCIE_BOOT environment variable.

Hm, digging around the issue tracker suggests that the

No module named 'pex'

issue could be related to a partially-deleted cache. https://github.com/pantsbuild/pants/pull/20385 is a doc update relevant to that. Can you try the same command removing

~/.pex

which seems to be a path also appearing there?

i did try that before. Everytime I got issue, the last resort is to clean nce/pex/pants cache 🙂 .But encountered the same issue.

Ah okay. That's unfortunate. I'm not exactly sure how to debug it