https://pantsbuild.org/ logo
Join Slack
Powered by
Is there a way to get more detailed reporting out ...
# general
b
Is there a way to get more detailed reporting out of the 
__run.sh
script that executes in the Pants sandbox? I was trying to debug a failure and saved the sandbox, but when I try to run 
__run.sh
I get this error (it fails trying to install the latest version of 
pip
):
Copy code
received exit code 1 during execution of `['/private/var/folders/0w/zhpm9fs12zzcmt3m5799xb9c0000gn/T/pants-sandbox-42OgcN/.tmp/tmp5vgvkisc/pip/bin/python3.12', '-s', '-E', '-m', 'pip', 'install', '--target', '/private/var/folders/0w/zhpm9fs12zzcmt3m5799xb9c0000gn/T/pants-sandbox-42OgcN/.tmp/tmp5vgvkisc/reqs/pip', 'pip==24.0']` while trying to execute `['/private/var/folders/0w/zhpm9fs12zzcmt3m5799xb9c0000gn/T/pants-sandbox-42OgcN/.tmp/tmp5vgvkisc/pip/bin/python3.12', '-s', '-E', '-m', 'pip', 'install', '--target', '/private/var/folders/0w/zhpm9fs12zzcmt3m5799xb9c0000gn/T/pants-sandbox-42OgcN/.tmp/tmp5vgvkisc/reqs/pip', 'pip==24.0']`
✅ 1
Found the 
PEX_VERBOSE
setting for PEX but that didn’t get me any closer to finding a root cause. Here’s the stack trace:
Copy code
pex: Executing: /private/var/folders/0w/zhpm9fs12zzcmt3m5799xb9c0000gn/T/pants-sandbox-42OgcN/.tmp/tmp0i26txax/pip/bin/python3.12 -s -E -m pip install --target /private/var/folders/0w/zhpm9fs12zzcmt3m5799xb9c0000gn/T/pants-sandbox-42OgcN/.tmp/tmp0i26txax/reqs/pip pip==24.0
Traceback (most recent call last):
  File "/Users/krishnanchandra/.cache/pants/named_caches/pex_root/installed_wheels/7309da7faa3a99f960f878ee917815b3ceb40d69205b7d496d88faa5a8c4db76/pex-2.2.1-py2.py3-none-any.whl/pex/result.py", line 105, in catch
    return func(*args, **kwargs)
  File "/Users/krishnanchandra/.cache/pants/named_caches/pex_root/installed_wheels/7309da7faa3a99f960f878ee917815b3ceb40d69205b7d496d88faa5a8c4db76/pex-2.2.1-py2.py3-none-any.whl/pex/bin/pex.py", line 1045, in do_main
    pex_builder = build_pex(
  File "/Users/krishnanchandra/.cache/pants/named_caches/pex_root/installed_wheels/7309da7faa3a99f960f878ee917815b3ceb40d69205b7d496d88faa5a8c4db76/pex-2.2.1-py2.py3-none-any.whl/pex/bin/pex.py", line 912, in build_pex
    resolve(
  File "/Users/krishnanchandra/.cache/pants/named_caches/pex_root/installed_wheels/7309da7faa3a99f960f878ee917815b3ceb40d69205b7d496d88faa5a8c4db76/pex-2.2.1-py2.py3-none-any.whl/pex/resolve/configured_resolve.py", line 42, in resolve
    resolve_from_lock(
  File "/Users/krishnanchandra/.cache/pants/named_caches/pex_root/installed_wheels/7309da7faa3a99f960f878ee917815b3ceb40d69205b7d496d88faa5a8c4db76/pex-2.2.1-py2.py3-none-any.whl/pex/resolve/lock_resolver.py", line 279, in resolve_from_lock
    file_download_managers_by_target = {
  File "/Users/krishnanchandra/.cache/pants/named_caches/pex_root/installed_wheels/7309da7faa3a99f960f878ee917815b3ceb40d69205b7d496d88faa5a8c4db76/pex-2.2.1-py2.py3-none-any.whl/pex/resolve/lock_resolver.py", line 282, in <dictcomp>
    downloader=ArtifactDownloader(
  File "<attrs generated init pex.resolve.downloads.ArtifactDownloader>", line 13, in __init__
    _inst_dict['pip'] = __attr_factory_pip(self)
  File "/Users/krishnanchandra/.cache/pants/named_caches/pex_root/installed_wheels/7309da7faa3a99f960f878ee917815b3ceb40d69205b7d496d88faa5a8c4db76/pex-2.2.1-py2.py3-none-any.whl/pex/resolve/downloads.py", line 62, in _pip
    return get_pip(
  File "/Users/krishnanchandra/.cache/pants/named_caches/pex_root/installed_wheels/7309da7faa3a99f960f878ee917815b3ceb40d69205b7d496d88faa5a8c4db76/pex-2.2.1-py2.py3-none-any.whl/pex/pip/installation.py", line 287, in get_pip
    pip = _resolved_installation(
  File "/Users/krishnanchandra/.cache/pants/named_caches/pex_root/installed_wheels/7309da7faa3a99f960f878ee917815b3ceb40d69205b7d496d88faa5a8c4db76/pex-2.2.1-py2.py3-none-any.whl/pex/pip/installation.py", line 131, in _resolved_installation
    return _pip_installation(
  File "/Users/krishnanchandra/.cache/pants/named_caches/pex_root/installed_wheels/7309da7faa3a99f960f878ee917815b3ceb40d69205b7d496d88faa5a8c4db76/pex-2.2.1-py2.py3-none-any.whl/pex/pip/installation.py", line 52, in _pip_installation
    for dist_location in iter_distribution_locations():
  File "/Users/krishnanchandra/.cache/pants/named_caches/pex_root/installed_wheels/7309da7faa3a99f960f878ee917815b3ceb40d69205b7d496d88faa5a8c4db76/pex-2.2.1-py2.py3-none-any.whl/pex/pip/installation.py", line 108, in bootstrap_pip
    venv.interpreter.execute(["-m", "pip", "install", "--target", target_dir, req])
  File "/Users/krishnanchandra/.cache/pants/named_caches/pex_root/installed_wheels/7309da7faa3a99f960f878ee917815b3ceb40d69205b7d496d88faa5a8c4db76/pex-2.2.1-py2.py3-none-any.whl/pex/interpreter.py", line 1459, in execute
    stdout, stderr = Executor.execute(cmd, stdin_payload=stdin_payload, env=env, **kwargs)
  File "/Users/krishnanchandra/.cache/pants/named_caches/pex_root/installed_wheels/7309da7faa3a99f960f878ee917815b3ceb40d69205b7d496d88faa5a8c4db76/pex-2.2.1-py2.py3-none-any.whl/pex/executor.py", line 99, in execute
    raise cls.NonZeroExit(cmd, process.returncode, stdout, stderr)
c
Is that the same error that lead you to debug it in the first place, or is 
__run.sh
not matching what Pants originally did?
b
It’s the same error, just not a particularly helpful one because I’m trying to look 1 level deeper and figure out what actually caused the non-zero exit code
It’s also hard to run the source command because PEX appears to clear out the 
.tmp
directory in the sandbox between runs
For completeness: Platform - Mac OS 14.3.1 Architecture - ARM (M1 Max) Pex version - 2.2.1 Pants version - 2.21.0.dev3 (latest dev release) Notably, I don’t get this issue on Linux so might be something Mac or ARM-specific.
c
The general answer to how to debug the 
__run.sh
scripts is "hack them up and add debugging", but you already found 
PEX_VERBOSE=9
or whatever` Can you execute 
/private/var/folders/0w/zhpm9fs12zzcmt3m5799xb9c0000gn/T/pants-sandbox-42OgcN/.tmp/tmp0i26txax/pip/bin/python3.12 -s -E -m pip install --target /private/var/folders/0w/zhpm9fs12zzcmt3m5799xb9c0000gn/T/pants-sandbox-42OgcN/.tmp/tmp0i26txax/reqs/pip pip==24.0
, or is that what you meant by the 
.tmp
dir not being preserved?
Thank you for trying the latest dev release!
b
Yes, that’s the problem - I can’t execute that command because the 
.tmp
directory appears to get wiped out between runs
What’s kinda weird here is that there are 2 other commands that Pex executes before the one that fails, and those actually work:
Copy code
These work!

---
pex: Executing: /private/var/folders/0w/zhpm9fs12zzcmt3m5799xb9c0000gn/T/pants-sandbox-Ik8tnJ/.tmp/tmp_l1f_p5j/pip/bin/python3.12 -s -E -m pip install -U pip
pex: Correcting mis-configured MACOSX_DEPLOYMENT_TARGET of 13 to 13.0 corresponding to a valid PEP-425 platform of macosx-13.0-arm64 for PythonInterpreter('/private/var/folders/0w/zhpm9fs12zzcmt3m5799xb9c0000gn/T/pants-sandbox-Ik8tnJ/.tmp/tmp_l1f_p5j/pip/bin/python3.12', PythonIdentity('/private/var/folders/0w/zhpm9fs12zzcmt3m5799xb9c0000gn/T/pants-sandbox-Ik8tnJ/.tmp/tmp_l1f_p5j/pip/bin/python3.12', 'cp312', 'cp312', 'macosx_14_0_arm64', (3, 12, 1))).
Copy code
This doesn't

---
pex: Executing: /private/var/folders/0w/zhpm9fs12zzcmt3m5799xb9c0000gn/T/pants-sandbox-Ik8tnJ/.tmp/tmp_l1f_p5j/pip/bin/python3.12 -s -E -m pip install --target /private/var/folders/0w/zhpm9fs12zzcmt3m5799xb9c0000gn/T/pants-sandbox-Ik8tnJ/.tmp/tmp_l1f_p5j/reqs/pip pip==24.0
c
hmm, is 
pip pip==24.0
even a valid pip command?
b
That’s a path suffix, the command ends with:
Copy code
pip install --target /private/var/folders/0w/zhpm9fs12zzcmt3m5799xb9c0000gn/T/pants-sandbox-Ik8tnJ/.tmp/tmp_l1f_p5j/reqs/pip pip==24.0
which seems okay to me?
c
(never mind misread the target)
👍 1
b
I tried upgrading PEX to the latest release as well but that didn’t help 😞 I’m pretty sure at this point that it’s a Mac-specific issue because it works fine on Linux and I’ve had no issues on our CI machines (Linux) or other Linux machines that I tried the same commands on
Figured this out, it actually had to do with a broken Pip config pointing at a private repo with expired credentials; would have never figured that out if it wasn’t for a completely unrelated failure of Pip on my machine
c
Glad you figured it out! We have this ever growing "is your workstation setup correctly?" verification script and it feels like half the checks are around pip & netrc
h
We have this ever growing “is your workstation setup correctly?” verification script and it feels like half the checks are around pip & netrc
We have a very similar situation going on also, which is also compounded by private indexes being behind a VPN. I’d be interested in seeing said script if possible to compare with our own. 😄
c
This probably gives the flavor:
Copy code
✔ pass PantsLauncher found
✔ pass PantsLauncherVersion (0, 10, 4) ok
✔ pass PythonVersion ('3', '10', '13') ok
✔ pass PythonImplimentation CPython ok
✔ pass AvoidCondaPythonVersion ok
✔ pass AvoidCondaPythonPath ok
✔ pass PipConf ok
✔ pass Netrc ok
✔ pass AwsCli found
✔ pass DockerCli found
✔ pass EcrCredentialsHelperBinary found
✔ pass EcrCredentialsHelperConfig ok
✔ pass EcrCredentialsMacOs n/a on Linux
✔ pass AwsProfile ok
✔ pass PipDownload ok
h
Awesome, thanks. Looks very similar to mine, except we also do a check for something equivalent to what Pants finds on 
[python-bootstrap].search_path
. It would be nice if that could be output from Pants itself.
3 Views
Open in Slack
PreviousNext
Powered by