https://pantsbuild.org/ logo
Join Slack
Powered by
Running `pants --keep-sandboxes=always --no-pants...
# general
h
Running 
pants --keep-sandboxes=always  --no-pantsd --no-local-cache generate-lockfiles
does not persist the pip download log in the sandbox as I expected based on github.com/pantsbuild/pants/pull/21980. Do I need to enable this somehow? From the docs it looks to be enabled by default. I’m on the latest version of Pants.
c
It's not persisted at all? Or you don't see it while Pex is running to 
tail
? (Based on various details, Pex may use .tmp files for the logs and then move to the given path, instead of writing directly there?)
b
It's there. This is main of example-python which runs Pants 2.31.0 and includes the change you mention @hundreds-carpet-28072:
Copy code
:; pants --keep-sandboxes=always  --no-pantsd --no-local-cache generate-lockfiles
14:00:20.12 [INFO] Preserving local process execution dir /tmp/pants-sandbox-j4pa1I for Find interpreter for constraints: CPython==3.12.*
14:00:26.58 [INFO] Preserving local process execution dir /tmp/pants-sandbox-qGvs9H for Generate lockfile for python-default
14:00:38.24 [INFO] Completed: Generate lockfile for python-default
14:00:38.25 [INFO] Wrote lockfile for the resolve `python-default` to python-default.lock

Lockfile diff: python-default.lock [python-default]

==                    Upgraded dependencies                     ==

  attrs                          25.3.0       -->   26.1.0
  iniconfig                      2.1.0        -->   2.3.0
  packaging                      25.0         -->   26.2
  tomli                          2.2.1        -->   2.4.1



:; ls -l /tmp/pants-sandbox-qGvs9H
total 5616
-rwxr-xr-x 1 jsirois jsirois    2437 Jul 10 14:00 __run.sh
-rwxr-xr-x 1 jsirois jsirois 4945588 Jul 10 14:00 pex
-rw-rw-r-- 1 jsirois jsirois  781122 Jul 10 14:00 pex-pip-download.log
-rw-rw-r-- 1 jsirois jsirois   13552 Jul 10 14:00 python-default.lock

:; head /tmp/pants-sandbox-qGvs9H/pex-pip-download.log
2026-07-10T14:00:27,848 Using pip 24.2 from /home/jsirois/.cache/pants/named_caches/pex_root/venvs/3/cf7cc78916945888f370c79db361bca292d78c00/81ddb10d39e90ce5032d62bbf2fb38485c236718/lib/python3.12/site-packages/pip (python 3.12)
2026-07-10T14:00:27,848 Non-user install because user site-packages disabled
2026-07-10T14:00:27,879 Created temporary directory: /home/jsirois/.cache/pants/named_caches/pex_root/pip/3/24.2/pip_cache/.tmp.5cvfd1fg/pip-build-tracker-k8c2qxk2
2026-07-10T14:00:27,879 Initialized build tracking at /home/jsirois/.cache/pants/named_caches/pex_root/pip/3/24.2/pip_cache/.tmp.5cvfd1fg/pip-build-tracker-k8c2qxk2
2026-07-10T14:00:27,879 Created build tracker: /home/jsirois/.cache/pants/named_caches/pex_root/pip/3/24.2/pip_cache/.tmp.5cvfd1fg/pip-build-tracker-k8c2qxk2
2026-07-10T14:00:27,879 Entered build tracker: /home/jsirois/.cache/pants/named_caches/pex_root/pip/3/24.2/pip_cache/.tmp.5cvfd1fg/pip-build-tracker-k8c2qxk2
2026-07-10T14:00:27,879 Created temporary directory: /home/jsirois/.cache/pants/named_caches/pex_root/pip/3/24.2/pip_cache/.tmp.5cvfd1fg/pip-install-1fj94iy7
2026-07-10T14:00:27,880 Created temporary directory: /home/jsirois/.cache/pants/named_caches/pex_root/pip/3/24.2/pip_cache/.tmp.5cvfd1fg/pip-ephem-wheel-cache-3j85_odt
2026-07-10T14:00:27,892 Looking in indexes: <https://pypi.org/simple/>
2026-07-10T14:00:27,893 1 location(s) to search for versions of ansicolors:

:; tail /tmp/pants-sandbox-qGvs9H/pex-pip-download.log
2026-07-10T14:00:36,305 No cache entry available
2026-07-10T14:00:36,399 <https://files.pythonhosted.org:443> "GET /packages/df/b2/87e62e8c3e2f4b32e5fe99e0b86d576da1312593b39f47d8ceef365e95ed/packaging-26.2-py3-none-any.whl HTTP/1.1" 200 100195
2026-07-10T14:00:36,399 Downloading packaging-26.2-py3-none-any.whl (100 kB)
2026-07-10T14:00:36,422 Ignoring unknown cache-control directive: immutable
2026-07-10T14:00:36,422 Updating cache with response from "<https://files.pythonhosted.org/packages/df/b2/87e62e8c3e2f4b32e5fe99e0b86d576da1312593b39f47d8ceef365e95ed/packaging-26.2-py3-none-any.whl>"
2026-07-10T14:00:36,422 etag object cached for 1209600 seconds
2026-07-10T14:00:36,422 Caching due to etag
2026-07-10T14:00:36,430 Downloading link <https://files.pythonhosted.org/packages/df/b2/87e62e8c3e2f4b32e5fe99e0b86d576da1312593b39f47d8ceef365e95ed/packaging-26.2-py3-none-any.whl> (from <https://pypi.org/simple/packaging/>) (requires-python:>=3.8) to /home/jsirois/.cache/pants/named_caches/pex_root/pip/3/24.2/pip_cache/.tmp.5cvfd1fg/pip-unpack-5ku2qexz/packaging-26.2-py3-none-any.whl
2026-07-10T14:00:36,452 Would install ansicolors-1.1.8 attrs-26.1.0 iniconfig-2.3.0 packaging-26.2 pluggy-1.6.0 py-1.11.0 pytest-7.1.3 setuptools-56.2.0 tomli-2.4.1 types-setuptools-57.4.18
2026-07-10T14:00:36,452 Removed build tracker: '/home/jsirois/.cache/pants/named_caches/pex_root/pip/3/24.2/pip_cache/.tmp.5cvfd1fg/pip-build-tracker-k8c2qxk2'
h
Simply not persisted in the sandbox dir it references related to “Generate lockfile for <resolve>”. I see 
__run.sh, pex
, 
python-default.lock
but no 
pex-pip-download.log
. Maybe another config option is affecting this?
b
I used example-python for full transparency. You can read all the setup in that repo. Perhaps you could do that and / or be as forthcoming with your setup info.
h
I’ll attach my equivalent pants.toml. Currently our locking process is: 
pyproject.toml -> uv export > constraints.txt -> pants generate-lockfiles
. This issue persists on FWIU is a clean setup as I have a cache/daemon wiping make function that I’ve ran:
Copy code
> make wipe-cache
+ rm -rf .pids && rm -rf .pants.d && rm -rf ~/.cache/pants/* && rm -rf dist/*
+ rm -rf ~/.cache/nce && rm -rf ~/Library/Caches/nce
pants.toml
b
So.. did you try changing relevant settings in example-python to match yours? You're 1 Pants version ahead of example-python. Reminder: I am not a Pants maintainer, I am unpaid, your repo is closed source.
h
Testing that now, I missed that the example repo was on the version behind. Will see the result in ~ 8 minutes 😅.
Reminder: I am not a Pants maintainer, I am unpaid, your repo is closed source. (edited)
I’m aware of who you are - every thread I’ve posted about slow locking in the past you’ve been there. I’m not unappreciative of the help - I just thought it worth asking whether there was a config option I was missing (the docs can be difficult to search for some things), before sharing my entire config.
b
It should be much less than 8 minutes if you're working on the shared reference point of example-python. That's the point! Use a shared public reference or else debugging is hard. You need to meet in the middle.
So, modify example-python to be like your repo, test there 1st. Does the issue repro there in the shared reference point?
That's just the price you should be expecting to pay for a closed source bug.
Basically, questions aren't free. They cause fielders to either ignore (mostly free for them) or engage and try to help (costs mount). In my eyes this makes extra legwork on the askers part be the right thing.
@hundreds-carpet-28072 sidestepping all this maybe, why the heck are you doing a uv export dance on that Pants version which has support for 
[python] resolver = "uv"
?
h
@hundreds-carpet-28072 sidestepping all this maybe, why the heck are you doing a uv export dance on that Pants version which has support for 
[python] resolver = "uv"
?
Because the uv resolver has a bug that prevents me using it. I’ve posted previously about it, and submitted an issue. I would like to use it ASAP.
b
Aha, ok.
Which issue @hundreds-carpet-28072, I'm not finding it.
h
b
Excellent, thanks.
Ok, so I did the exercise of trying to make example-python look like your setup: + used same pants version as your setup + used all the same python plugins in case those interact wierldy + added a constraints file + tried a split universal lock in case your requirements are split
Copy code
:; git diff --cached origin/main
diff --git a/constraints.txt b/constraints.txt
new file mode 100644
index 0000000..dfe9541
--- /dev/null
+++ b/constraints.txt
@@ -0,0 +1,3 @@
+# Just trying to simulate Danny's setup, so pin one requirement.
+setuptools==56.2.0
+
diff --git a/pants.toml b/pants.toml
index 5c04bfd..c06dbc6 100644
--- a/pants.toml
+++ b/pants.toml
@@ -2,15 +2,17 @@
 # Licensed under the Apache License, Version 2.0 (see LICENSE).

 [GLOBAL]
-pants_version = "2.31.0"
+pants_version = "2.32.0"
 backend_packages.add = [
-  "pants.backend.build_files.fmt.black",
+  "pants.backend.build_files.fmt.black",
+  "pants.backend.codegen.protobuf.python",
   "pants.backend.python",
   "pants.backend.python.lint.docformatter",
   "pants.backend.python.lint.black",
   "pants.backend.python.lint.flake8",
   "pants.backend.python.lint.isort",
   "pants.backend.python.typecheck.mypy",
+  "pants.backend.python.mixed_interpreter_constraints",
 ]

 [anonymous-telemetry]
@@ -38,6 +40,9 @@ enable_resolves = true

 resolves = { python-default = "python-default.lock"}

+[python.resolves_to_constraints_file]
+python-default = "constraints.txt"
+
 [python-bootstrap]
 # We search for interpreters both on the $PATH and in the `$(pyenv root)/versions` folder.
 #  If you're using macOS, you may want to leave off the <PATH> entry to avoid using the
diff --git a/requirements.txt b/requirements.txt
index 979167d..c78ea76 100644
--- a/requirements.txt
+++ b/requirements.txt
@@ -1,7 +1,8 @@
 # Copyright 2020 Pants project contributors.
 # Licensed under the Apache License, Version 2.0 (see LICENSE).

-ansicolors==1.1.8
+ansicolors==1.1.8; platform_system == 'Linux'
+ansicolors; platform_system != 'Linux'
 setuptools>=56.2.0,<57
 types-setuptools>=56.2.0,<58
-pytest==7.1.3
\ No newline at end of file
+pytest==7.1.3
That still produces a pip log file though:
Copy code
:; pants --keep-sandboxes=always  --no-pantsd --no-local-cache generate-lockfiles
07:32:49.69 [INFO] Preserving local process execution dir /tmp/pants-sandbox-ptkqOi for Find interpreter for constraints: CPython==3.12.*
07:32:51.20 [INFO] Preserving local process execution dir /tmp/pants-sandbox-AzipDW for Generate pex lockfile for python-default
07:32:58.03 [INFO] Completed: Generate pex lockfile for python-default
07:32:58.04 [INFO] Wrote lockfile for the resolve `python-default` to python-default.lock

:; head /tmp/pants-sandbox-AzipDW/pex-pip-download.log
1/2]universal-CPython-linux: 2026-07-12T07:32:52,595 Using pip 24.2 from /home/jsirois/.cache/pants/named_caches/pex_root/venvs/3/389f00aa4a5c8cea31f5852201ca33f6daea75d0/81ddb10d39e90ce5032d62bbf2fb38485c236718/lib/python3.12/site-packages/pip (python 3.12)
1/2]universal-CPython-linux: 2026-07-12T07:32:52,595 Non-user install because user site-packages disabled
1/2]universal-CPython-linux: 2026-07-12T07:32:52,625 Created temporary directory: /home/jsirois/.cache/pants/named_caches/pex_root/pip/3/24.2/pip_cache/.tmp.tkbgur0_/pip-build-tracker-8vuqtflb
1/2]universal-CPython-linux: 2026-07-12T07:32:52,625 Initialized build tracking at /home/jsirois/.cache/pants/named_caches/pex_root/pip/3/24.2/pip_cache/.tmp.tkbgur0_/pip-build-tracker-8vuqtflb
1/2]universal-CPython-linux: 2026-07-12T07:32:52,625 Created build tracker: /home/jsirois/.cache/pants/named_caches/pex_root/pip/3/24.2/pip_cache/.tmp.tkbgur0_/pip-build-tracker-8vuqtflb
1/2]universal-CPython-linux: 2026-07-12T07:32:52,625 Entered build tracker: /home/jsirois/.cache/pants/named_caches/pex_root/pip/3/24.2/pip_cache/.tmp.tkbgur0_/pip-build-tracker-8vuqtflb
1/2]universal-CPython-linux: 2026-07-12T07:32:52,625 Created temporary directory: /home/jsirois/.cache/pants/named_caches/pex_root/pip/3/24.2/pip_cache/.tmp.tkbgur0_/pip-install-6bkopzdt
1/2]universal-CPython-linux: 2026-07-12T07:32:52,627 Created temporary directory: /home/jsirois/.cache/pants/named_caches/pex_root/pip/3/24.2/pip_cache/.tmp.tkbgur0_/pip-ephem-wheel-cache-6kx21rsd
1/2]universal-CPython-linux: 2026-07-12T07:32:52,636 Looking in indexes: <https://pypi.org/simple/>
1/2]universal-CPython-linux: 2026-07-12T07:32:52,637 Ignoring ansicolors: markers 'platform_system != "Linux"' don't match your environment

:; tail /tmp/pants-sandbox-AzipDW/pex-pip-download.log
2/2]universal-CPython-mac: 2026-07-12T07:32:57,682 Looking up "<https://files.pythonhosted.org/packages/df/b2/87e62e8c3e2f4b32e5fe99e0b86d576da1312593b39f47d8ceef365e95ed/packaging-26.2-py3-none-any.whl>" in the cache
2/2]universal-CPython-mac: 2026-07-12T07:32:57,683 Current age based on date: 149541
2/2]universal-CPython-mac: 2026-07-12T07:32:57,683 Ignoring unknown cache-control directive: immutable
2/2]universal-CPython-mac: 2026-07-12T07:32:57,683 Freshness lifetime from max-age: 365000000
2/2]universal-CPython-mac: 2026-07-12T07:32:57,683 The response is "fresh", returning cached response
2/2]universal-CPython-mac: 2026-07-12T07:32:57,683 365000000 > 149541
2/2]universal-CPython-mac: 2026-07-12T07:32:57,683 Using cached packaging-26.2-py3-none-any.whl (100 kB)
2/2]universal-CPython-mac: 2026-07-12T07:32:57,683 Downloading link <https://files.pythonhosted.org/packages/df/b2/87e62e8c3e2f4b32e5fe99e0b86d576da1312593b39f47d8ceef365e95ed/packaging-26.2-py3-none-any.whl> (from <https://pypi.org/simple/packaging/>) (requires-python:>=3.8) to /home/jsirois/.cache/pants/named_caches/pex_root/pip/3/24.2/pip_cache/.tmp.s0rz119z/pip-unpack-1fgmrkdk/packaging-26.2-py3-none-any.whl
2/2]universal-CPython-mac: 2026-07-12T07:32:57,709 Would install ansicolors-1.1.8 attrs-26.1.0 iniconfig-2.3.0 packaging-26.2 pluggy-1.6.0 py-1.11.0 pytest-7.1.3 setuptools-56.2.0 tomli-2.4.1 types-setuptools-57.4.18
2/2]universal-CPython-mac: 2026-07-12T07:32:57,709 Removed build tracker: '/home/jsirois/.cache/pants/named_caches/pex_root/pip/3/24.2/pip_cache/.tmp.s0rz119z/pip-build-tracker-7qyrumjx'
I will note your setup has:
Copy code
pantsrc_files = [
  ".pants.rc",
]
That's an easy way to inject chaos locally in a 
.pants.rc
you forgot about when debugging an issue. May not be in play here - but definitely a debugging foot gun.
h
I've been doing the same, implementing one by one from my setup into example-python and have yet to find what is causing that log file to not be persisted. Will update when I do. I made sure to not have a 
.pants.rc
active, we use that for switchable user modes but it’s definitely not in play here. I'm trying to find a grep-able log from either Pip or Pex from when that log file is created. Any clue on that one would be useful but happy to find out for myself.
b
Well, forget Python, Pex, Pip: for any Pants action, use 
-ldebug
. That prints out command lines IIRC. Look for this argument or lack thereof:
Copy code
:; pex -h | grep pip | grep log
  --pip-log, --preserve-pip-download-log, --no-preserve-pip-download-log [PIP_LOG]
                        With no argument, preserve the `pip download` log and
Yeah:
Copy code
:; pants --keep-sandboxes=always  --no-pantsd --no-local-cache generate-lockfiles -ldebug
...
08:28:04.30 [DEBUG] spawned local process as Some(75852) for Process { argv: ["/home/jsirois/.cache/nce/4f07959fafe3296d25d3e3d2c92165cc6933455f0caa80a335a8aaa14c295fb3/bindings/venvs/2.32.0/bin/python3.14", "./pex", "lock", "create", "--tmpdir", ".tmp", "--no-emit-warnings", "--python-path", "/home/jsirois/.pyenv/versions/2.7.18/bin:/home/jsirois/.pyenv/versions/3.10.20/bin:/home/jsirois/.pyenv/versions/3.11.15/bin:/home/jsirois/.pyenv/versions/3.12.13/bin:/home/jsirois/.pyenv/versions/3.13.14/bin:/home/jsirois/.pyenv/versions/3.14.6/bin:/home/jsirois/.pyenv/versions/3.14.6t/bin:/home/jsirois/.pyenv/versions/3.15.0b3/bin:/home/jsirois/.pyenv/versions/3.5.9/bin:/home/jsirois/.pyenv/versions/3.6.15/bin:/home/jsirois/.pyenv/versions/3.7.17/bin:/home/jsirois/.pyenv/versions/3.8.20/bin:/home/jsirois/.pyenv/versions/3.9.25/bin:/home/jsirois/.pyenv/versions/pypy2.7-7.3.22/bin:/home/jsirois/.pyenv/versions/pypy3.10-7.3.19/bin:/home/jsirois/.pyenv/versions/pypy3.11-7.3.22/bin:/home/jsirois/.pyenv/versions/pypy3.5-7.0.0/bin:/home/jsirois/.pyenv/versions/pypy3.6-7.3.3/bin:/home/jsirois/.pyenv/versions/pypy3.7-7.3.9/bin:/home/jsirois/.pyenv/versions/pypy3.8-7.3.11/bin:/home/jsirois/.pyenv/versions/pypy3.9-7.3.16/bin:/home/jsirois/.pyenv/shims:/home/jsirois/.pyenv/bin:/home/jsirois/.nvm/versions/node/v24.2.0/bin:/home/jsirois/.cargo/bin:/home/jsirois/.local/bin:/home/jsirois/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/home/jsirois/go/bin", "--output=python-default.lock", "--style=universal", "--pip-version", "24.2", "--resolver-version", "pip-2020-resolver", "--preserve-pip-download-log", "pex-pip-download.log", "--target-system", "linux", "--target-system", "mac", "--indent=2", "--python-path=/home/jsirois/.pyenv/versions/3.12.13/bin/python3.12", "--no-pypi", "--index=<https://pypi.org/simple/>", "--manylinux", "manylinux2014", "--constraints=constraints.txt", "--interpreter-constraint", "CPython==3.12.*", "ansicolors; platform_system != \"Linux\"", "ansicolors==1.1.8; platform_system == \"Linux\"", "pytest==7.1.3", "setuptools<57,>=56.2.0", "types-setuptools<58,>=56.2.0"]
...
Notice: 
"--preserve-pip-download-log", "pex-pip-download.log"
in there.
So, that's not from Pip or Pex, its from Pants, but it will confirm Pants is - or is not - asking Pex to retain the Pip log. If Pants isn't passing the flag - Pants bug. If it is - Pex bug.
We can narrow from there.
Or ... just look at the run file in the sandbox:
Copy code
:; cat /tmp/pants-sandbox-xNG3Rh/__run.sh
#!/usr/bin/env bash
# This command line should execute the same process as pants did internally.
cd /tmp/pants-sandbox-xNG3Rh
env -i CPPFLAGS='' LANG=en_US.UTF-8 LDFLAGS='' PATH=$'/home/jsirois/.pyenv/shims:/home/jsirois/.pyenv/bin:/home/jsirois/.nvm/versions/node/v24.2.0/bin:/home/jsirois/.cargo/bin:/home/jsirois/.local/bin:/home/jsirois/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/home/jsirois/go/bin' PEX_IGNORE_RCFILES=true PEX_ROOT=.cache/pex_root PEX_SCRIPT=pex3 /home/jsirois/.cache/nce/4f07959fafe3296d25d3e3d2c92165cc6933455f0caa80a335a8aaa14c295fb3/bindings/venvs/2.32.0/bin/python3.14 ./pex lock create --tmpdir .tmp --no-emit-warnings --python-path $'/home/jsirois/.pyenv/versions/2.7.18/bin:/home/jsirois/.pyenv/versions/3.10.20/bin:/home/jsirois/.pyenv/versions/3.11.15/bin:/home/jsirois/.pyenv/versions/3.12.13/bin:/home/jsirois/.pyenv/versions/3.13.14/bin:/home/jsirois/.pyenv/versions/3.14.6/bin:/home/jsirois/.pyenv/versions/3.14.6t/bin:/home/jsirois/.pyenv/versions/3.15.0b3/bin:/home/jsirois/.pyenv/versions/3.5.9/bin:/home/jsirois/.pyenv/versions/3.6.15/bin:/home/jsirois/.pyenv/versions/3.7.17/bin:/home/jsirois/.pyenv/versions/3.8.20/bin:/home/jsirois/.pyenv/versions/3.9.25/bin:/home/jsirois/.pyenv/versions/pypy2.7-7.3.22/bin:/home/jsirois/.pyenv/versions/pypy3.10-7.3.19/bin:/home/jsirois/.pyenv/versions/pypy3.11-7.3.22/bin:/home/jsirois/.pyenv/versions/pypy3.5-7.0.0/bin:/home/jsirois/.pyenv/versions/pypy3.6-7.3.3/bin:/home/jsirois/.pyenv/versions/pypy3.7-7.3.9/bin:/home/jsirois/.pyenv/versions/pypy3.8-7.3.11/bin:/home/jsirois/.pyenv/versions/pypy3.9-7.3.16/bin:/home/jsirois/.pyenv/shims:/home/jsirois/.pyenv/bin:/home/jsirois/.nvm/versions/node/v24.2.0/bin:/home/jsirois/.cargo/bin:/home/jsirois/.local/bin:/home/jsirois/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/home/jsirois/go/bin' $'--output=python-default.lock' $'--style=universal' --pip-version 24.2 --resolver-version pip-2020-resolver --preserve-pip-download-log pex-pip-download.log --target-system linux --target-system mac $'--indent=2' $'--python-path=/home/jsirois/.pyenv/versions/3.12.13/bin/python3.12' --no-pypi $'--index=<https://pypi.org/simple/>' --manylinux manylinux2014 $'--constraints=constraints.txt' --interpreter-constraint $'CPython==3.12.*' $'ansicolors; platform_system != "Linux"' $'ansicolors==1.1.8; platform_system == "Linux"' $'pytest==7.1.3' $'setuptools<57,>=56.2.0' $'types-setuptools<58,>=56.2.0'
Also has 
--preserve-pip-download-log pex-pip-download.log
.
So @hundreds-carpet-28072 - that last thing is probably easiest: what does your 
__run.sh
file say its doing?
Then maybe paired with, in my example sandbox for example:
Copy code
:; /tmp/pants-sandbox-xNG3Rh/pex -V
2.95.1
h
So both setups contain the relevant flag in their 
__run.sh
scripts:
Copy code
--pip-version 24.2 --resolver-version pip-2020-resolver --preserve-pip-download-log pex-pip-download.log --target-system linux --target-system mac
My private setup’s Pex version matches that of example-python:
Copy code
❯ /private/var/folders/4w/8m7ytrp52vn4zxkrt6p86d780000gp/T/pants-sandbox-example-python/pex -V
2.95.1
❯ /private/var/folders/4w/8m7ytrp52vn4zxkrt6p86d780000gp/T/pants-sandbox-private-python/pex -V
2.95.1
Interestingly the shebang failed as my 
python
command is not symlinked or alias’d, but don’t think this is relevant since the 
generate-lockfiles
step does function just fine:
Copy code
❯ /private/var/folders/4w/8m7ytrp52vn4zxkrt6p86d780000gp/T/pants-sandbox-private-python/pex -V
env: python: No such file or directory
I think I’ve found the culprit for my slow locking, and potentially the culprit for the 
pex-pip-download.log
file not being created. I have a specific dependency that seems to be using up an extreme amount of time when locking: 
pyopengl
. I’ve mirrored everything from my private setup into example-python to the point where the 
__run.sh
generated is identical other than one thing, and once I include 
pyopengl
in the dependencies list (either via an actual file import dependency or by manually adding it to the 
__run.sh
the locking time shoots up and the 
pex-pip-download.log
file is simply not created in the sandbox throughout.
Copy code
pex: Resolving for:
  universal resolve targeting linux and mac for CPython using cp312-cp312-macosx_15_0_arm64 interpreter at /Users/danny-todd/.local/share/uv/python/cpython-3.12.13-macos-aarch64-none/bin/python3.12 from split by requirement 'PyOpenGL==3.1.7'
  universal resolve targeting linux and mac for CPython using cp312-cp312-macosx_15_0_arm64 interpreter at /Users/danny-todd/.local/share/uv/python/cpython-3.12.13-macos-aarch64-none/bin/python3.12 from split by requirement 'pyopengl': 858624.4ms
pex:   Searching for 1856 fingerprints in database: 8.4ms
pex:   Making 350 PEP-691 JSON API requests across 10 threads to fingerprint 1813 artifacts: 12181.2ms
pex:   Caching 1813 fingerprints in database: 7.5ms
pex:   Searching for 1856 fingerprints in database: 8.6ms
pex: Creating lock from resolve: 25.2ms
pex: Indexing downloads: 3.0ms
I’ll attach the full log also out of interest ^. My problem now is that 
export PEX_VERBOSE=3
doesn’t seem to affect the nested pex calls that this process is making for these “split” resolves, so I still can’t investigate why this issue is occurring around 
pyopengl
specifically. In a resolve with just 
pyopengl
, the issue doesn’t occur and it logs its process of pulling the single sdist and the single wheel and locking to both in the same manner. Setting 
PIP_LOG
to another file also doesn’t help.
slow-generate-lockfile-pyopengl.log
b
@hundreds-carpet-28072 on the slow front, you should be using 
--pip-version latest-compatible
. Partially defeated by custom indexes without .metadata support, but not fully defeated. Can you share your requirement list? Are you sure it's split?
Aha, reqs are in log. Looking...
@hundreds-carpet-28072 yeah, so for a split resolve, the final 
pex-pip-download.log
won't be created until all resolves end and they can be merged. You would need to grep for the Pex-calling-Pip args like 
--log /private/var/folders/4w/8m7ytrp52vn4zxkrt6p86d780000gp/T/pants-sandbox-kBqjLQ/.tmp/pex-pip-log.3psom2gh/pip.universal-CPython-linux-and-mac-58d82f7bb2a568a3ff146e224ca1d1423143de43.log
in your output and tail those. Towards that end, your attached log does not include the full original requirements list. That would be useful to confirm Pex is doing the right thing to split on PyOpenGL.
So you'd need to do something like 
tail -f <sandbox dir>/.tmp/pex-pip-log*/*.log
if you want to read the on-going logging before resolve completion, when the merged 
pex-pip-download.log
should be presented in the sandbox unless one of the split resolves fails.
h
I’ve found the reason for the split resolves and it’s obvious now looking at it, my eyes glossed over it previously but we have a project with a BUILD configured with a 
python_requirement
like:
Copy code
python_sources(
    sources=["src/**/*.py"],
    dependencies=[":opengl_req"],
)

python_requirement(
    name="opengl_req",
    requirements=["pyopengl"],
)
We have a few cases of explicit dependencies pointing at requirements in 
python_requirements
targets which I mistook the above as:
Copy code
dependencies=[
        "//:reqs#PyOpenGL",
    ],
Ergh. So I don’t think pyopengl is the issue with my current slow locking, which I’m taking a further look into now - it was just a dodgy configuration that meant I confused the info in the logs. Weirdly enough, on fixing this configuration like above my locking has jumped from ~ 6 mins to 15 mins - I’m assuming it wasn’t doing a lot of work it should have been, prior. * Although that may have been 6 mins with the pex cache intact, so maybe not weird.
I appreciate your help here @brief-scientist-13682, apologies for the somewhat stupid cause. After fixing the above, I did experience another problem. There was an incorrect line in the RECORD manifest of my private package that was causing the wheel to be installed (i.e. .dist-info and .pex-info directories extracted into the running environment) but no source files were being included - this was then throwing ModuleNotFoundError on invoking 
pants test
on files dependent on this private package. Module mappings and 
pants peek
-ing into the dependency graph all looked correct, but the actual source files and namespace were not installed correctly. I couldn’t find anything relevant in the logs up until I jumped into the sandbox and directly ran the 
requirements.pex
executable which should’ve contained said package. I then could see the relevant warning which was breaking my setup:
Copy code
❯ python3 ./requirements.pex
/private/var/folders/4w/8m7ytrp52vn4zxkrt6p86d780000gp/T/pants-sandbox-fm4VJs/requirements.pex/.bootstrap/pex/pep_427.py:1003: PEXWarning: The wheel private-package.whl has a bad RECORD. Skipping install of non-existent file badpath/file.py and possibly others.
I went back to check the logs (with all possible flags mind you, 
PEX_VERBOSE=3 -ldebug --print-stacktrace
) and this warning was not spat out anywhere other that direct run on 
requirements.pex
. So again, another problem caused from our side entirely but the impact being quite breaking and the warning not being surfaced may be a problem worth investigating at some point. Although, I’d imagine 99% of people aren’t rolling their own wheel builds. 🤷
b
Rolling your own is fine - but you take on the standards burden at that point. It sounds like whoever rolled these did not do that thoroughly.
A few things though to wrap up - all unrelated: + the sandbox pip log will only appear at the end of a completed resolve when the resolve is split. For those, to real-time tail, you need to find the component logs in the sandbox .tmp/ dir and tail those + the speed thing: I refuse to go into details yet again - uv is the answer, but I believe you are still not using 
pip_version
to tell Pex 
--pip-version latest-compatible
- you absolutely want that to get modern metadata support (instead of downloading wheels) and modern pip report support - both make things as fast as they can be using Pip. + The bad wheels thing: you need to swallow this or else file Pants bugs if you really want warning from Pex auto-surfaced. Talk here will black hole (and it still may in the Pants issue you file, but its your best bet if your serious about wanting this).
h
> + the speed thing: I refuse to go into details yet again - uv is the answer, but I believe you are still not using 
pip_version
to tell Pex 
--pip-version latest-compatible
- you absolutely want that to get modern metadata support (instead of downloading wheels) and modern pip report support - both make things as fast as they can be using Pip. We were using 
pip_version = latest
already but I’d disabled it when encountering my first set of problems. 
latest-compatible
is indeed enabled now but I haven’t AB tested the resolve times, ~ 8 mins currently which I can work with for the time being. > + The bad wheels thing: you need to swallow this or else file Pants bugs if you really want warning from Pex auto-surfaced. Talk here will black hole (and it still may in the Pants issue you file, but its your best bet if your serious about wanting this). Acknowledged. Thanks for your help again, sorry for the parts of troubleshooting that were just a me problem. I hope to see Pants improve in the areas that other tools have excelled soon, as it’s becoming a huge maintenance burden in some cases.
Open in Slack
PreviousNext
Powered by