https://pantsbuild.org/ logo
Join Slack
Powered by
just wanted to see if I could get some help here: ...
# general
c
just wanted to see if I could get some help here: I am trying to install 
detectron2
via 
pants generate-lockfiles
. I've added the following line to my `requirements.txt`:
Copy code
detectron2 @ git+<https://github.com/facebookresearch/detectron2.git@2a420edb307c9bdf640f036d3b196bed474b8593>
The error I get is
Copy code
17:37:06.86 [INFO] Scheduler initialized.
17:37:07.68 [INFO] Completed: Generate lockfile for setuptools
17:37:08.43 [INFO] Completed: Generate lockfile for mypy
17:37:09.45 [INFO] Completed: Generate lockfile for python-default
17:37:09.45 [ERROR] 1 Exception encountered:

Engine traceback:
  in `generate-lockfiles` goal

ProcessExecutionFailure: Process 'Generate lockfile for python-default' failed with exit code 1.
stdout:

stderr:
pid 77806 -> ~/.cache/pants/named_caches/pex_root/venvs/0deb89b23adbbad7a5b9aac06d2362acf7572cfa/d819101e76d1257235093025e8afc37bee57f317/bin/python -sE ~/.cache/pants/named_caches/pex_root/venvs/0deb89b23adbbad7a5b9aac06d2362acf7572cfa/d819101e76d1257235093025e8afc37bee57f317/pex --disable-pip-version-check --no-python-version-warning --exists-action a --no-input --isolated -q --cache-dir ~/.cache/pants/named_caches/pex_root/pip/24.0/pip_cache --log /private/var/folders/wq/xvgpz9v17qsbhb1vwz087v5w0000gn/T/pants-sandbox-y1SsV5/.tmp/pex-pip-log.vqga2q8u/pip.log download --dest /private/var/folders/wq/xvgpz9v17qsbhb1vwz087v5w0000gn/T/pants-sandbox-y1SsV5/.tmp/tmpbwvn219_/opt.homebrew.Cellar.python@3.12.3.12.4.Frameworks.Python.framework.Versions.3.12.bin.python3.12 Pillow==10.3.0 PyMuPDF==1.23.26 PyPDF2==3.0.1 SQLAlchemy==2.0.23 SQLAlchemy_Utils==0.41.1 absl_py==2.1.0 aiofiles==23.2.1 alembic==1.13.1 anyio==4.3.0 beautifulsoup4==4.12.3 boto3-stubs~=1.34.141 boto3~=1.34.141 botocore~=1.34.95 colorama~=0.4.6 datasets==2.14.6 detectron2@ git+<https://github.com/facebookresearch/detectron2.git@2a420edb307c9bdf640f036d3b196bed474b8593> docker==7.0.0 dspy_ai==2.4.12 fastapi==0.110.0 fire==0.5.0 google_api_python_client==2.134.0 google_auth==2.29.0 google_auth_oauthlib==1.0.0 httpx==0.27.0 instructor==1.3.4 llama-index==0.10.40 matplotlib==3.9.1 ml_collections==0.1.1 msal==1.28.0 networkx==3.3 nltk==3.8.1 numpy==1.26.4 omegaconf==2.3.0 openai==1.23.2 opencv-python==4.10.0.82 outlines==0.0.34 pandas-stubs pandas==2.2.2 pdfminer.six==20221105 pdfplumber==0.10.3 pex==2.2.2 playwright==1.41.2 protobuf==4.25.3 psycopg2_binary==2.9.9 pycocoevalcap==1.2 pydantic==2.7.4 pylibmagic==0.5.0 pypdfium2==4.30.0 pytesseract==0.3.10 pytest==8.2.2 pytest_mock==3.14.0 python-magic==0.4.27 python-multipart==0.0.9 pytorch_lightning==2.1.3 ragas==0.1.9 requests==2.31.0 rich==13.7.1 scikit_learn==1.5.0 setuptools==70.0.0 sqlmodel==0.0.18 starlette~=0.36.3 streamlit==1.35.0 structlog~=24.2.0 tenacity==8.2.3 tensorboard==2.16.2 torch==2.3.0 torchvision==0.18.0 tqdm==4.66.4 transformers==4.41.2 trio==0.24.0 types-Pillow==9.5.0 types-beautifulsoup4==4.12.0.20240511 types-colorama~=0.4.6 types-requests==2.31.0 types-tqdm==4.66.0.5 unstructured-client~=0.22.0 unstructured~=0.14.4 uvicorn==0.28.0 wandb==0.17.4 watchdog~=4.0.0 weaviate_client==4.6.5 websockets~=12.0.0 wheel~=0.43.0 --index-url <https://pypi.org/simple/> --retries 5 --timeout 15 exited with 1 and STDERR:
  error: subprocess-exited-with-error
  
  × python setup.py egg_info did not run successfully.
  │ exit code: 1
  ╰─> See above for output.
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.




Use `--keep-sandboxes=on_failure` to preserve the process chroot for inspection.
✅ 1
b
Hard to say what the issue is without seeing what the underlying error is - if you add 
--keep-sandboxes=on_failure
as an option to pants for the first command, you’ll get a sandbox that contains the script used to run this, and you should be able to run the underlying 
pip
command using that to get more detailed output
c
Does this help?
Copy code
#!/usr/bin/env bash
# This command line should execute the same process as pants did internally.
cd /private/var/folders/wq/xvgpz9v17qsbhb1vwz087v5w0000gn/T/pants-sandbox-q1g7OH
env -i CPPFLAGS= LANG=en_US.UTF-8 LDFLAGS= PATH=$'~/.local/bin:~/anaconda3/envs/innovation/bin:~/anaconda3/condabin:/opt/homebrew/bin:/opt/homebrew/sbin:/usr/local/bin:/System/Cryptexes/App/usr/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Applications/quarto/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/local/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/appleinternal/bin' PEX_IGNORE_RCFILES=true PEX_PYTHON=~/Library/Caches/nce/cf6c1988edaa5d40bbf385ae79a75e6d2abba937217d1b672c71eaf0ea495bcd/bindings/venvs/2.21.0/bin/python3.9 PEX_ROOT=.cache/pex_root PEX_SCRIPT=pex3 ~/Library/Caches/nce/cf6c1988edaa5d40bbf385ae79a75e6d2abba937217d1b672c71eaf0ea495bcd/bindings/venvs/2.21.0/bin/python3.9 ./pex lock create --tmpdir .tmp --no-emit-warnings --python-path $'~/.local/bin:~/anaconda3/envs/innovation/bin:~/anaconda3/condabin:/opt/homebrew/bin:/opt/homebrew/sbin:/usr/local/bin:/System/Cryptexes/App/usr/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Applications/quarto/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/local/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/appleinternal/bin' $'--output=lock.json' $'--style=universal' --pip-version 24.0 --resolver-version pip-2020-resolver --target-system linux --target-system mac $'--indent=2' --no-pypi $'--index=<https://pypi.org/simple/>' --manylinux manylinux2014 --interpreter-constraint $'CPython==3.12.*' $'Pillow==10.3.0' $'PyMuPDF==1.23.26' $'PyPDF2==3.0.1' $'SQLAlchemy==2.0.23' $'SQLAlchemy_Utils==0.41.1' $'absl_py==2.1.0' $'aiofiles==23.2.1' $'alembic==1.13.1' $'anyio==4.3.0' $'beautifulsoup4==4.12.3' $'boto3-stubs~=1.34.141' $'boto3~=1.34.141' $'botocore~=1.34.95' $'colorama~=0.4.6' $'datasets==2.14.6' $'detectron2@ git+<https://github.com/facebookresearch/detectron2.git@2a420edb307c9bdf640f036d3b196bed474b8593>' $'docker==7.0.0' $'dspy_ai==2.4.12' $'fastapi==0.110.0' $'fire==0.5.0' $'google_api_python_client==2.134.0' $'google_auth==2.29.0' $'google_auth_oauthlib==1.0.0' $'httpx==0.27.0' $'instructor==1.3.4' $'llama-index==0.10.40' $'matplotlib==3.9.1' $'ml_collections==0.1.1' $'msal==1.28.0' $'networkx==3.3' $'nltk==3.8.1' $'numpy==1.26.4' $'omegaconf==2.3.0' $'openai==1.23.2' $'opencv-python==4.10.0.82' $'outlines==0.0.34' pandas-stubs $'pandas==2.2.2' $'pdfminer.six==20221105' $'pdfplumber==0.10.3' $'pex==2.2.2' $'playwright==1.41.2' $'protobuf==4.25.3' $'psycopg2_binary==2.9.9' $'pycocoevalcap==1.2' $'pydantic==2.7.4' $'pylibmagic==0.5.0' $'pypdfium2==4.30.0' $'pytesseract==0.3.10' $'pytest==8.2.2' $'pytest_mock==3.14.0' $'python-magic==0.4.27' $'python-multipart==0.0.9' $'pytorch_lightning==2.1.3' $'ragas==0.1.9' $'requests==2.31.0' $'rich==13.7.1' $'scikit_learn==1.5.0' $'setuptools==70.0.0' $'sqlmodel==0.0.18' $'starlette~=0.36.3' $'streamlit==1.35.0' $'structlog~=24.2.0' $'tenacity==8.2.3' $'tensorboard==2.16.2' $'torch==2.3.0' $'torchvision==0.18.0' $'tqdm==4.66.4' $'transformers==4.41.2' $'trio==0.24.0' $'types-Pillow==9.5.0' $'types-beautifulsoup4==4.12.0.20240511' $'types-colorama~=0.4.6' $'types-requests==2.31.0' $'types-tqdm==4.66.0.5' $'unstructured-client~=0.22.0' $'unstructured~=0.14.4' $'uvicorn==0.28.0' $'wandb==0.17.4' $'watchdog~=4.0.0' $'weaviate_client==4.6.5' $'websockets~=12.0.0' $'wheel~=0.43.0'
b
That’s the content of the script that gets run - what actually happens when you run it? Do you get the same error as before?
c
yeah same error
b
There should be a log file for 
pip
too, which was in your original command:
Copy code
/private/var/folders/wq/xvgpz9v17qsbhb1vwz087v5w0000gn/T/pants-sandbox-y1SsV5/.tmp/pex-pip-log.vqga2q8u/pip.log
That may contain some more info as to what went wrong
c
that doesn't seem to exist
b
Right, probably because that was from yesterday’s output. If you run the same command again and save sandboxes, the output should contain a similar log file path that you can look at to see what is happening at the pip level
c
no I ran it again today
this is the command to run right?
Copy code
pants --keep-sandboxes=on_failure generate-lockfiles
b
Yup, that’s right
Your output from yesterday started with this log line in stderr:
Copy code
~/.cache/pants/named_caches/pex_root/venvs/0deb89b23adbbad7a5b9aac06d2362acf7572cfa/d819101e76d1257235093025e8afc37bee57f317/bin/python -sE ~/.cache/pants/named_caches/pex_root/venvs/0deb89b23adbbad7a5b9aac06d2362acf7572cfa/d819101e76d1257235093025e8afc37bee57f317/pex --disable-pip-version-check --no-python-version-warning --exists-action a --no-input --isolated -q --cache-dir ~/.cache/pants/named_caches/pex_root/pip/24.0/pip_cache --log /private/var/folders/wq/xvgpz9v17qsbhb1vwz087v5w0000gn/T/pants-sandbox-y1SsV5/.tmp/pex-pip-log.vqga2q8u/pip.log ...
That log file is very useful because it shows you the output of 
pip
c
sorry am I doing something wrong here:
Copy code
cat /private/var/folders/wq/xvgpz9v17qsbhb1vwz087v5w0000gn/T/pants-sandbox-0YLu14/.tmp/pex-pip-log.kq7a7tuj/pip.log
cat: /private/var/folders/wq/xvgpz9v17qsbhb1vwz087v5w0000gn/T/pants-sandbox-0YLu14/.tmp/pex-pip-log.kq7a7tuj/pip.log: No such file or directory
b
Hmm, no that looks fine to me. Does the sandbox directory of the parent exist? Ex:
Copy code
ls /private/var/folders/wq/xvgpz9v17qsbhb1vwz087v5w0000gn/T/pants-sandbox-0YLu14/
c
Copy code
ls /private/var/folders/wq/xvgpz9v17qsbhb1vwz087v5w0000gn/T/pants-sandbox-0YLu14/
__run.sh	pex
same thing the log doesn't exist here 😞
b
I’m guessing that log is a temporary file that gets removed when the script ends. One thing you could try is executing 
__run.sh
, and then tailing the log file while it is running to see the output
c
okay yeah I'll give that a go
b
For the 
ls
command, you may also have to specify 
-a
to see the 
.tmp
directory since it is hidden, so:
Copy code
ls -la /private/var/folders/wq/xvgpz9v17qsbhb1vwz087v5w0000gn/T/pants-sandbox-0YLu14/
c
ah good point but nothing exists in 
.tmp
b
Yeah, so probably a temporary file - I'd recommend the run + tail workaround I mentioned earlier
c
how would I know what the pip file i called when the script is run?
b
It should show that in the console output
Alternatively, you can just run the 
env -i
command directly
c
could you provide me the whole command cause the script runs quite quickly. I'm a bit confused
seems impossible to tail the pip log file
b
The command is local to you, so I can’t provide it - it’ll be the contents of the 
__run.sh
script which is what Pants runs underneath
c
oh okay I see
b
I’m a little surprised that the script runs that quickly, usually generating lockfiles is a long process unless you have relatively few dependencies
c
okay when I cat the 
__run.sh
file the pip.log file no longer shows
it doesn't print the logging directory until failure
so what I'm trying to do rn is run the lockfile generation, find the directly in which it has the 
__run.sh
file and the rerun the command but with tailing the file it's going to output the pip.log file to
the issue here is the pip.log file is not showing in 
__run.sh
b
Hard to say exactly what the problem is without seeing further into your environment, but there can be multiple sandboxes used during the generation process for different parts of it; you could instead make Pants retain all sandboxes for debugging purposes: https://www.pantsbuild.org/2.22/reference/global-options#keep_sandboxes and then you could look through all of them to figure out which one has the pip log
c
actually just remembered I posted something on the discussions forum as well. I'll follow this for debugging: https://github.com/pantsbuild/pants/discussions/21210
thanks for your help though!
11 Views
Open in Slack
PreviousNext
Powered by