best-florist-45041
12/14/2021, 7:07 AMpants
and ran into with an interesting failure case with installing a 3rd party dependency (avro-python3==1.10.0).
Short version: using ca_certs_path + indexes.add
for a company pypi repo caused installation of the setup_requires dependencies to fail. After commenting out those options in pants.toml
the installation would succeed. All other dependencies (including private ones) succeed fine with the ca_certs_path + indexes.add. pip install 'avro-python3==1.10.0'
works fine either way.
Details: thanks to running with --pex-verbosity=9
(wish that was on the troubleshooting page!) found the following error:
OSError: Could not find a suitable TLS CA certificate bundle, invalid path: cert.pem
Given I provided a full path for ca_certs_path
I'm wondering what could be stripping off the rest?curved-television-6568
12/14/2021, 7:51 AMbest-florist-45041
12/14/2021, 8:35 PM--cert cert.pem
in the options. All well and good. But then when pex-vendored setuptools installs setup_requires dependencies I believe that occurs in another temporary directory, one without the relative cert.pem, and so fails to find it during the install.curved-television-6568
12/14/2021, 8:51 PMenough-analyst-54434
12/14/2021, 8:54 PM__run.sh
to make the path absolute and then running it.__run.sh
experiment works. Its fundamentally hard right now to specify an absolute path in Pants sandbox process executions without introducing wrapper scripts which greatly complicates the implementation. We may need to support some controlled form of process argument env var interpolation so that we can say things like `argv=[..., "$PWD/cert.pem", ...]`and avoid the tricky wrappers.best-florist-45041
12/14/2021, 9:10 PM__run.sh
located? The one in the tmp dir created?enough-analyst-54434
12/14/2021, 9:10 PM--no-process-execution-local-cleanup
best-florist-45041
12/14/2021, 9:14 PMpid 77344 -> /Users/alexandervr/.cache/pants/named_caches/pex_root/venvs/f4fe22a38ad1716580d169deb527663c98f3b86d/571cba2af0d433b83b5f820563460668bf7bc7b8/pex --disable-pip-version-check --no-python-version-warning --exists-action a --isolated -q --cache-dir /Users/alexandervr/.cache/pants/named_caches/pex_root --log /private/var/folders/h_/y0km0n8d3x77lxb7932_6lmm0000gn/T/process-executionZ9CdEV/.tmp/tmp6oq_i19z/pip.log download --dest /private/var/folders/h_/y0km0n8d3x77lxb7932_6lmm0000gn/T/process-executionZ9CdEV/.tmp/tmp3_74ez1u/Users.alexandervr..asdf.installs.python.3.9.8.bin.python3.9 avro-python3==1.10.0
enough-analyst-54434
12/14/2021, 9:16 PM--no-process-execution-local-cleanup
and `--no-process-execution-local-cache`the sandbox to inspect should be the very last one printed out since Pants should fail trying to run that sandbox.best-florist-45041
12/14/2021, 9:19 PMPreserving local process execution dir /private/var/folders/h_/y0km0n8d3x77lxb7932_6lmm0000gn/T/process-executionyPxlO2 for "Building requirements.pex with 1 requirement: avro-python3==1.10.0"
edited that cert.pem to be absolute and reran with those two options again but no diceenough-analyst-54434
12/14/2021, 9:20 PMpid 77344 -> ...
... Could not fetch URL <https://il-app-61/repository/pypi_group/simple/setuptools/>: There was a problem confirming the ssl certificate: ...
From my reading of that bug and your report, the nail in the coffin would be if the problematic dependency in fact uses a PEP-517 build (pyproject.toml).python setup.py egg_info
to get metadata in Requires-Python and Requires-Dist to continue the resolve).best-florist-45041
12/14/2021, 9:33 PMenough-analyst-54434
12/14/2021, 9:36 PMbest-florist-45041
12/14/2021, 9:36 PMModuleNotFoundError: No module named 'pex'
enough-analyst-54434
12/14/2021, 9:37 PMs|<path to>/pex|pip|
.--cert
, using REQUESTS_CA_BUNDLE=... pip ...
instead and using PIP_CERT=... pip ...
instead.best-florist-45041
12/14/2021, 9:39 PMpip
runs fine, but I don't need the --cert option given my pathenough-analyst-54434
12/14/2021, 9:41 PM--isolated
best-florist-45041
12/14/2021, 9:42 PMpip --version
pip 21.2.4 from /Users/alexandervr/.asdf/installs/python/3.9.8/lib/python3.9/site-packages/pip (python 3.9)
enough-analyst-54434
12/14/2021, 9:44 PM--isolated
and the 3 combos with pip 20.3.4 instead. Then try 21.2.4 with said same 3 combos.best-florist-45041
12/14/2021, 9:45 PM--isolated
flag. Running
pip --disable-pip-version-check --no-python-version-warning --exists-action a --isolated -q --cache-dir /Users/alexandervr/.cache/pants/named_caches/pex_root --log /private/var/folders/h_/y0km0n8d3x77lxb7932_6lmm0000gn/T/process-executionyPxlO2/.tmp/tmpzv7z441t/pip.log download --dest /private/var/folders/h_/y0km0n8d3x77lxb7932_6lmm0000gn/T/process-executionyPxlO2/.tmp/tmp4leuaqum/Users.alexandervr..asdf.installs.python.3.9.8.bin.python3.9 'avro-python3==1.10.0' --index-url <https://pypi.org/simple/> --extra-index-url https://<<omitted>>
enough-analyst-54434
12/14/2021, 9:49 PMbest-florist-45041
12/14/2021, 9:49 PMpip download
pip install
enough-analyst-54434
12/14/2021, 9:50 PMbest-florist-45041
12/14/2021, 9:50 PMenough-analyst-54434
12/14/2021, 9:51 PMbest-florist-45041
12/14/2021, 9:52 PM/Users/alexandervr/.cache/pants/named_caches/pex_root/venvs/s/6551319a/venv/bin/python3.9', '-m', 'pip', '--disable-pip-version-check', 'wheel', '--no-deps', '-w', '/private/var/folders/h_/y0km0n8d3x77lxb7932_6lmm0000gn/T/process-executionUcA5s5/.tmp/tmp2gg1758p', '--quiet', 'pycodestyle'
enough-analyst-54434
12/14/2021, 9:53 PMpython setup.py egg_info
to get that dependency metadatat out of an sdist.best-florist-45041
12/14/2021, 9:53 PMenough-analyst-54434
12/14/2021, 9:57 PM--isolated
?best-florist-45041
12/14/2021, 9:58 PMenough-analyst-54434
12/14/2021, 9:58 PMbest-florist-45041
12/14/2021, 10:00 PMpip download
being the cause hereenough-analyst-54434
12/14/2021, 10:02 PMpip wheel ...
?best-florist-45041
12/14/2021, 10:03 PM.../pex
full command but there it failed due to not having pexenough-analyst-54434
12/14/2021, 10:04 PM--isolated
in all 4 commands is critical for the repro as well.best-florist-45041
12/14/2021, 10:06 PMpip wheel --isolated 'avro-python3==1.10.0' --no-cache --cert /opt/homebrew/etc/openssl@1.1/cert.pem
enough-analyst-54434
12/14/2021, 10:06 PMbest-florist-45041
12/14/2021, 10:07 PMpip install 'pip==20.3.4' 'setuptools==57.5.0' 'wheel==0.37.0'
enough-analyst-54434
12/14/2021, 10:07 PMbest-florist-45041
12/14/2021, 10:07 PMenough-analyst-54434
12/14/2021, 10:14 PMbest-florist-45041
12/14/2021, 10:17 PMpip wheel --isolated 'avro-python3==1.10.0' --no-cache --cert /opt/homebrew/etc/openssl@1.1/cert.pem --index=<https://pypi.org/simple/> --extra-index-url=https://<redacted>
REQUESTS_CA_BUNDLE=/opt/homebrew/etc/openssl@1.1/cert.pem pip wheel --isolated 'avro-python3==1.10.0' --no-cache --index=<https://pypi.org/simple/> --extra-index-url=https://
pip wheel --isolated 'avro-python3==1.10.0' --no-cache --index=<https://pypi.org/simple/> --extra-index-url=https://
Successfully built avro-python3
WARNING: You are using pip version 20.3.4; however, version 21.3.1 is available.
enough-analyst-54434
12/14/2021, 10:19 PM[python-repos]
?indexs=[...]
or indexes.add=[....]
?indexes.add
since they had both IIRC.best-florist-45041
12/14/2021, 10:22 PMenough-analyst-54434
12/14/2021, 10:22 PMbest-florist-45041
12/14/2021, 10:23 PM--isolated
REQUESTS_CA_BUNDLE=cert.pem pip wheel --isolated 'avro-python3==1.10.0' --no-cache --index=<https://pypi.org/simple/> --extra-index-url=https://
fails on setup_requires installenough-analyst-54434
12/14/2021, 10:34 PMbest-florist-45041
12/14/2021, 10:36 PMenough-analyst-54434
12/14/2021, 10:47 PMbest-florist-45041
12/14/2021, 10:48 PM--cert cert.pem
is fineREQUESTS_CA_BUNDLE=cert.pem pip wheel ....
fails, but succeeds if given full pathenough-analyst-54434
12/14/2021, 10:54 PMbest-florist-45041
12/14/2021, 11:05 PM__run.sh
had no effect. Because it appears that even with --no-process-execution-local-cleanup --no-process-execution-local-cache
pants was using a new directory each timeenough-analyst-54434
12/14/2021, 11:12 PM__run.sh
and then run it yourself then and there. Pants doesn't actually use or generate that script normally. It's just for debugging and tinkering by hand in a saved sandbox.best-florist-45041
12/14/2021, 11:22 PM# This command line should execute the same process as pants did internally.
enough-analyst-54434
12/14/2021, 11:25 PMbest-florist-45041
12/16/2021, 1:07 AM[GLOBAL] ca_certs_path
with
[subprocess-environment]
env_vars.add = [
"REQUESTS_CA_BUNDLE"
]
enough-analyst-54434
12/17/2021, 4:21 AM