https://pantsbuild.org/ logo
Powered by
b

best-florist-45041

12/14/2021, 7:07 AM
Hi all, I'm testing out 
pants
and ran into with an interesting failure case with installing a 3rd party dependency (avro-python3==1.10.0). Short version: using 
ca_certs_path + indexes.add
for a company pypi repo caused installation of the setup_requires dependencies to fail. After commenting out those options in 
pants.toml
the installation would succeed. All other dependencies (including private ones) succeed fine with the ca_certs_path + indexes.add. 
pip install 'avro-python3==1.10.0'
works fine either way. Details: thanks to running with 
--pex-verbosity=9
(wish that was on the troubleshooting page!) found the following error:
Copy code
OSError: Could not find a suitable TLS CA certificate bundle, invalid path: cert.pem
Given I provided a full path for 
ca_certs_path
I'm wondering what could be stripping off the rest?
c

curved-television-6568

12/14/2021, 7:51 AM
We ❤️ improvements to the docs. There’s a “suggest edits” button on each page. Feel free to add to the troubleshooting page there 🙂
💯 1
b

best-florist-45041

12/14/2021, 8:35 PM
Thanks, just did so. I might be blind but I could not see any way to preview the changes, so hope it looks ok. 🙂
👍 1
I think I see how this error is occurring. If I understand correctly, these lines are replacing the cert path with a local file in the working directory for the pex build process. https://github.com/pantsbuild/pants/blob/release_2.9.0.dev3/src/python/pants/backend/python/util_rules/pex_cli.py#L130-L138 So when pants calls pex I see 
--cert cert.pem
in the options. All well and good. But then when pex-vendored setuptools installs setup_requires dependencies I believe that occurs in another temporary directory, one without the relative cert.pem, and so fails to find it during the install.
c

curved-television-6568

12/14/2021, 8:51 PM
ping @enough-analyst-54434 might have good insight here..
e

enough-analyst-54434

12/14/2021, 8:54 PM
Sounds like @best-florist-45041 is a good debugger. Pants has a bug in exactly the way he describes. @best-florist-45041 you can tie a bow on your gift by editing 
__run.sh
to make the path absolute and then running it.
... to verify that will actually fix things.
This will be interesting to fix for real if the 
__run.sh
experiment works. Its fundamentally hard right now to specify an absolute path in Pants sandbox process executions without introducing wrapper scripts which greatly complicates the implementation. We may need to support some controlled form of process argument env var interpolation so that we can say things like `argv=[..., "$PWD/cert.pem", ...]`and avoid the tricky wrappers.
b

best-florist-45041

12/14/2021, 9:10 PM
Thanks, but where is this 
__run.sh
located? The one in the tmp dir created?
e

enough-analyst-54434

12/14/2021, 9:10 PM
Yup.
It's changed recently, but hopefully this works for your Pants version: 
--no-process-execution-local-cleanup
b

best-florist-45041

12/14/2021, 9:14 PM
sorry, which tmp dir again. In this output I would guess /private/var/folders/h_/y0km0n8d3x77lxb7932_6lmm0000gn/T/process-executionZ9CdEV/.tmp but that is empty
Copy code
pid 77344 -> /Users/alexandervr/.cache/pants/named_caches/pex_root/venvs/f4fe22a38ad1716580d169deb527663c98f3b86d/571cba2af0d433b83b5f820563460668bf7bc7b8/pex --disable-pip-version-check --no-python-version-warning --exists-action a --isolated -q --cache-dir /Users/alexandervr/.cache/pants/named_caches/pex_root --log /private/var/folders/h_/y0km0n8d3x77lxb7932_6lmm0000gn/T/process-executionZ9CdEV/.tmp/tmp6oq_i19z/pip.log download --dest /private/var/folders/h_/y0km0n8d3x77lxb7932_6lmm0000gn/T/process-executionZ9CdEV/.tmp/tmp3_74ez1u/Users.alexandervr..asdf.installs.python.3.9.8.bin.python3.9 avro-python3==1.10.0
e

enough-analyst-54434

12/14/2021, 9:16 PM
If you re-run the pants command with both 
--no-process-execution-local-cleanup
and `--no-process-execution-local-cache`the sandbox to inspect should be the very last one printed out since Pants should fail trying to run that sandbox.
The last log line matching "Preserving local process execution dir".
b

best-florist-45041

12/14/2021, 9:19 PM
ah
Copy code
Preserving local process execution dir /private/var/folders/h_/y0km0n8d3x77lxb7932_6lmm0000gn/T/process-executionyPxlO2 for "Building requirements.pex with 1 requirement: avro-python3==1.10.0"
edited that cert.pem to be absolute and reran with those two options again but no dice
e

enough-analyst-54434

12/14/2021, 9:20 PM
Ok, so that's not the problem then.
This rings a Pip bug bell. Can you try running the pip command directly with the absolute cert path? The pip command is the one you snipped above with the 
pid 77344 -> ...
You'd want to look for a line like that bug though:
Copy code
... Could not fetch URL <https://il-app-61/repository/pypi_group/simple/setuptools/>: There was a problem confirming the ssl certificate: ...
From my reading of that bug and your report, the nail in the coffin would be if the problematic dependency in fact uses a PEP-517 build (pyproject.toml).
Ah - its a pure sdist and I think pip does use pep-517 by default even if not explicit in this case. So one workaround would be to pre-build a wheel for that and host the wheel internally.
(even though the command you snip above is a pip downloadand the issue is pip install, download still needs to run a build (
python setup.py egg_info
to get metadata in Requires-Python and Requires-Dist to continue the resolve).
b

best-florist-45041

12/14/2021, 9:33 PM
running that command with additional --cert still fails, but wondering if env var takes precedence to the option
oh wait nvm there is no env var
e

enough-analyst-54434

12/14/2021, 9:36 PM
I forgot about this, but there actually is, internally Pex does this: https://github.com/pantsbuild/pex/blob/7ff27a66cd0a8e6ae848dd1a329732163f77da8e/pex/pip.py#L147-L151
b

best-florist-45041

12/14/2021, 9:36 PM
and running with -vvvv shows different errors, e.g. 
ModuleNotFoundError: No module named 'pex'
e

enough-analyst-54434

12/14/2021, 9:37 PM
Ok. Well, the best debug would be to just translate the command line directly to pip and try that. There should be very little translation. I think just the 1st argument where you 
s|<path to>/pex|pip|
.
You'd want to try all 3 permutations of passing pip 
--cert
, using 
REQUESTS_CA_BUNDLE=... pip ...
instead and using 
PIP_CERT=... pip ...
instead.
b

best-florist-45041

12/14/2021, 9:39 PM
so replacing the pex binary with just 
pip
runs fine, but I don't need the --cert option given my path
err, .config/pip/pip.conf
e

enough-analyst-54434

12/14/2021, 9:41 PM
2 things then to try: 1. pip --version 2. disable pip.conf, Pex does this by passing 
--isolated
Thanks for the debug data - this is helpful.
b

best-florist-45041

12/14/2021, 9:42 PM
Copy code
pip --version
pip 21.2.4 from /Users/alexandervr/.asdf/installs/python/3.9.8/lib/python3.9/site-packages/pip (python 3.9)
e

enough-analyst-54434

12/14/2021, 9:44 PM
Ok, Pex uses a vendored copy of 20.3.4 so two paths to debug. Try 
--isolated
and the 3 combos with pip 20.3.4 instead. Then try 21.2.4 with said same 3 combos.
b

best-florist-45041

12/14/2021, 9:45 PM
ah it already has the 
--isolated
flag. Running
Copy code
pip --disable-pip-version-check --no-python-version-warning --exists-action a --isolated -q --cache-dir /Users/alexandervr/.cache/pants/named_caches/pex_root --log /private/var/folders/h_/y0km0n8d3x77lxb7932_6lmm0000gn/T/process-executionyPxlO2/.tmp/tmpzv7z441t/pip.log download --dest /private/var/folders/h_/y0km0n8d3x77lxb7932_6lmm0000gn/T/process-executionyPxlO2/.tmp/tmp4leuaqum/Users.alexandervr..asdf.installs.python.3.9.8.bin.python3.9 'avro-python3==1.10.0' --index-url <https://pypi.org/simple/> --extra-index-url https://<<omitted>>
e

enough-analyst-54434

12/14/2021, 9:49 PM
Ok. If any of the 3 pip 20.3.4 permutations work, obviously that's our fix. If not, its time to hunt down the bugfix 21.2.4 has over 20.3.4 and see if that can be patched in to our vendored copy.
b

best-florist-45041

12/14/2021, 9:49 PM
So this last command is doing 
pip download
not 
pip install
e

enough-analyst-54434

12/14/2021, 9:50 PM
Yup.
b

best-florist-45041

12/14/2021, 9:50 PM
downloading works fine, since it's not yet hitting the setup_requires
e

enough-analyst-54434

12/14/2021, 9:51 PM
Aha. Your snip here: https://pantsbuild.slack.com/archives/C046T6T9U/p1639516479189300?thread_ts=1639465673.165700&amp;cid=C046T6T9U had download so I assumed that was where you were hitting the error.
b

best-florist-45041

12/14/2021, 9:52 PM
that's the last thing in the logs
perhaps a red herring
ah with the pex-verbosity=9 I see
Copy code
/Users/alexandervr/.cache/pants/named_caches/pex_root/venvs/s/6551319a/venv/bin/python3.9', '-m', 'pip', '--disable-pip-version-check', 'wheel', '--no-deps', '-w', '/private/var/folders/h_/y0km0n8d3x77lxb7932_6lmm0000gn/T/process-executionUcA5s5/.tmp/tmp2gg1758p', '--quiet', 'pycodestyle'
e

enough-analyst-54434

12/14/2021, 9:53 PM
Then that's where it's failing. In download. It may not be obvious, but to download, pip has to do a resolve. That requires metadata to know install_requires and that requires setup_requires to run 
python setup.py egg_info
to get that dependency metadatat out of an sdist.
Aha, ok.
b

best-florist-45041

12/14/2021, 9:53 PM
pycodestyle being one the setup_requires deps
e

enough-analyst-54434

12/14/2021, 9:57 PM
Is that the full commend line? no 
--isolated
?
b

best-florist-45041

12/14/2021, 9:58 PM
that's in the logs without --isolated
e

enough-analyst-54434

12/14/2021, 9:58 PM
That can only happen if you pass Pex --client-cert. I don't think Pants does that though...
That must be pip calling itself.
b

best-florist-45041

12/14/2021, 10:00 PM
yes
I think I understand what you mean about 
pip download
being the cause here
would it help to see a whole stacktrace?
e

enough-analyst-54434

12/14/2021, 10:02 PM
So this is pointing to a bug being fixed in pip etween 20.3.4 and 21.2.4. This has been a bit messy though and I've lost track. Can you confirm the 2x2 matrix of [20.3.4, 21.2.4] x [--cert abspath, REQUESTS_CA_BUNDLE=abspath] for 
pip wheel ...
?
b

best-florist-45041

12/14/2021, 10:03 PM
for clarity, just in a separate venv right? I haven't gotten any of the cmds to fail outside of running pants
except for calling the 
.../pex
full command but there it failed due to not having pex
e

enough-analyst-54434

12/14/2021, 10:04 PM
Yes, in a seperate venv. Passing 
--isolated
in all 4 commands is critical for the repro as well.
b

best-florist-45041

12/14/2021, 10:06 PM
so like this?
Copy code
pip wheel --isolated 'avro-python3==1.10.0' --no-cache --cert /opt/homebrew/etc/openssl@1.1/cert.pem
e

enough-analyst-54434

12/14/2021, 10:06 PM
Yes, once for each version of pip.
Then if both work, we're done. If old pip fails, please try the env var.
b

best-florist-45041

12/14/2021, 10:07 PM
success after
Copy code
pip install 'pip==20.3.4' 'setuptools==57.5.0' 'wheel==0.37.0'
e

enough-analyst-54434

12/14/2021, 10:07 PM
oh wait
no
Missing the --index args
b

best-florist-45041

12/14/2021, 10:07 PM
lol right
still works with both --index and --extra-index-url
e

enough-analyst-54434

12/14/2021, 10:14 PM
For old pip? and abs path to cert? Can you paste the working command line with redactions if so?
And also if so, can you re-try the same command line but with the relative path to the cert to emulate the current Pants setup?
b

best-florist-45041

12/14/2021, 10:17 PM
These three all work:
Copy code
pip wheel --isolated 'avro-python3==1.10.0' --no-cache --cert /opt/homebrew/etc/openssl@1.1/cert.pem --index=<https://pypi.org/simple/> --extra-index-url=https://<redacted>

REQUESTS_CA_BUNDLE=/opt/homebrew/etc/openssl@1.1/cert.pem pip wheel --isolated 'avro-python3==1.10.0' --no-cache  --index=<https://pypi.org/simple/> --extra-index-url=https://

pip wheel --isolated 'avro-python3==1.10.0' --no-cache  --index=<https://pypi.org/simple/> --extra-index-url=https://
with results:
Copy code
Successfully built avro-python3
WARNING: You are using pip version 20.3.4; however, version 21.3.1 is available.
even if I put in an invalid cert path it works ??
e

enough-analyst-54434

12/14/2021, 10:19 PM
Yeah, so we are still missing some piece of the repro puzzle.
What is your relevant redacted pants.toml for 
[python-repos]
?
is it 
indexs=[...]
or 
indexes.add=[....]
?
I think your verbose snips from above implied 
indexes.add
since they had both IIRC.
b

best-florist-45041

12/14/2021, 10:22 PM
indexes.add
e

enough-analyst-54434

12/14/2021, 10:22 PM
I'm going to need to disappear here in ~10 minutes. You may want to gather up all the pertinent details in an issue. I do want to re-highlight that if this is the only problematic dist, pre-building its wheel and placing that in your custom index should work around.
b

best-florist-45041

12/14/2021, 10:23 PM
also I unset my REQUESTS_CA_BUNDLE and .config/pip/pip.conf and finally got to where passing the wrong cert file would error. So something is not so isolated with 
--isolated
big thanks for the deep dive, I'll try a few more things and report back
hah, finally got the repro!
Copy code
REQUESTS_CA_BUNDLE=cert.pem pip wheel --isolated 'avro-python3==1.10.0' --no-cache  --index=<https://pypi.org/simple/> --extra-index-url=https://
fails on setup_requires install
need to use relative path in the directory
that contains cert.pem
and it works on pip==21.2.4 fails with slightly different error
so it was the pip bug
e

enough-analyst-54434

12/14/2021, 10:34 PM
Ok. So we currently have no workaround except for pre-building the wheel. Can you do that and does that make sense?
Thanks for hanging in there with me on the debugging!
b

best-florist-45041

12/14/2021, 10:36 PM
not the best solution as would require building the wheel for at least the macos arm arch (local machine), but typically only CI-built packages go into our local artifactory
is there a way to specify a local wheel as a source in pants? And then have a script to build locally?
and this was a pretty interesting debugging session -- nice intro to the setuptools + pip internals
e

enough-analyst-54434

12/14/2021, 10:47 PM
Ah, sorry. So your repro proves the issue is in fact the relative path issue?
If so we can fix that
b

best-florist-45041

12/14/2021, 10:48 PM
yes, but only with env variable. Using 
--cert cert.pem
is fine
REQUESTS_CA_BUNDLE=cert.pem pip wheel ....
fails, but succeeds if given full path
e

enough-analyst-54434

12/14/2021, 10:54 PM
Ok, so this can be fixed on our end. I'll file an issue later tonight. Probably best to fix in Pex.
b

best-florist-45041

12/14/2021, 11:05 PM
great, thanks! And sorry we went down such a long hole. Reviewing above I should have caught that editing 
__run.sh
had no effect. Because it appears that even with 
--no-process-execution-local-cleanup --no-process-execution-local-cache
pants was using a new directory each time
yes, it appears that pants will reuse the same local cache only if the file is unchanged 😞
e

enough-analyst-54434

12/14/2021, 11:12 PM
Ah, I meant for you to edit 
__run.sh
and then run it yourself then and there. Pants doesn't actually use or generate that script normally. It's just for debugging and tinkering by hand in a saved sandbox.
Pants invokes the process directly from rust code normally.
b

best-florist-45041

12/14/2021, 11:22 PM
oh 😞 crap, I did not see the comment:
Copy code
# This command line should execute the same process as pants did internally.
e

enough-analyst-54434

12/14/2021, 11:25 PM
No worries. It's a miracle to get someone with the patience and mindset for a debugging partner. Thanks for digging and sticking with it.
➕ 1
Remote debugging by proxy is hard!
I want to set up a repro so this can be encapsulated in a failing test (Pex does this for --cert + --proxy using mitmproxy already so there is test infra for this sort of thing), but issue to follow along with is here: https://github.com/pantsbuild/pex/issues/1537
b

best-florist-45041

12/16/2021, 1:07 AM
the following works nicely: Replace the 
[GLOBAL] ca_certs_path
with
Copy code
[subprocess-environment]
env_vars.add = [
  "REQUESTS_CA_BUNDLE"
]
e

enough-analyst-54434

12/17/2021, 4:21 AM
Ah, yes. Great. Thanks for figuring that workaround out.
Alright, proper fix up here: https://github.com/pantsbuild/pex/pull/1538 I'll try to get this incorporated n the Pants 2.9.0rc0 release tomorrow via a Pex 2.1.58 release.
7 Views
Powered by