https://pantsbuild.org/ logo
Join Slack
Powered by
Thoughts on how to proceed w/ Pex lockfiles not ha...
# development
h
Thoughts on how to proceed w/ Pex lockfiles not having the same SSL fallbacks as when it uses pip to install things? https://github.com/pantsbuild/pex/issues/1674
✅ 1
For Pants users: the issue seems to be when you a) have not set 
[GLOBAL].ca_certs_path
and b) have 
SSL_CERT_FILE
in your env *but did not tell Pants about it. Consuming a lockfile might give you a certificate issue. For pantsbuild/pants, this breaks our Linux Build Wheels job which uses the manylinux2014 docker image. Everything else is fine though. There are 3 solutions from Pants's perspective, I think: 1. Expect users to set 
[GLOBAL].ca_certs_path
2. Add 
SSL_CERT_FILE
to the default env vars 3. Change Pex to behave more like pip
b
I wouldn't even know what to set 
ca_certs_path
to 🙈
h
fwit, I found it by running 
env
and then copying the value for 
SSL_CERT_FILE
. But yeah, definitely confusing not intuitive I suspect 2 or 3 are the better options
Okay cool, John commented on the ticket that using 
certifi
is a sensible fallback. Problem solved
Problem not yet solved - @enough-analyst-54434 cautions in https://github.com/pantsbuild/pants/pull/14771#discussion_r827534126 that "solving" this problem by having Pex embed its own root certs via 
certifi
is not perfect. It doesn't help us address how we handle other language integrations like Go & JS that need certs. Not guaranteed we can have those integrations embed their own root certs via something like `certifi`; even if they could:
having every vertical embed its own root cert bundle of varying birth date & pedigree is ... clearly not awesome.
Which would suggest we consider options 1 or 2. -- cc @happy-kitchen-89482 @witty-crayon-22786 @bitter-ability-32190, thoughts? This reminds me of the discussion at https://github.com/pantsbuild/pants/issues/14281 on Pants lockfile headers: how much to unify a solution vs deal with things on a per-language ecosystem.
I do think option 2 of adding 
SSL_CERT_FILE
to the default env is reasonable, user-friendly, and easy to implement. I think my vote is for that
w
yea, i agree. with a comment in the option help reminding people that it’s possible to remove from the default set via 
option.remove =
👍 2
h
Related, @happy-kitchen-89482 @witty-crayon-22786 do you remember why we do not set 
HTTPS_PROXY
and 
HTTP_PROXY
by default?
b
Whatever "works" works for me. I just have an aversion to having to boilerplate my root toml with settings I don't quite understand (option 1)
w
i think that with any of these env vars, there is some amount of risk of them being different in meaningful ways between machines, which could cause cache misses. they are much lower risk than 
PATH
though. see https://github.com/pantsbuild/pants/issues/12203
👍 1
and so the goal would probably be to choose the right ones to bring to an administrator’s attention in order to make them consistent
it looks like (at least with openssl) the location is fairly standard across linuxes
👍 1
so i don’t think that we would be masking something that was important to be aware of.
but proxies seem more likely to be different between machines, and seem like something that an admin should be paying more attention to.
👍 1
c
I think it would be good with some form of 
info
goal, to help identify settings that are potential cache “hotspots” if we accumulate more of these default settings that may have a negative impact on performance due to cache misses in favour of usability.
settings I don’t quite understand (option 1)
(this is perhaps not the place, but) the 
ca_cert_path
, or 
SSL_CERT_FILE
etc, what all these options do, is to allow providing your own set of certificates (root certs, sub ca certs etc) to support communicating securely with servers that have custom/in house created certificates, and thus won’t work with only the officially provided certificate bundles.
so as long as you only communicate with public servers, these won’t come into play for you…
h
The issue with adding env vars by default is that they reduce cache hits
HTTPS_PROXY is likely to be set to the same value for everyone, so that's less of a problem
But if SSL_CERT_FILE is /home/myusername/.ssl/certs or whatever then that will ruin remote cache hits
Even if you don't need SSL_CERT_FILE at all
w
@happy-kitchen-89482: see above: it lives in a relatively standard location
h
Arguably if you don't need it, it won't be set in your environment, I guess
h
Arguably if you don't need it, it won't be set in your environment, I guess
Exactly this. Note that our Linux and macOS jobs were all fine because they don't set SSL_CERT_FILE. It was only the manylinux2014 docker image
w
i also thought that 
HTTPS_PROXY
would be more likely to be set differently on different hosts: particularly laptops. but maybe i don’t understand how it is used.
h
Thinking more widely: I could make an argument that we should not mix the environment into process fingerprints (or at least not by default).
c
or at least not by default
Or, it’s a list of env vars to consider for the fingerprint. So you can have those that potentially affect the resulting data of the Process included, like PATH, while those that are only used to facilitate the execution to function, like ssl and proxy settings to be excluded.
Open in Slack
PreviousNext
Powered by