https://pantsbuild.org/ logo
#general
Title
# general
s

salmon-nest-5801

05/01/2021, 10:22 PM
Hello everyone 👋. I'm running into a strange issue that I can't seem to sort out and I'm wondering if anyone else has any suggestions. My company uses dynamic SSL Inspection so we have an SSL Certificate we need to inject into our CA bundle. I've done that on my system. I've generated a pem bundle (I've even written a nice tool that does this from the macOS keychain, and puts it in a place the homebrew openssl expects it to be). I've set the requisite
REQUESTS_CA_BUNDLE
and
PIP_CERT
, etc. I've also experimented with
PANTS_CA_CERTS_PATH
and
--ca-certs-path=${SSL_CERT_FILE}
(the variable being a handy reference to the bundle I generated). But regardless of what I set, I always get the following when trying to do anything other than
--version
with pants:
Copy code
Exception: Error downloading file: error sending request for url (<https://github.com/pantsbuild/pex/releases/download/v2.1.35/pex>): error trying to connect: invalid certificate: BadDER
I thought maybe there was some sort of issue with pants not liking the openssl that homebrew has installed, and that maybe I needed to rebuild it from sources, so I attempted that - same output. I have longer logs from that if anyone wants to see (I didn't post them here as this is already too long). Regardless of what I set, if I set those variables, if I don't set them, I get the same response. I know
pip
and anything
requests
based work perfectly fine on my system - I needed those awhile ago and I sorted them out; I know for sure that the generated bundle works perfectly fine with those tools, and other things like
pip
,
pipenv
,
poetry
, etc. Any ideas? I'm just looking for a workaround that isn't getting out from behind the SSL Inspection server (I could, but then this won't work for anyone if they're ever inside the company intranet, and that's not really a solution). I can set whatever complicated environment I need. I know I could host the artifacts myself a-la proxy limitations, but I'd really rather not do that either. I really think this should work, I just can't figure out why it doesn't.
f

fast-nail-55400

05/01/2021, 11:10 PM
pex
is downloaded via the download intrinsic of the Rust engine. Thus, configuring Python HTTP libraries would not have any effect on the Rust code since the Rust code is using the
reqwest
crate.
a

average-vr-56795

05/01/2021, 11:12 PM
One thing I'll note is that right now there's a flag to set the CA certs for remote execution, and it would probably make sense to generalise that to be "CA certs for everything Pants knows how to configure"...
f

fast-nail-55400

05/01/2021, 11:13 PM
actually I spoke too soon, the code right before the client setup does setup certs:
Copy code
// These certs are for downloads, not to be confused with the ones used for remoting.
    let ca_certs = Self::load_certificates(ca_certs_path)?;

    let http_client_builder = ca_certs
      .iter()
      .fold(reqwest::Client::builder(), |builder, cert| {
        builder.add_root_certificate(cert.clone())
      });
    let http_client = http_client_builder
      .build()
      .map_err(|err| format!("Error building HTTP client: {}", err))?;
but I believe that just sets up the certs as what the server cert will be checked against
@salmon-nest-5801: Did your cert need to be set as a client certificate offered by Pants to the remote server?
s

salmon-nest-5801

05/01/2021, 11:20 PM
No, it's not a client certificate. It's a CA certificate. The SSL Inspection works by rewriting the incoming cert with its own certificate, and resigning things against the CA I need to add. It just needs to be in the set of CA certificates that is offered up when
reqwest
connects to the server. Is it not using openssl under the covers? I'm kind of surprised - the bundle I have (which does include this additional CA) is in a place that homebrew openssl intrinsincally knows to find on this machine...
f

fast-nail-55400

05/01/2021, 11:22 PM
No, the
reqwest
crate is configured to use
rustls
(which is a Rust implementation of TLS)
s

salmon-nest-5801

05/01/2021, 11:23 PM
Ah, that'd do it. Any particular reason? That's fine as long as it handles the same inputs (or the differences are documented)
f

fast-nail-55400

05/01/2021, 11:24 PM
It would be useful to see debug logging with Rust debug logging as well. Use
-ldebug --log-show-rust-3rdparty --show-log-target
s

salmon-nest-5801

05/01/2021, 11:25 PM
Was just about to ask, thanks
f

fast-nail-55400

05/01/2021, 11:25 PM
the interesting log lines will be from
reqwest
and
rustls
s

salmon-nest-5801

05/01/2021, 11:25 PM
Let me get setup with something that isn't building pants itself so debugging is easier (unless theres some easy way to inject that I don't know about)
👍 1
Back in a little bit
f

fast-nail-55400

05/01/2021, 11:35 PM
Ah, that’d do it. Any particular reason? That’s fine as long as it handles the same inputs (or the differences are documented)
rustls doesn’t suffer from the memory unsafety issues of C which openssl is written in
s

salmon-nest-5801

05/01/2021, 11:40 PM
Gotcha. Managed to get you the logs
20210501194000_pants_debug.log
f

fast-nail-55400

05/01/2021, 11:43 PM
invalid certificate: BadDER
s

salmon-nest-5801

05/01/2021, 11:44 PM
I've checked the cert. It's well formed
It's decodable as a complete certificate in the packet
f

fast-nail-55400

05/01/2021, 11:45 PM
this issue on rustls seems relevant: https://github.com/ctz/rustls/issues/127
s

salmon-nest-5801

05/01/2021, 11:47 PM
Indeed, thanks. Now that I know what's involved I can find things like that 😛
I've got to go make dinner but I'll try to keep looking through this afterwards, and definitely when I'm back to work formally on Monday
This was just bugging me so I wanted to take a look today
👍 1
Last thing - that does indeed look like the issue. The SSL Inspection cert (it's the intermediate CA signing cert) and the issuing CA cert both only have a Common Name set, and no subjectAltName extensions. So if I'm understanding this issue correctly, the request implodes because one of the methods goes looking for the subjectAltName extensions and can't find them.
I'm not sure what to do about that...
All I can say is that these type of dynamic SSL Injection/Introspection systems are becoming incredibly common at large corporations for security reasons, so that malicious actors can't hide messages home in SSL packets. I promise needing this to work isn't a one-off sort of thing; it's only going to become more and more common as SSL becomes more and more pervasive and more companies adopt these systems.
I've never programmed rust, so, I'm probably not the person you want tampering with the source code to patch out the issue. I can try to create a MR but I'd have to learn the language first (I'm an embedded software engineer with a comprehensive understanding of low-level system structure if that'd help...)
Anywho, now I've gotta go 😅
f

fast-nail-55400

05/02/2021, 12:53 AM
Actually, maybe just open an issue at https://github.com/pantsbuild/pants/issues when you get a chance next week with your use case and the log.?Then one of the people with Rust experience can figure out next steps to fix.
👍 1
Whether that is something in code to handle this kind of certificate or figuring out whether these certificates need the subjectAltName claim.
h

happy-kitchen-89482

05/03/2021, 12:37 PM
To close the loop: the issue was not subjectAltName, that turned out to be a red herring. It was that one of the certs in the bundle was not a CA cert. See https://github.com/pantsbuild/pants/issues/12000.
s

salmon-nest-5801

05/03/2021, 8:05 PM
Half unclosing the loop: While that was a red herring, there seems to be a deeper issue with bytestream decoding. More discussion is happening on the issue: https://github.com/pantsbuild/pants/issues/12000 😄