Hello everyone 👋. I'm running into a strange issue that I can't seem to sort out and I'm wondering if anyone else has any suggestions. My company uses dynamic SSL Inspection so we have an SSL Certificate we need to inject into our CA bundle. I've done that on my system. I've generated a pem bundle (I've even written a nice tool that does this from the macOS keychain, and puts it in a place the homebrew openssl expects it to be). I've set the requisite

REQUESTS_CA_BUNDLE

and

PIP_CERT

, etc. I've also experimented with

PANTS_CA_CERTS_PATH

and

--ca-certs-path=${SSL_CERT_FILE}

(the variable being a handy reference to the bundle I generated). But regardless of what I set, I always get the following when trying to do anything other than

--version

with pants:

Exception: Error downloading file: error sending request for url (<https://github.com/pantsbuild/pex/releases/download/v2.1.35/pex>): error trying to connect: invalid certificate: BadDER

I thought maybe there was some sort of issue with pants not liking the openssl that homebrew has installed, and that maybe I needed to rebuild it from sources, so I attempted that - same output. I have longer logs from that if anyone wants to see (I didn't post them here as this is already too long). Regardless of what I set, if I set those variables, if I don't set them, I get the same response. I know

pip

and anything

requests

based work perfectly fine on my system - I needed those awhile ago and I sorted them out; I know for sure that the generated bundle works perfectly fine with those tools, and other things like

pip

,

pipenv

,

poetry

, etc. Any ideas? I'm just looking for a workaround that isn't getting out from behind the SSL Inspection server (I could, but then this won't work for anyone if they're ever inside the company intranet, and that's not really a solution). I can set whatever complicated environment I need. I know I could host the artifacts myself a-la proxy limitations, but I'd really rather not do that either. I really think this should work, I just can't figure out why it doesn't.

pex

is downloaded via the download intrinsic of the Rust engine. Thus, configuring Python HTTP libraries would not have any effect on the Rust code since the Rust code is using the

reqwest

crate.

One thing I'll note is that right now there's a flag to set the CA certs for remote execution, and it would probably make sense to generalise that to be "CA certs for everything Pants knows how to configure"...

actually I spoke too soon, the code right before the client setup does setup certs:

// These certs are for downloads, not to be confused with the ones used for remoting.
    let ca_certs = Self::load_certificates(ca_certs_path)?;

    let http_client_builder = ca_certs
      .iter()
      .fold(reqwest::Client::builder(), |builder, cert| {
        builder.add_root_certificate(cert.clone())
      });
    let http_client = http_client_builder
      .build()
      .map_err(|err| format!("Error building HTTP client: {}", err))?;

No, it's not a client certificate. It's a CA certificate. The SSL Inspection works by rewriting the incoming cert with its own certificate, and resigning things against the CA I need to add. It just needs to be in the set of CA certificates that is offered up when

reqwest

connects to the server. Is it not using openssl under the covers? I'm kind of surprised - the bundle I have (which does include this additional CA) is in a place that homebrew openssl intrinsincally knows to find on this machine...

No, the

reqwest

crate is configured to use

rustls

(which is a Rust implementation of TLS)

Ah, that'd do it. Any particular reason? That's fine as long as it handles the same inputs (or the differences are documented)

It would be useful to see debug logging with Rust debug logging as well. Use

-ldebug --log-show-rust-3rdparty --show-log-target

Was just about to ask, thanks

the interesting log lines will be from

reqwest

and

rustls

Let me get setup with something that isn't building pants itself so debugging is easier (unless theres some easy way to inject that I don't know about)

Ah, that’d do it. Any particular reason? That’s fine as long as it handles the same inputs (or the differences are documented)

rustls doesn’t suffer from the memory unsafety issues of C which openssl is written in

Gotcha. Managed to get you the logs

I've checked the cert. It's well formed

this issue on rustls seems relevant: https://github.com/ctz/rustls/issues/127

Indeed, thanks. Now that I know what's involved I can find things like that 😛

Actually, maybe just open an issue at https://github.com/pantsbuild/pants/issues when you get a chance next week with your use case and the log.?Then one of the people with Rust experience can figure out next steps to fix.

To close the loop: the issue was not subjectAltName, that turned out to be a red herring. It was that one of the certs in the bundle was not a CA cert. See https://github.com/pantsbuild/pants/issues/12000.

Half unclosing the loop: While that was a red herring, there seems to be a deeper issue with bytestream decoding. More discussion is happening on the issue: https://github.com/pantsbuild/pants/issues/12000 😄