Does anyone here use a private key with remote-caching? We have an existing bazel cache setup for another team and I figured I'd piggy back. However, their config is 3 files:

ca-cert.pem

,

client-cert.pem

, and

client-key.pem

. I've tried to merge them into one pem, but it fails with

Failed to read from remote cache (4 occurrences so far): Unauthenticated: "could not verify peer certificate"

-- it's quite sparse with logs, unfortunately.

do the private certificates need to be installed on the dev machine too? e.g. the user using the build cache?

Yes; that's how we have it set up. Self-signed certificate, including client-cert and ca-cert, and a private-key for the client-cert.

I've seen that error before when one of the intermediate certificates or the root one is off, (i doubt that's helpful but fwiw)

I think that might be correct; but both wget and curl are fine with it. So not quite sure.

Can you share the relevant parts of your Pants configuration?

Sure @fast-nail-55400 :

remote_ca_certs_path = ".certs/ca-cert.pem"
remote_mtls_certs_path = ".certs/merged.pem"  # this is something I've added. File is a concatenation of client-cert.pem, client-key.pem. But atm I'm running pants-from-source with hardcoded certs and it still fails
remote_cache_read = true
remote_cache_write = false
remote_execution = false
remote_execution_address = "grpcs://<snip>:30002"
remote_store_address = "grpcs://<snip>:30002"
remote_store_rpc_timeout_millis = 3600000

Will look at those settings, I've used the --log-show-rust-3rdparty already, and can see that it actually goes through a proper TLS negotiation but fails here: https://github.com/buchgr/bazel-remote/blob/2bcc2f59e111f71b4de4d84013f8e93a1b981872/server/grpc.go#L227-L229

What version of Pants?

I've done 2.17.0 and 2.18.0a0 which is what I had checked out locally.

If you are able to run Pants from source, adding some debugging statements to this part of the code https://github.com/pantsbuild/pants/blob/eecd07908868af3e3e0d97a91852de2785b7b16c/src/rust/engine/grpc_util/src/tls.rs#L78-L83 would allow us to see if the mTLS config is actually being added to the rustls config.

I got it working, after some more wiring. Put up a PR earlier today.

I was trying to see if the mTLS config was actually propagated down to the grpc_util crate and found https://github.com/pantsbuild/pants/blob/eecd07908868af3e3e0d97a91852de2785b7b16c/src/rust/engine/src/context.rs#L172 which would suggest that mTLS config was not being propagated.

Correct!