I know this is outside the scope of pants, but would anyone happen to have any guides/tips for setting up a remote caching server? It would be nice to know a bit of context for myself before I go ask devops to set it up for my team

Tip #1: Don't use htpasswd auth to connect to it. Use mTLS. Then your cache service can be exposed to the internet. With htpasswd you are open to brute force attacks. Tip #2: Monitor your CPU/Memory usage and fine tune your container resoucres Tip #3: Leverage built in auth to connect to the storage backend. IN my case I use the default azure credential. Tip #4: If you were previously caching some artifacts locally, you need to bust previous caches Tip #5: Even though mTLS is on, leverage network limitations to deny public traffic from the internet. Only whitelisting the hosts you want to connect. We run this as a python script that updates the ingress to allow connections from the github actions worker and then we cleanup at the end with another script. Maybe I will remember more but it's not an easy job to set everything up and many different aspects to think about. But the final result is really rewarding.

As a warning, I've been a bit underwhelmed by pants' reuse of cache across machines and have run into lots of misses. I think pants puts things into the cache keys that vary slightly between machines (eg env vars that you use, maybe even $PATH, I am not totally clear). Makes sense for fully reproducible builds, but makes it less useful than expected. Seems to be more useful for CI.