I had a question about starting to use remote caching (e.g. https://www.toolchain.com). I like the idea and we'd like to do it BUT we do something odd with our dependencies. Per a previous conversation on this Slack channel we're using PyTorch and sometimes need the GPU version of it and sometime the CPU version. Annoyingly, the CPU and GPU versions are the same pip version and the only difference between them is which

--extra-index-url

you pass to

pip

. To make things realitvely easy we control this with an env variable and a wrapper script: if the variable is set we set the

PANTS_PYTHON_REPOS_INDEXES

and

PANTS_PYTHON_RESOLVES

environment variables to pull the right dependencies. So.... given our funky setup, can we still use remote caching or is it going to be unaware of those variables and thus deliver the wrong dependencies (e.g. if I build with CPU and my collegue sets those env vars and thus expects a GPU build will they end up pulling the CPU dependencies from the cache)?

Oh hey! I have a novel way of solving the torch issue, FWIW. Was gonna blog about it in the next months or so. DM me if youre interested and I can give you the cliffs notes

So.... given our funky setup, can we still use remote caching or is it going to be unaware of those variables and thus deliver the wrong dependencies (e.g. if I build with CPU and my collegue sets those env vars and thus expects a GPU build will they end up pulling the CPU dependencies from the cache)?

Pants is aware of all environment variables fed to processes, and will do the right thing here.

Thanks!

Happy to help with onboarding to Toolchain! And we have a Slack channel for support which I can send you an invite for.

Thanks! We definitely want to move in that direction eventually but I think we're going to back-burner for now. hopefully return to it in a few weeks.

It’s very easy, just a web signup and a config change.