So.... given our funky setup, can we still use remote caching or is it going to be unaware of those variables and thus deliver the wrong dependencies (e.g. if I build with CPU and my collegue sets those env vars and thus expects a GPU build will they end up pulling the CPU dependencies from the cache)?

Pants is aware of all environment variables fed to processes, and will do the right thing here.