sorry if I misunderstand, but you are using remote cache, would you consider disabling local cache so it's not being used at all? Given the access is configured properly and you are not transferring hundreds of megabytes as artifacts, you could just rely on the remote cache and have local cache disabled, see
https://www.pantsbuild.org/2.18/reference/global-options#local_cache.
If your CI builds take place in a fresh environment (i.e. each build starts a new container with no local cache), then IIUC the local cache will only be shared between Pants calls in your build. Maybe it's worth checking the performance of your builds in CI with local cache disabled to confirm relying on remote cache only is feasible?