i have a pants command that i invoke like this in my ci pipeline:

pants package src/python/mypackage:pex-tool-in-mypackage

then i run it:

python dist/tool-in-mypackge.pex args...

my ci pipeline instances are ephemeral and don't keep track of any state unless i go out of my way to cache it the sources that

tool-in-mypackage

depend on rarely change is there a way to extract from pants a list of all the files that this tool depends on, so i can fingerprint those files, cache the pex, and get the pex file from cache instead of re-creating it every time? it would save a ton of my time in my ci pipeline

Are you familiar with the CI caching recommendations in https://www.pantsbuild.org/2.21/docs/using-pants/using-pants-in-ci#directories-to-cache?

Computing such a coarse hash, and saving and restoring large directories, can be unwieldy. So this may be impractical and slow on medium and large repos.

it's a super duper large mono repo and i'm trying to develop a cache for a small job

pants will automatically manage its own cache outside CI

You could just enable the Pants cache only for that one job. The issue I see is you are suggesting avoiding all of the code in the existing Pants cache system in order to develop your own caching mechanism ("is there a way to extract from pants a list of all the files that this tool depends on, so i can fingerprint those files, cache the pex"). That seems duplicative. I would like to understand the limitations you have which are preventing use of the Pants caching mechanism.

see these flags

--changed-since=<any git ref>
--changed-dependents=transitive

you can supply a path at the end of the command like my/sub/path/some/project/:: as well

thanks will check this

but yeah, I agree with @fast-nail-55400 and would definitely try to see if you can tune pants rather than try to roll something outside

also, recent work has added an experimental "github actions" cache provider which can read/write cache entries directly to GitHub Actions cache. See https://www.pantsbuild.org/2.21/docs/using-pants/remote-caching-and-execution/remote-caching#github-actions-cache.

there is some behavior around the cache that i dont understand we have a pants cache based on the pants.toml and we save the cache right after installing pants i restore it, and in some of our older jobs, it seems like the cache is ignored and this happens:

Bootstrapping Pants 2.17.0
Installing pantsbuild.pants==2.17.0 into a virtual environment at /root/.cache/nce/60b513559c7b53eb2acecbd7b8aaaeb942686f3997d07fa77377b51324f58fda/bindings/venvs/2.17.0
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0.0/2.1 MB ? eta -:--:--     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸ 2.0/2.1 MB 81.3 MB/s eta 0:00:01     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.1/2.1 MB 54.6 MB/s eta 0:00:00
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0.0/819.3 kB ? eta -:--:--     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 819.3/819.3 kB 52.8 MB/s eta 0:00:00
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0.0/65.8 kB ? eta -:--:--     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 65.8/65.8 kB 8.6 MB/s eta 0:00:00
New virtual environment successfully created at /root/.cache/nce/60b513559c7b53eb2acecbd7b8aaaeb942686f3997d07fa77377b51324f58fda/bindings/venvs/2.17.0.

but something changed recently,

Failed to create Pants virtual environment.
Error: Command '['/home/circleci/.cache/nce/ab1acf935c4cc43338c604ae7d0f6aa2419f2415d94eb9cae381601dbba70a61/bindings/pex_root/venvs/10065e08ffb3b880c0f2fc1126cc528a64b9e10b/327f457f25e419eeb572d4624fc6cd8ccb017a05/bin/python', '/tmp/tmpcru8_3dl.pex', 'venv', '--prompt', 'Pants 2.17.0', '--compile', '--pip', '--collisions-ok', '--no-emit-warnings', '--disable-cache', '/root/.cache/nce/ab1acf935c4cc43338c604ae7d0f6aa2419f2415d94eb9cae381601dbba70a61/bindings/venvs/2.17.0']' returned non-zero exit status 1., output:
-----
b"[Errno 17] File exists: '__main__.py' -> '/root/.cache/nce/ab1acf935c4cc43338c604ae7d0f6aa2419f2415d94eb9cae381601dbba70a61/bindings/venvs/2.17.0/pex'\n"
-----

Error: Failed to establish atomic directory /root/.cache/nce/ab1acf935c4cc43338c604ae7d0f6aa2419f2415d94eb9cae381601dbba70a61/locks/install-c76582505cca55d0fdefa5a1b1424a5b0f84fb766dd39aa6b6f89de484fc6c25. Population of work directory failed: Boot binding command failed: exit status: 1

is it possible to share pants caches between users? the pants cache is created in a container execution environment where the user is

circleci

i'm trying to use the pants cache in a machine exection environment (based on the same ubuntu version:

root@fc6a56d12058:~# ls -al .cache/
total 8
drwxr-xr-x 2 root root 4096 Jun  4 15:27 .
drwx------ 1 root root 4096 Jun  4 15:27 ..
lrwxrwxrwx 1 root root   25 Jun  4 15:27 nce -> /home/circleci/.cache/nce
lrwxrwxrwx 1 root root   27 Jun  4 15:27 pants -> /home/circleci/.cache/pants