Hi team, a question on pex/pants cache. We would like to use cache in our jenkins CI pipeline. Currently, we config

PEX_ROOT

,

PANT_LOCAL_STORE_DIR

and

PANTS_NAMED_CACHES_DIR

in our CI. With those, we can see the pex cache are used within builds across a single branch, i.e, after the first builds, the following builds on that branch will re-use the pex cache. The question is, given we config the

PEX_ROOT

,

PANT_LOCAL_STORE_DIR

and

PANTS_NAMED_CACHES_DIR

to be the same place across all build, won’t the cache be shared across all builds and branches? For example, if one build on a branch resolves the ‘python-default’ from build-support/lockfile.txt, can it be shared?

Yes, the cache is shareable across different Git branches etc. You can convince yourself of that by locally running a test, and then creating a new branch (w/ no changes) and re-run the same test

oh thanks for that. Btw, in CI, I am wondering shall we use pants demon or not?

Are you running

./pants

multiple times? If so, it usually is a good idea to improve performance. All the daemon does is keeps the results from prior runs memoized for subsequent runs

Yes. We do use ./pants multiple times so I guess I will keep the daemon.

What do you mean different places? The only thing that matters is the relative path from your build root (where

./pants

is located), and that file content is the same

For example, in Jenkins, the branch will be put to /tmp/jenkins_tmp/workspace/branch-1, /tmp/jenkins_tmp/workspace/branch-2 and etc.

Got it. That should be fine. @ancient-vegetable-10556 actually just gave a talk about Pycon how this works! tl;dr is that we use relative paths and "snapshot" the files so we have a lightweight way to reference them, e.g. insert the files into a sandbox (tempory directory)

I just did the following experiment locally, Background: 1. a git commit with all configs/files (e.g. pants.toml, BUILD files) required by pants. Experiment 1: 1. check out the commit to /tmp 2. download the pants binary into /tmp 3. run ./pants check :: (it builds all pex cache such mypy.pex/etc and resolves ‘python-default’ dependencies) Experiment 2: 1. check out the same commit to /tmp1 2. download the pants binary into /tmp1 3. run ./pants check :: , it still builds all pex cache such mypy.pex/etc and resolves ‘python-default’ dependencies again. Question: in this case, can experiment2 use the cache from experiment1? or I am doing something wrong?

hmmm it should be reusing the cache - it is global so that it can be shared

yes. if i re-run the same command in tmp/tmp1, the cache is re-used.

are you setting

PANT_LOCAL_STORE_DIR

and

PANTS_NAMED_CACHES_DIR

or using the default?

using the default. I dont set them locally.