https://pantsbuild.org/ logo
#general
Title
# general
f

full-toothbrush-75676

02/04/2022, 2:35 AM
Hi Pants! I have some questions in the pants / pex caching weeds. We’re currently trying to reduce the amount of added CI / CD time incurred by adding or updating 3rdparty dependencies in our monorepo. I’ve been following https://github.com/pantsbuild/pants/issues/14127 and was wondering if there are any workarounds that resemble incremental 3rdparty dependency resolution. Does Pants re-install all 3rdparty dependencies from indexes if it can’t find the full resolve in its cache? I noticed that
pex
also caches
installed_wheels
in
~/.cache/pants/named_caches/pex_root
. Is Pants (through
pex
) able to consult that cache when re-resolving dependencies? In other words, does it still re-install packages from indexes even if the exact package / version is in that
pex
named cache? We currently save / restore the three Pants cache directories in our build pipelines and bust these caches whenever our 3rdparty lockfile changes. I’m wondering if persisting
~/.cache/pants/named_caches/
even in the case of a 3rdparty lockfile change would allow the build to not have to re-download packages from indexes, therefore saving time. Do these questions / this line of thought make sense? Thanks in advance 🙏
w

witty-crayon-22786

02/04/2022, 5:12 PM
Is Pants (through 
pex
) able to consult that cache when re-resolving dependencies? In other words, does it still re-install packages from indexes even if the exact package / version is in that 
pex
 named cache?
yes, PEX uses that cache when re-resolving dependencies
🙌 1
time spent in a re-resolve is mostly time spent in pip re-executing the resolve step, which does lots of network access, and then executing the incremental work
so yes: caching that directory can help.
f

full-toothbrush-75676

02/04/2022, 8:50 PM
Got it! To clarify, we already cache that directory, but we bust is whenever our lockfile changes. I’m wondering if it would make more sense to not bust that cache, and just let it grow over time so that we can utilize it even when the lockfile changes (and then nuke it when it gets too big). Does that make sense?
w

witty-crayon-22786

02/04/2022, 9:11 PM
yea, that would be the current recommendation. although it occurs to me that the precision might not be great: https://github.com/pantsbuild/pants/issues/14364
👍 1
f

full-toothbrush-75676

02/05/2022, 1:14 AM
Awesome, thanks for opening that ticket! I would find some extra details on exactly what to cache + maybe even what cache keys should be used very useful myself 🙏