What does pants cache in the remote cache for pyth...
# generalh
helpful-salesclerk-86050
07/26/2023, 7:20 PMWhat does pants cache in the remote cache for python repos? Is it just input hash -> test results and any `pex_binary`s we might have created? Is it caching wheels built from requirements? Some kind of intermediate output?
e
enough-analyst-54434
07/26/2023, 7:33 PMUniformly it remotely caches the results of all subprocesses it runs regardless of backend (Python, Java, Scala, Go, etc.). So for Python, for example, but not exhaustively:
+ test: These are run by default as 1 process per test file; so they are remote cached at that granularity
+ package: This operation only has the final PEX binary cached remotely and 0 intermediate results. Those (resolved wheels, etc) are only cached locally in Pants' Pex cache.
When I say remotely caches - that more broadly means caches to disk. You should have a local lmdb cache entry as well.
All the rest - intermediate rule results - are only cached in memory in pantsd.
h
helpful-salesclerk-86050
07/26/2023, 7:35 PMWhat about a pants run call with a python program, any intermediates pulled from the remote cache?
helpful-salesclerk-86050
07/26/2023, 7:36 PMBased on your description then the storage needs for the remote cache with just pants test are pretty small, right? Since it’s not caching the test files just the result and input metadata?
helpful-salesclerk-86050
07/26/2023, 7:37 PMOr is it caching some built artifact of a test target
e
enough-analyst-54434
07/26/2023, 7:39 PMWhat about a pants run call with a python program, any intermediates pulled from the remote cache?I'm not sure. There are several
runBased on your description then the storage needs for the remote cache with just pants test are pretty small, right? Since it’s not caching the test files just the result and input metadata?I think our Python test process outputs include junit.xml by default, so those files IIRC - but still not too big. If you enable coverage, then those files too.
h
helpful-salesclerk-86050
07/26/2023, 7:40 PMOkay thank you. Would be nice to know what caching run might use but not urgent
e
enough-analyst-54434
07/26/2023, 7:41 PMYeah - there is no tracking issue for that that I'm aware of. You really need to have a test rig you can use to answer this for yourself for any given Pants version. Folks have been using a bazel remote cache docker image thing and presumably spinning that up on your machine and running a quick check is the way to go.
enough-analyst-54434
07/26/2023, 7:43 PMSuper ugly because we code-gen actions, but, for example: https://github.com/pantsbuild/pants/blob/989c5d0fcac1af2aa28d2847eb3767c044c19dc0/.github/workflows/test.yaml#L549-L566
The project is here: https://github.com/buchgr/bazel-remote
The image is here: https://hub.docker.com/r/buchgr/bazel-remote-cache/
enough-analyst-54434
07/26/2023, 7:44 PMI have never used this, but you might find searching slack for these new keywords nets you answers / people in the know.
b
better-van-82973
07/26/2023, 7:47 PM\q
h
helpful-salesclerk-86050
07/26/2023, 7:51 PMYeah I have some caches I can use to test, thanks
h
happy-kitchen-89482
07/26/2023, 8:29 PMThe
pants runhappy-kitchen-89482
07/26/2023, 8:29 PM(It uses InteractiveProcess instead of Process)
happy-kitchen-89482
07/26/2023, 8:30 PMThis is because you may expect side-effects or non-idempotent behavior when you
pants runhappy-kitchen-89482
07/26/2023, 8:30 PMAll the processes leading up to the actual run may be cached (dependency resolution, for example), but the actual binary you run is not
h
helpful-salesclerk-86050
07/26/2023, 8:31 PMWould that dependency resolution step upload artifacts to the cache, like wheels?
h
happy-kitchen-89482
07/26/2023, 8:37 PMGenerally yes
happy-kitchen-89482
07/26/2023, 8:38 PMBased on your description then the storage needs for the remote cache with just pants test are pretty small, right? Since it’s not caching the test files just the result and input metadata?This is inaccurate, since many intermediate process inputs and outputs may be cached.
3 Views