Hello everyone. I am trying to better understand how pants utilizes its dependency cache. I am running pants 1.30.4 (hopefully for not much longer as 2.x upgrade should be very soon). When i look in our cached dirs i see things like this. Shown below are two diff UUIDs that have the exact same requirement downloaded into the cache. I am trying to solve for an issue we are having where our cache is massive. We run ~25 docker containers that all share a volume mounted cache with the host. Each container is a “pants base image” that has an entrypoint to a pants run command. Any insight as to why this may be happening or what we can do to optimize would be helpful. It almost looks as if each docker container has its own cache for dependencies regardless if it is running something that should be already cached.

48K	.pants.d/pyprep/requirements/CPython-3.6.8/4b1ce979d2bd82f3e9da3c387f429ab17df5e00d/.deps/fastapi-0.63.0-py3-none-any.whl/fastapi-0.63.0.dist-info
48K	.pants.d/pyprep/requirements/CPython-3.6.8/4b1ce979d2bd82f3e9da3c387f429ab17df5e00d/.deps/fastapi-0.63.0-py3-none-any.whl/fastapi/openapi/__pycache__
92K	.pants.d/pyprep/requirements/CPython-3.6.8/4b1ce979d2bd82f3e9da3c387f429ab17df5e00d/.deps/fastapi-0.63.0-py3-none-any.whl/fastapi/openapi
28K	.pants.d/pyprep/requirements/CPython-3.6.8/4b1ce979d2bd82f3e9da3c387f429ab17df5e00d/.deps/fastapi-0.63.0-py3-none-any.whl/fastapi/middleware
32K	.pants.d/pyprep/requirements/CPython-3.6.8/4b1ce979d2bd82f3e9da3c387f429ab17df5e00d/.deps/fastapi-0.63.0-py3-none-any.whl/fastapi/dependencies/__pycache__
72K	.pants.d/pyprep/requirements/CPython-3.6.8/4b1ce979d2bd82f3e9da3c387f429ab17df5e00d/.deps/fastapi-0.63.0-py3-none-any.whl/fastapi/dependencies
116K	.pants.d/pyprep/requirements/CPython-3.6.8/4b1ce979d2bd82f3e9da3c387f429ab17df5e00d/.deps/fastapi-0.63.0-py3-none-any.whl/fastapi/__pycache__
40K	.pants.d/pyprep/requirements/CPython-3.6.8/4b1ce979d2bd82f3e9da3c387f429ab17df5e00d/.deps/fastapi-0.63.0-py3-none-any.whl/fastapi/security/__pycache__
80K	.pants.d/pyprep/requirements/CPython-3.6.8/4b1ce979d2bd82f3e9da3c387f429ab17df5e00d/.deps/fastapi-0.63.0-py3-none-any.whl/fastapi/security
564K	.pants.d/pyprep/requirements/CPython-3.6.8/4b1ce979d2bd82f3e9da3c387f429ab17df5e00d/.deps/fastapi-0.63.0-py3-none-any.whl/fastapi
616K	.pants.d/pyprep/requirements/CPython-3.6.8/4b1ce979d2bd82f3e9da3c387f429ab17df5e00d/.deps/fastapi-0.63.0-py3-none-any.whl
48K	.pants.d/pyprep/requirements/CPython-3.6.8/f9da1695c3ec3dc06d31877bc4eb4990d89ec75d/.deps/fastapi-0.63.0-py3-none-any.whl/fastapi-0.63.0.dist-info
48K	.pants.d/pyprep/requirements/CPython-3.6.8/f9da1695c3ec3dc06d31877bc4eb4990d89ec75d/.deps/fastapi-0.63.0-py3-none-any.whl/fastapi/openapi/__pycache__
92K	.pants.d/pyprep/requirements/CPython-3.6.8/f9da1695c3ec3dc06d31877bc4eb4990d89ec75d/.deps/fastapi-0.63.0-py3-none-any.whl/fastapi/openapi
28K	.pants.d/pyprep/requirements/CPython-3.6.8/f9da1695c3ec3dc06d31877bc4eb4990d89ec75d/.deps/fastapi-0.63.0-py3-none-any.whl/fastapi/middleware
32K	.pants.d/pyprep/requirements/CPython-3.6.8/f9da1695c3ec3dc06d31877bc4eb4990d89ec75d/.deps/fastapi-0.63.0-py3-none-any.whl/fastapi/dependencies/__pycache__
72K	.pants.d/pyprep/requirements/CPython-3.6.8/f9da1695c3ec3dc06d31877bc4eb4990d89ec75d/.deps/fastapi-0.63.0-py3-none-any.whl/fastapi/dependencies
116K	.pants.d/pyprep/requirements/CPython-3.6.8/f9da1695c3ec3dc06d31877bc4eb4990d89ec75d/.deps/fastapi-0.63.0-py3-none-any.whl/fastapi/__pycache__
40K	.pants.d/pyprep/requirements/CPython-3.6.8/f9da1695c3ec3dc06d31877bc4eb4990d89ec75d/.deps/fastapi-0.63.0-py3-none-any.whl/fastapi/security/__pycache__
80K	.pants.d/pyprep/requirements/CPython-3.6.8/f9da1695c3ec3dc06d31877bc4eb4990d89ec75d/.deps/fastapi-0.63.0-py3-none-any.whl/fastapi/security
564K	.pants.d/pyprep/requirements/CPython-3.6.8/f9da1695c3ec3dc06d31877bc4eb4990d89ec75d/.deps/fastapi-0.63.0-py3-none-any.whl/fastapi
616K	.pants.d/pyprep/requirements/CPython-3.6.8/f9da1695c3ec3dc06d31877bc4eb4990d89ec75d/.deps/fastapi-0.63.0-py3-none-any.whl

Hi Brian, sorry you're having issues with this. 2.x uses completely different fingerprinting and caching mechanisms ,so hopefully an upgrade should solve this. Or at least if something like it persists, debugging it will be different and easier.

correct we need to be on the v1 engine for now

Debugging this would require quite a bit of deep diving into old v1 code, so I guess my lazy question would be how urgent is this given your upcoming upgrade? 😉

we are just going to do a workaround. We are close to deprecating the final piece of legacy code that will open the path to v2. We will circle back… If this is the same issue on v2 we will have to dig deeper.