https://pantsbuild.org/ logo
Join Slack
Powered by
How are people caching pants in CI (for faster exe...
# general
p
How are people caching pants in CI (for faster execution)? We’re getting this error now: `MDB_CORRUPTED: Located page was wrong type`:
Exception: Snapshot failed: Failed to digest inputs: Throw { val: Error storing Digest { hash: Fingerprint<c93b26416b75378875d53b1a3a96569ddcbf9ae40b87c5f4cde555f5c625cc71>, size_bytes: 217 }: MDB_CORRUPTED: Located page was wrong type, python_traceback: “Traceback (no traceback):\n <pants native internals>\nException: Error storing Digest { hash: Fingerprint<c93b26416b75378875d53b1a3a96569ddcbf9ae40b87c5f4cde555f5c625cc71>, size_bytes: 217 }: MDB_CORRUPTED: Located page was wrong type”, engine_traceback: [“digest_file”] }
Looking around seems like a few people here this as well in CI. The workarounds seem to be either using 
--no-pantsd
(didn’t work) or blowing up the cache dir (not a feasible solution for us).
For context we are running in GitLab CI/CD (but this could be anywhere) in Ubuntu CI executors. We’re storing the cache on each executor. A lot of the times (but not always 🤔) CI fails with that error. What seems to be happening is when one CI job generates the cache but then another job (with a different reqs and lock files) tries to reuse it.
For more context, we don’t see this error when using S3 as the cache mechanism. But our cache is in GBs and up-/down-loading the cache takes too long.
After hitting my head my against the wall (ouch) for a few days, I’m looking into REAPI now: https://www.pantsbuild.org/docs/remote-caching-execution Is this the recommended way forward? I’m highly interested in REAPI’s ability to just download what a CI needs.
e
It probably is, yes. The Pants repo itself uses this in its CI. In that case the backend is provided by Toolchain.
👍 1
h
https://www.toolchain.com/ for a Pants-friendly implementation
p
Thanks. Currently leaning towards “build” on the build-vs-buy spectrum. Any self-hosted suggestions? Wondering if bazel the way to go here.
b
The page you found above has a few options listed like BuildBarn
p
I went ahead and tried the docker option in https://github.com/buchgr/bazel-remote it works extremely well, reduced our builds by an order of magnitude.
Happy to jump on a call to share the details.
And I also have a follow up Q: it seems like an REAPI server can only be used to cache 
PANTS_NAMED_CACHES_DIR
. Is this right? using S3 to cache 
PANTS_SETUP_CACHE
can’t be cached since pants needs to actually install itself and 
PANTS_LOCAL_STORE_DIR
are test results which don’t need to be cached?
h
You're right that pants setup can't be cached, for bootstrapping reasons, but what is cached over REAPI is the contents of the local_store_dir.
The named caches are specifically a local optimization
p
Ah 🤔 so no way to cache pip dependencies so we don’t have to reinstall them on each run?
Also I’ve been unsuccessful in checking locally how do I know when things are is coming from cache vs being generated
t
I've just started trying to setup caching on gitlab. Did you have any success. Any recommended guidelines?
p
Yes — use remove caching. 
pants.toml
Copy code
remote_cache_read = true
remote_cache_write = true
remote_store_address = "<grpc://bazel-remote-cache.sys.enigma.com:80>"
remote_instance_name = "main"
<http://bazel-remote-cache.sys.enigma.com|bazel-remote-cache.sys.enigma.com>
is something we set up. (I was about to recommend Toolchain which was a managed build/ remote caching service for pants but it no longer exists.) wrt local caching — it happens automatically and pants handles of it without any additional config. But from my testing local caching can’t be shared between CI workers — we were getting weird errors in CI that we think are due to shared cache
t
Thanks for sharing, sorry about the delayed response, I was traveling.
👌 1
8 Views
Open in Slack
PreviousNext
Powered by