https://pantsbuild.org/ logo
Join Slack
Powered by
anyone familiar with what this message means in PA...
# general
b
anyone familiar with what this message means in PANTS "Canceled: Remote cache lookup for:" - given I am attempting to use remote caching in my pants.toml?
h
Hi Roy, welcome! With remote caching, Pants uses "speculation", meaning that it will start running the process locally at the same time as doing a cache lookup. Whichever finishes first will win and cancel the other. See https://blog.pantsbuild.org/fast-incremental-builds-speculation-cancellation/
How's it been going setting up remote caching?
b
@hundreds-father-404 - thanks! i guess how do i force pants to just do the remote caching?
instead of locally caching?
our team wants to use more remote caching on our buildfarm worker server -> but i am seeing my local setup winning vs. the remote cache
h
You can use the global option 
--no-process-execution-local-cache
or in 
pants.toml
Copy code
[GLOBAL]
process_execution_local_cache = false
b
ahh genius!
thank you @hundreds-father-404 - I will report back and test and let you know!
❤️ 1
🤞 1
🙏 1
w
to be clear though: speculation will still happen even with the local cache disabled: if it’s faster to run the process locally than to fetch it from the cache, the cache lookup will be cancelled.
👍 1
cache lookups should only be rendered at Debug level though
b
@witty-crayon-22786 - thank you- this is very helpful - i guess there is no way to force the pants build to use remote cache on buildfarm? i guess in most cases local wins?
w
the local process will definitely win sometimes: in particular, some of the processes that we run are very short (import extraction for python completes in <50ms), but hitting the remote cache might only take 300ms, so longer processes should be able to hit.
✅ 1
you can use 
--stats-log
to get an idea of how many hits/misses you got
👍 2
✅ 1
b
thank you so much! I will report back more once I do more testing
Copy code
01:16:37.96 [DEBUG] Completed: Remote cache lookup for: Building <redacted>
01:16:37.96 [DEBUG] remote cache response: digest=Digest { hash: Fingerprint<redacted>, size_bytes: 142 }: None
@witty-crayon-22786 @hundreds-father-404 - does this mean the remote caching worked? Sorry i don't cannot know how to read this - this end of the pants build
before that there was
Copy code
01:16:37.80 [DEBUG] Canceled: Remote cache lookup for: Determine Python imports for <redacted>
h
Yeah, it looks like it was used but a cache miss. Did you use --stats-log? It prints a summary of remote cache performance
b
@hundreds-father-404 - how do you use the --stats-log, do you add it to the pants command when build? can you provide an example of the command- thanks!
h
./pants --stats-log lint ::
b
thank you!
Copy code
local_cache_read_errors: 0
  local_cache_requests: 0
  local_cache_requests_cached: 0
  local_cache_requests_uncached: 0
  local_cache_total_time_saved_ms: 0
  local_cache_write_errors: 0
  local_execution_requests: 20
  local_process_total_time_run_ms: 4122
  remote_cache_read_errors: 0
  remote_cache_requests: 8
  remote_cache_requests_cached: 1
  remote_cache_requests_uncached: 7
  remote_cache_speculation_local_completed_first: 13
  remote_cache_speculation_remote_completed_first: 1
  remote_cache_total_time_saved_ms: 40
  remote_cache_write_attempts: 18
  remote_cache_write_errors: 0
  remote_cache_write_successes: 18
  remote_execution_errors: 0
  remote_execution_requests: 0
  remote_execution_rpc_errors: 0
  remote_execution_rpc_execute: 0
  remote_execution_rpc_retries: 0
  remote_execution_rpc_wait_execution: 0
  remote_execution_success: 0
  remote_execution_timeouts: 0
  remote_process_total_time_run_ms: 0
  remote_store_blob_bytes_downloaded: 407214
  remote_store_blob_bytes_uploaded: 0
  remote_store_missing_digest: 0
what does "remote_cache_speculation_remote_completed_first:" mean?
h
It means that there was a remote cache hit and it was faster to look that up than it was to rerun the process locally
Hmm @witty-crayon-22786 @fast-nail-55400 I realize we should have made this option output JSON, that's way more helpful
f
we can always add a 
--stats-log-format
option
✅ 1
b
@hundreds-father-404 - for this https://pantsbuild.slack.com/archives/C046T6T9U/p1633361372326300?thread_ts=1633117644.313000&amp;cid=C046T6T9U - what is the purpose of "lint ::" are these needed for my command?
h
No, I was giving you an example run. You could do 
./pants --stats-log test project/foo.py
for example. The idea is 
--stats-log
comes at the beginning of the command. See https://www.pantsbuild.org/docs/options#setting-options
b
oh sorry! doh!
doh
h
No worries! I recently learned about the curse of knowledge: https://en.wikipedia.org/wiki/Curse_of_knowledge. Things that are intuitive to us maintainers with Pants are not always intuitive to users, so it's actually extremely helpful to ask when you're confused on things like this! It helps us to realize where docs could be improved etc
b
thank you!
Open in Slack
PreviousNext
Powered by