https://pantsbuild.org/ logo
Join Slack
Powered by
How can I debug why a large `package` goal isn't a...
# general
a
How can I debug why a large 
package
goal isn't actually storing the result in the remote cache? It publishes something for CAS, but not for AC. There's nothing that is helpful in even the -ldebug logs...
f
What about debug logs with Rust logging included?
Also are any of the writes failing? Dump 
—stats-log
and it should show error counts.
a
We're using 
bazel-remote
, and there are some writes at the end failing, I thought they're because we stop it before it's finished writing, but added a sleep 60 at the end of the whole thing, we still got (and this is logs from pants and 
bazel-remote
together, but it's easy to see which is which):
Copy code
20:28:58.04 [DEBUG] computed 1 nodes in 1287.068876 seconds. there are 433742 total nodes.
20:28:58.04 [DEBUG] waiting for 1 session end task(s) to complete
2025/01/27 20:28:59 S3 CONTAINS ci-pants-remote-cache cas.v2/f5/f51d2703cc52c1ad526fa2f592e3f9009c903486ce4449281d5e607821b806b7 NOT FOUND
2025/01/27 20:29:00 S3 CONTAINS ci-pants-remote-cache cas.v2/f9/f9943d6f9cb5b49471e5d27dc0da7c18288595d441e022270898606418d458f3 NOT FOUND
20:29:01.04 [DEBUG] 1 session end task(s) failed to complete within timeout: remote cache write Digest { hash: Fingerprint<8db245cf53e00fd0453f62d09ca6274add8b2ef77ac3ec8b20eed886c458862a>, size_bytes: 142 }
2025/01/27 20:29:01 GRPC BYTESTREAM WRITE FAILED: main/uploads/72a51a29-bc29-4960-a27d-425e769fbe23/blobs/f51d2703cc52c1ad526fa2f592e3f9009c903486ce4449281d5e607821b806b7/731725779 rpc error: code = Internal desc = rpc error: code = Canceled desc = context canceled
2025/01/27 20:29:01 GRPC BYTESTREAM WRITE FAILED: main/uploads/e8b7308a-3a01-48c3-8f33-cd3962f77b02/blobs/f9943d6f9cb5b49471e5d27dc0da7c18288595d441e022270898606418d458f3/755761508 rpc error: code = Internal desc = rpc error: code = Canceled desc = context canceled
I'm not sure what 
-stats-log
is, let me read about that, I'll do another run
Okay, so added 
PANTS_LOG_SHOW_RUST_3RDPARTY=True
and 
PANTS_STATS_LOG=True
, increased the sleep after the job was done. I got distracted was gonna try the sleep increase earlier
I'll have to take a look at these tomorrow, it's getting too late here
f
Copy code
20:29:01.04 [DEBUG] 1 session end task(s) failed to complete within timeout: remote cache write Digest { hash:
That's likely the issue. The cache write is not completing in time at the end of the run.
a
That was it, thank you!
5 Views
Open in Slack
PreviousNext
Powered by