#20674 remote cache ResourceExhausted/OutOfRange errors Issue created by jasonwbarnett Describe the bug When remote caching is enabled and you use

adhoc_tool

that outputs a directory with a ton of files (i.e. 180K), pants attempts to call

FindMissingBlobs

(

/build.bazel.remote.execution.v2.ContentAddressableStorage/FindMissingBlobs

) and the message exceeds 4MB which leads to a

ResourceExhausted

or

OutOfRange

error depending on the configuration of the remote caching service. In my case I am using bazel-remote. I started by modifying bazel-remote's MaxRecvMsgSize which revealed a different error (

OutOfRange

). Before increasing MaxRecvMsgSize in bazel-remote

[WARN] Failed to write to remote cache (1 occurrences so far): ResourceExhausted: "grpc: received message larger than max (5790951 vs. 4194304)"

This error message is coming from bazel-remote. After increasing MaxRecvMsgSize in bazel-remote

[WARN] Failed to write to remote cache (1 occurrences so far): OutOfRange: "Error, message length too large: found 5790600 bytes, the limit is: 4194304 bytes"

This error is coming from tonic. I validated this by modifying tonic and updating pants to use a branch on my tonic fork. I ended up submitting this PR: hyperium/tonic#1658. I took this one step further. I updated tonic's

DEFAULT_MAX_RECV_MESSAGE_SIZE

(jwb/set-max-message-size-very-high) and validated that everything functions correctly after I do that. Which to be clear, both the client (i.e. pants via tonic) and the server (i.e. bazel-remote) needed to be modified to support the large

FindMissingBlobs

message. I see at least two possible solutions: 1. Add a configuration option to pants which controls tonic's max_decoding_message_size. To be clear, I'm not sure this there is a relationship between max_decoding_message_size and

DEFAULT_MAX_RECV_MESSAGE_SIZE

, but I'm hoping so 🤞 2. Update pants so that chunks or streams the FindMissingBlobs calls to the server. Pants version 2.19.0 OS Both Linux and Mac OS. pantsbuild/pants