#19049 Upload large files directly from disk, without loading into memory Issue created by huonw Is your feature request related to a problem? Please describe. With #18153, large files produced are now cached as standalone files, on disk, rather than in the core LMDB structures. Uploading to a remote cache should be able to pipe these directly from that location on disk. Currently they're pulled into memory, written to a temporary file (with an additional bug: it's sync IO in async code), and then that temporary file is uploaded via mmap. Breadcrumbs, in `src/rust/engine/fs/store`: 1.

store::Store::ensure_remote_has_recursive

calls

store::Store::store_large_blob_remote

2.

store_large_blob_remote

calls

store::remote::ByteStore::store_buffered

3.

store_buffered

creates temporary files... 4. ...then calls the closure that

store_large_blob_remote

provided with a

std::fs::File

for that temporary file 5. that closure then reads the whole large-file into memory and splats it into the temporary file (synchronously) Describe the solution you'd like

store::Store::ensure_remote_has_recursive

should pass a file handle into the remote cache, and that's manipulated/uploaded directly, without going through memory. The following code could be adjusted to just decide whether the blob is on disk (if so, go via the disk) or is in LMDB (if so, just pull into memory), resolving the TODO by removing the linking to wire chunk size: pants/src/rust/engine/fs/store/src/lib.rs Lines 831 to 839 in </pantsbuild/pants/commit/4c708a001b19813fdc1fdbcae6ab7d2010208443|4c708a0> Describe alternatives you've considered N/A Additional context Add any other context or screenshots about the feature request here. pantsbuild/pants