Re big blobs in the engine - it looks like when re...
# developmente
enough-analyst-54434
09/29/2022, 11:59 PMRe big blobs in the engine - it looks like when reading from the remote cache we buffer full blobs in memory? This would be not good for things like pytorch 2GB wheels if so: https://github.com/pantsbuild/pants/blob/d8fba9afbe854ccd5a293820b665aee867818001/src/rust/engine/fs/store/src/remote.rs#L383-L402
enough-analyst-54434
09/30/2022, 12:02 AMI can file a gedanken-issue if someone can check my reading there.
w
witty-crayon-22786
09/30/2022, 12:04 AMyea, true. that’s in the general theme of “we should mmap for both reads and writes”
witty-crayon-22786
09/30/2022, 12:04 AMwhatever replaces LMDB would hopefully allow for async streaming of writes without holding a lock the whole time.
witty-crayon-22786
09/30/2022, 12:05 AMbut we could probably also do a “stream appends to a temporary LMDB key then move to another one inside of a transaction” strategy in the meantime
e
enough-analyst-54434
09/30/2022, 12:23 AMI stuck to the issue and offered no solutions: https://github.com/pantsbuild/pants/issues/17065
👍 1