https://pantsbuild.org/ logo
#development
Title
# development
e

enough-analyst-54434

09/29/2022, 11:59 PM
Re big blobs in the engine - it looks like when reading from the remote cache we buffer full blobs in memory? This would be not good for things like pytorch 2GB wheels if so: https://github.com/pantsbuild/pants/blob/d8fba9afbe854ccd5a293820b665aee867818001/src/rust/engine/fs/store/src/remote.rs#L383-L402
I can file a gedanken-issue if someone can check my reading there.
w

witty-crayon-22786

09/30/2022, 12:04 AM
yea, true. that’s in the general theme of “we should mmap for both reads and writes”
whatever replaces LMDB would hopefully allow for async streaming of writes without holding a lock the whole time.
but we could probably also do a “stream appends to a temporary LMDB key then move to another one inside of a transaction” strategy in the meantime
e

enough-analyst-54434

09/30/2022, 12:23 AM
I stuck to the issue and offered no solutions: https://github.com/pantsbuild/pants/issues/17065
👍 1