bitter-ability-32190
06/22/2022, 2:24 PMfiles()
them causes Pants to treat them as individual targets and then finding all targets
takes forever for those who have the files locally.
There's maybe a few ways we can handle this, but my first thought is "what if we had a target type which represented the files as one target?" Like a directory
target?FileSourceField
generator (or resource. damn you distinction!)
Honestly it doesn't even have to a directory
, it just as easily could be a lazy_files
or lazy_resources
witty-crayon-22786
06/22/2022, 4:17 PMbitter-ability-32190
06/22/2022, 4:18 PMError launching process: Os { code: 7, kind: ArgumentListTooLong, message: "Argument list too long" }
lolwitty-crayon-22786
06/22/2022, 4:28 PM-@ file lists. If a file list is specified as -@ [Not on MacOS], zip takes the list of input files from standard input instead of from the command line. For example,
zip -@ foo
bitter-ability-32190
06/22/2022, 4:29 PMwitty-crayon-22786
06/22/2022, 4:29 PM@rule
. need more error contextbitter-ability-32190
06/22/2022, 4:30 PM12:01:53.49 [WARN] Error storing process execution result to local cache: Error storing fingerprints ["000102030405060708090a0b0c0d0e0f101112131415161718191a1b1c1d1e1f20", "000102030405060708090a0b0c0d0e0f101112131415161718191a1b1c1d1e1f20", "000102030405060708090a0b0c0d0e0f101112131415161718191a1b1c1d1e1f20", "000102030405060708090a0b0c0d0e0f101112131415161718191a1b1c1d1e1f20", "000102030405060708090a0b0c0d0e0f101112131415161718191a1b1c1d1e1f20", "000102030405060708090a0b0c0d0e0f101112131415161718191a1b1c1d1e1f20"]: MDB_MAP_FULL: Environment mapsize limit reached - ignoring and continuing
lolwitty-crayon-22786
06/22/2022, 5:04 PMbitter-ability-32190
06/22/2022, 5:05 PMwitty-crayon-22786
06/22/2022, 5:05 PMbitter-ability-32190
06/22/2022, 5:12 PMPathGlobs("bigfatdir/*")
into a snaptshot.
I think the dir is like 1.5GB in sizewitty-crayon-22786
06/22/2022, 5:14 PMpantsd
periodically garbage collects the store to attempt to keep the content at a fraction of the configured max size (16GB for directory entries, 256GB for files)~/.cache/pants/lmdb_store/directories
pantsd
hasnāt getting a chance to garbage collect recently, 2) you captured some very, very large directories such that you bursted all the way to the limit.pantsd
?find bigfatdir | wc
?bitter-ability-32190
06/22/2022, 5:20 PM101262 <tel:1012624759269|101262 4759269>
witty-crayon-22786
06/22/2022, 5:22 PMdu -h ~/.cache/pants/lmdb_store/directories
is roughly 16G ?bitter-ability-32190
06/22/2022, 5:30 PMwitty-crayon-22786
06/22/2022, 5:32 PMdiff --git a/src/rust/engine/sharded_lmdb/src/lib.rs b/src/rust/engine/sharded_lmdb/src/lib.rs
index 2a925c33fc..9145d6382d 100644
--- a/src/rust/engine/sharded_lmdb/src/lib.rs
+++ b/src/rust/engine/sharded_lmdb/src/lib.rs
@@ -425,7 +425,7 @@ impl ShardedLmdb {
"Error storing fingerprints {:?}: {}",
batch
.iter()
- .map(|(key, _)| key.to_hex())
+ .map(|(key, value)| format!("{} len: {}", key.to_hex(), value.len()))
.collect::<Vec<_>>(),
e
)
bitter-ability-32190
06/22/2022, 5:36 PMmain
, as then I'd have to upgrade my plugins and that aint easy š
I'm behind on my version upgrades34G /home/joshuacannon/.cache/pants/lmdb_store/files
hmmmwitty-crayon-22786
06/22/2022, 5:40 PMbitter-ability-32190
06/22/2022, 5:54 PMwitty-crayon-22786
06/22/2022, 6:01 PMbitter-ability-32190
06/22/2022, 6:12 PMwitty-crayon-22786
06/22/2022, 6:12 PMbitter-ability-32190
06/22/2022, 6:13 PM954M /home/joshuacannon/.cache/pants/lmdb_store/directories/5
Is a problem then?witty-crayon-22786
06/22/2022, 6:13 PMbitter-ability-32190
06/22/2022, 6:13 PMwitty-crayon-22786
06/22/2022, 6:14 PM/home/joshuacannon/.cache/pants/lmdb_store/directories/5
is only one shard⦠missed that abovebitter-ability-32190
06/22/2022, 6:14 PMwitty-crayon-22786
06/22/2022, 6:15 PMbitter-ability-32190
06/22/2022, 6:15 PMwitty-crayon-22786
06/22/2022, 6:16 PMbitter-ability-32190
06/22/2022, 6:16 PMI think the dir is like 1.5GB in sizeš
witty-crayon-22786
06/22/2022, 6:16 PMbitter-ability-32190
06/22/2022, 6:17 PMwitty-crayon-22786
06/22/2022, 6:23 PMbitter-ability-32190
06/22/2022, 6:24 PMwitty-crayon-22786
06/22/2022, 6:24 PMbitter-ability-32190
06/22/2022, 7:19 PMwitty-crayon-22786
06/22/2022, 9:31 PM