#development
Title
b
bitter-ability-32190
06/22/2022, 2:24 PMI'm musing a challenge we have (might be one of our last ones for migration!)
One of our services is packaged using files on disk on dev boxes which are gitignored (a command generates them). Trying to
files()finding all targetsdirectoryThis might actually work!
It'd be a
FileSourceFielddirectorylazy_fileslazy_resourcesI wonder how long it will take to fingerprint 100k files š¤®
w
witty-crayon-22786
06/22/2022, 4:17 PMfilesystems suck, heh. you might also see whether the files can be zipped/tarād and then extracted
b
bitter-ability-32190
06/22/2022, 4:18 PMYeah thats my backup
Error launching process: Os { code: 7, kind: ArgumentListTooLong, message: "Argument list too long" }It's only 100k files, cmon!
w
witty-crayon-22786
06/22/2022, 4:28 PMheh. iirc, zip allows input files from stdin
Copy code
-@ file lists. If a file list is specified as -@ [Not on MacOS], zip takes the list of input files from standard input instead of from the command line. For example,
zip -@ foob
bitter-ability-32190
06/22/2022, 4:29 PMDo we use that tho?
w
witty-crayon-22786
06/22/2022, 4:29 PMoh. sorry, i thought that you were writing a
@ruleb
bitter-ability-32190
06/22/2022, 4:30 PMI'd love to provide more, but my terminal buffer is just a list of the files š then maybe 20 lines of the bottom of the error JSON š
I'm almost certain this is not the path I want to go down š
Copy code
12:01:53.49 [WARN] Error storing process execution result to local cache: Error storing fingerprints ["000102030405060708090a0b0c0d0e0f101112131415161718191a1b1c1d1e1f20", "000102030405060708090a0b0c0d0e0f101112131415161718191a1b1c1d1e1f20", "000102030405060708090a0b0c0d0e0f101112131415161718191a1b1c1d1e1f20", "000102030405060708090a0b0c0d0e0f101112131415161718191a1b1c1d1e1f20", "000102030405060708090a0b0c0d0e0f101112131415161718191a1b1c1d1e1f20", "000102030405060708090a0b0c0d0e0f101112131415161718191a1b1c1d1e1f20"]: MDB_MAP_FULL: Environment mapsize limit reached - ignoring and continuingw
witty-crayon-22786
06/22/2022, 5:04 PMā¦ i think that that means that you hit https://www.pantsbuild.org/docs/reference-global#section-local-store-directories-max-size-bytes ā¦?
16GB worth of directory entries
b
bitter-ability-32190
06/22/2022, 5:05 PMYeah, I think archiving is the path forward for us šµ
w
witty-crayon-22786
06/22/2022, 5:05 PMwhich you can increase, but ā¦
b
bitter-ability-32190
06/22/2022, 5:12 PMWhat slice of my work would make this occur?
Right now I have a custom target with a codegen rule, in the rule I
PathGlobs("bigfatdir/*")As a zip it'd be 900-ish MB
w
witty-crayon-22786
06/22/2022, 5:14 PMpantsddirectories go in
~/.cache/pants/lmdb_store/directorieson my machine (for the repositories that i build), the ratio between files and directories is 14G to 38M
the limit is fairly arbitrary.
but either: 1)
pantsdis there anything odd about how you are using
pantsdand what is the output of
find bigfatdir | wcb
bitter-ability-32190
06/22/2022, 5:20 PM101262 <tel:1012624759269|101262 4759269>Nothing pantsd odd
w
witty-crayon-22786
06/22/2022, 5:22 PMand i assume that
du -h ~/.cache/pants/lmdb_store/directoriesyea, if the filenames are collectively only 4.7MB itās pretty unlikely that that would cause you to hit that limit
very weirdā¦
b
bitter-ability-32190
06/22/2022, 5:30 PM`987M /home/joshuacannon/.cache/pants/lmdb_store/directories```
w
witty-crayon-22786
06/22/2022, 5:32 PMok, then: super-extra weird. i think that that MAP_FULL error is a lieā¦ some other error masquerading as āmap fullā.
do you think that you could adjust that error message to also include the value length? https://github.com/pantsbuild/pants/blob/6564898b02016af70e2c50215955a40a2d7f62a7/src/rust/engine/sharded_lmdb/src/lib.rs#L424-L431
something like
Copy code
diff --git a/src/rust/engine/sharded_lmdb/src/lib.rs b/src/rust/engine/sharded_lmdb/src/lib.rs
index 2a925c33fc..9145d6382d 100644
--- a/src/rust/engine/sharded_lmdb/src/lib.rs
+++ b/src/rust/engine/sharded_lmdb/src/lib.rs
@@ -425,7 +425,7 @@ impl ShardedLmdb {
"Error storing fingerprints {:?}: {}",
batch
.iter()
- .map(|(key, _)| key.to_hex())
+ .map(|(key, value)| format!("{} len: {}", key.to_hex(), value.len()))
.collect::<Vec<_>>(),
e
)b
bitter-ability-32190
06/22/2022, 5:36 PMAh, can't use
main34G /home/joshuacannon/.cache/pants/lmdb_store/filesI can wipe the cache and try this one operation and see what comes out the other end
w
witty-crayon-22786
06/22/2022, 5:40 PMthat should be fine. the relevant method only stores directories, so iām pretty sure that this is directory entry related.
but yea, feel free. i expect that the error is lying, and that it has to do with the item size. so wiping is unlikely to affect it
b
bitter-ability-32190
06/22/2022, 5:54 PMoh yeah I suppose that makes sense. Wont wipe then
w
witty-crayon-22786
06/22/2022, 6:01 PManother shot in the dark: did you adjust https://www.pantsbuild.org/docs/reference-global#section-local-store-shard-count ?
b
bitter-ability-32190
06/22/2022, 6:12 PM, no but I can for testing
w
witty-crayon-22786
06/22/2022, 6:12 PMnah, donāt worry about it. the default should mean that the maximum size of a directory is 1GB
b
bitter-ability-32190
06/22/2022, 6:13 PMHmmm maybe:
Copy code
954M /home/joshuacannon/.cache/pants/lmdb_store/directories/5w
witty-crayon-22786
06/22/2022, 6:13 PMshouldnāt be: max size for a single directory
b
bitter-ability-32190
06/22/2022, 6:13 PMWel if it was going to write more than 50-ish MB what woul happen?
w
witty-crayon-22786
06/22/2022, 6:14 PMooooh, i see what youāre saying.
Copy code
/home/joshuacannon/.cache/pants/lmdb_store/directories/5b
bitter-ability-32190
06/22/2022, 6:14 PMcan I wipe that shard and try again? Is that kosher?
w
witty-crayon-22786
06/22/2022, 6:15 PMwhat are all of the sizes in there?
b
bitter-ability-32190
06/22/2022, 6:15 PMMostly 2ish MB except that one
w
witty-crayon-22786
06/22/2022, 6:16 PMwowā¦ ok. so yea, this was a single huge directory entry then probably.
b
bitter-ability-32190
06/22/2022, 6:16 PMI think the dir is like 1.5GB in sizeš
w
witty-crayon-22786
06/22/2022, 6:16 PMumā¦ it might be fine to wipe only one? probably safer to move it aside
@bitter-ability-32190: the directory entries only contain file names and digestsā¦ (not even the full path: just the name)
b
bitter-ability-32190
06/22/2022, 6:17 PMah
moving the shard an re-running worked
Now that shard has 12M
w
witty-crayon-22786
06/22/2022, 6:23 PMok. more points against LMDB. these limits can be adjusted, but itās a pain in the ass both that we have to shard, and that we have to set a fixed max size. bumped an aggregator ticket on the topic.
if you want to pursue that path, you can adjust the size limit and/or the shard count. iām still curious why that directory entry is so large, so if you do end up getting a chance to add debug output, it would be appreciated.
b
bitter-ability-32190
06/22/2022, 6:24 PMAt a minimum we have 101k filename+digest combos š¤·āāļø
w
witty-crayon-22786
06/22/2022, 6:24 PMprobably easier though is to see if you can find the single largest directory (in terms of direct/non-recursive children)
b
bitter-ability-32190
06/22/2022, 7:19 PMWe're just gonna zip up the relevant files. It's much saner that way š
Well I've learned Pants doesn't like >100k targets š
w
witty-crayon-22786
06/22/2022, 9:31 PMYet ā¢ļø
ā¤ļø 1