Ergonomic, hermetic, user-friendly software build system for Python, Go, Java, Kotlin, Protobuf, Scala, Shell. Pants lets you fearlessly scale up your codebase!

Pants

well, i was thinking maybe not even "pathological" because it doesn't seem like this is necessarily an intrinsically easy problem to solve

but definitely pathological in the sense that yeah there's something very specific there that's causing this slowdown

thinking about using a `ShardedLmdb` to store a trie of all of `PathStat` s that belong to each `Digest`

oh em gee wait it should do structural sharing somehow

the merkle tree is already structure shared

the query of `Digest` =&gt; `Vec&lt;PathStat&gt;` is currently slow for large snapshots. makes sense

i think i was getting to what you're about to say maybe

but there are a lot of other codepaths that use `Store::walk` , and we haven’t confirmed that any of the others are this slow

as in, i think that codepath can absolutely be optimized

yea, i mean something else. i’m saying that it may not be `Digest -&gt; Vec&lt;PathStat&gt;` that is slow. but i am not sure.

~i don't think `Store::walk()` is likely to be bad~ nvm

i'm probably gonna try drawing some things on paper

i’d recommend creating a new benchmark for snapshotsubset here: <https://github.com/pantsbuild/pants/blob/faf5e85d288102cee62a598238568661131e770c/src/rust/engine/fs/store/benches/store.rs#L56-L69>

then you can run just your new benchmark with eg
```./build-support/bin/native/cargo bench --manifest-path=src/rust/engine/Cargo.toml --package store -- $mybenchname```

does that have profiling? or is that a command line i can give to instruments?

i am aware there is benchmarking but not sure if profiling is included

i’d launch it, and then attach instruments once it has started

if you fiddle with the `measurement_time`, you can get it to basically go in a loop for a while.

and i'm rebased to today's master so no sweat about merge conflicts this code is very separate

ok, cool. i can see how i can slot in a snapshot subset benchmark for the large henries snapshot here

~will probably extract the materialize_directory() call to a private `fn`~

took me a second to understand `let path_buf = clean_line.split_whitespace().collect::&lt;PathBuf&gt;();`

i have `MODE=debug` in my environment....lol. gonna make the benchmark still but that would probably affect performance

i can definitely repro the issue without `MODE=debug` in a criterion benchmark

the filesystem traces don't work though i'm gonna google that

the time profiler very clearly points `ingest_directory_from_sorted_path_stats` taking up most of the time and being incredibly recursive

which is what the comment on the method says as well

Screen Shot 2020-05-11 at 18.08.51.png

it looks like `merge_directories_recursive()` may also be the issue or another issue

and it appears to be used in the snapshot subsetting operation

`get_snapshot_subset() =&gt; from_path_stats() =&gt; ingest_directory_from_sorted_path_stats()`, which is recursive

and there are many different threads of execution that took e.g. 800ms

i think this makes sense if there's massive contention at some point, either on lmdb or on a lock somewhere

but the join_alls don't necessarily seem to end

Screen Shot 2020-05-11 at 18.27.17.png

373 ms on cloning a hashbrown table, it seems

we might be able to remove the hashmap here and work with slices

i'm beginning to think more about another `ShardedLmdb` to cache `Digest =&gt; Vec&lt;FileContent&gt;` instead of recomputing it each time in `Store::contents_for_directory()` (to improve `SnapshotSubset` time). playing around with it now

so, from what i can tell you have only selected that last portion of the profile

headed to dinner, but: looking at only one portion of of the time will lie a bit

recommend clicking the button at the top right of the “Heaviest Stack Trace” part

as an example: `materialize_directory` , which does the equivalent of `walk` to read from the database, _and_ writes a bunch of stuff to disk… only takes 6 seconds for 10k files.