https://pantsbuild.org/ logo
Join Slack
Powered by
this is my thread for this afternoon
# development
a
this is my thread for this afternoon
here
i'm probably gonna try drawing some things on paper
w
would profile it probably
a
yeah
xcode installed finally
on monday i think
now i have instruments
šŸ‘ 1
w
a
omfg
i love criterion
w
then you can run just your new benchmark with eg
Copy code
./build-support/bin/native/cargo bench --manifest-path=src/rust/engine/Cargo.toml --package store -- $mybenchname
a
does that have profiling? or is that a command line i can give to instruments?
i am aware there is benchmarking but not sure if profiling is included
w
iā€™d launch it, and then attach instruments once it has started
šŸ‘ 1
if you fiddle with the 
measurement_time
, you can get it to basically go in a loop for a while.
and then attach.
a
this is great!
and i'm rebased to today's master so no sweat about merge conflicts this code is very separate
ok, cool. i can see how i can slot in a snapshot subset benchmark for the large henries snapshot here
will probably extract the materialize_directory() call to a private 
fn
took me a second to understand 
let path_buf = clean_line.split_whitespace().collect::<PathBuf>();
gonna leave a comment
i have 
MODE=debug
in my environment....lol. gonna make the benchmark still but that would probably affect performance
i can definitely repro the issue without 
MODE=debug
in a criterion benchmark
instruments is so nice to use
the filesystem traces don't work though i'm gonna google that
w
i mostly use Time Profiler
havenā€™t had much luck with the rest.
a
yep that is what i am settling on
thanks
the time profiler very clearly points 
ingest_directory_from_sorted_path_stats
taking up most of the time and being incredibly recursive
which is what the comment on the method says as well
Screen Shot 2020-05-11 at 18.08.51.png
which aligns with our earlier hunches
so to clarify a bit more
it looks like 
merge_directories_recursive()
may also be the issue or another issue
in that it is recursive
and it appears to be used in the snapshot subsetting operation
oh no
i was right first
get_snapshot_subset() => from_path_stats() => ingest_directory_from_sorted_path_stats()
, which is recursive
and there are many different threads of execution that took e.g. 800ms
i think this makes sense if there's massive contention at some point, either on lmdb or on a lock somewhere
but the join_alls don't necessarily seem to end
Screen Shot 2020-05-11 at 18.27.17.png
373 ms on cloning a hashbrown table, it seems
we might be able to remove the hashmap here and work with slices
i'm beginning to think more about another 
ShardedLmdb
to cache 
Digest => Vec<FileContent>
instead of recomputing it each time in 
Store::contents_for_directory()
(to improve 
SnapshotSubset
time). playing around with it now
w
so, from what i can tell you have only selected that last portion of the profile
what was happening during the rest?
a
great question
w
headed to dinner, but: looking at only one portion of of the time will lie a bit
a
i strongly agree
thank you!
w
recommend clicking the button at the top right of the ā€œHeaviest Stack Traceā€ part
šŸ‘ 1
Open in Slack
PreviousNext
Powered by