https://pantsbuild.org/ logo
#development
Title
# development
a

aloof-angle-91616

05/11/2020, 11:49 PM
this is my thread for this afternoon
here
i'm probably gonna try drawing some things on paper
w

witty-crayon-22786

05/11/2020, 11:55 PM
would profile it probably
a

aloof-angle-91616

05/11/2020, 11:55 PM
yeah
xcode installed finally
on monday i think
now i have instruments
👍 1
w

witty-crayon-22786

05/11/2020, 11:57 PM
a

aloof-angle-91616

05/11/2020, 11:57 PM
omfg
i love criterion
w

witty-crayon-22786

05/11/2020, 11:57 PM
then you can run just your new benchmark with eg
Copy code
./build-support/bin/native/cargo bench --manifest-path=src/rust/engine/Cargo.toml --package store -- $mybenchname
a

aloof-angle-91616

05/11/2020, 11:58 PM
does that have profiling? or is that a command line i can give to instruments?
i am aware there is benchmarking but not sure if profiling is included
w

witty-crayon-22786

05/11/2020, 11:58 PM
i’d launch it, and then attach instruments once it has started
👍 1
if you fiddle with the
measurement_time
, you can get it to basically go in a loop for a while.
and then attach.
a

aloof-angle-91616

05/11/2020, 11:59 PM
this is great!
and i'm rebased to today's master so no sweat about merge conflicts this code is very separate
ok, cool. i can see how i can slot in a snapshot subset benchmark for the large henries snapshot here
will probably extract the materialize_directory() call to a private
fn
took me a second to understand
let path_buf = clean_line.split_whitespace().collect::<PathBuf>();
gonna leave a comment
i have
MODE=debug
in my environment....lol. gonna make the benchmark still but that would probably affect performance
i can definitely repro the issue without
MODE=debug
in a criterion benchmark
instruments is so nice to use
the filesystem traces don't work though i'm gonna google that
w

witty-crayon-22786

05/12/2020, 1:01 AM
i mostly use Time Profiler
haven’t had much luck with the rest.
a

aloof-angle-91616

05/12/2020, 1:01 AM
yep that is what i am settling on
thanks
the time profiler very clearly points
ingest_directory_from_sorted_path_stats
taking up most of the time and being incredibly recursive
which is what the comment on the method says as well
Screen Shot 2020-05-11 at 18.08.51.png
which aligns with our earlier hunches
so to clarify a bit more
it looks like
merge_directories_recursive()
may also be the issue or another issue
in that it is recursive
and it appears to be used in the snapshot subsetting operation
oh no
i was right first
get_snapshot_subset() => from_path_stats() => ingest_directory_from_sorted_path_stats()
, which is recursive
and there are many different threads of execution that took e.g. 800ms
i think this makes sense if there's massive contention at some point, either on lmdb or on a lock somewhere
but the join_alls don't necessarily seem to end
Screen Shot 2020-05-11 at 18.27.17.png
373 ms on cloning a hashbrown table, it seems
we might be able to remove the hashmap here and work with slices
i'm beginning to think more about another
ShardedLmdb
to cache
Digest => Vec<FileContent>
instead of recomputing it each time in
Store::contents_for_directory()
(to improve
SnapshotSubset
time). playing around with it now
w

witty-crayon-22786

05/12/2020, 2:04 AM
so, from what i can tell you have only selected that last portion of the profile
what was happening during the rest?
a

aloof-angle-91616

05/12/2020, 2:04 AM
great question
w

witty-crayon-22786

05/12/2020, 2:04 AM
headed to dinner, but: looking at only one portion of of the time will lie a bit
a

aloof-angle-91616

05/12/2020, 2:05 AM
i strongly agree
thank you!
w

witty-crayon-22786

05/12/2020, 2:05 AM
recommend clicking the button at the top right of the “Heaviest Stack Trace” part
👍 1