#development
Title
h
hundreds-father-404
11/04/2021, 8:13 PMAny thoughts if REAPI would consider adding an ignore mechanism to
output_pathsoutput_paths=("dir/", "!dir/ignore_me")pkg/modpkg/mod/cacheA substantial slowdown for our Go support is loading all the downloaded modules.I found in a trace last week that loading the
process.output_digestw
witty-crayon-22786
11/04/2021, 8:37 PMcould you relocate the cache instead?
h
hundreds-father-404
11/04/2021, 8:38 PMWhat do you mean? Using
SnapshotSubsetProcessw
witty-crayon-22786
11/04/2021, 8:38 PMfwiw, i do think that our experience with gitignore style excludes has been really positive, so they might be accepted as an extension. but i also haven’t been involved there for a while. maybe @average-vr-56795 has thoughts
👍 1
What do you mean? Usingthe latter. reconfiguring the cache location.etc? Or changing theSnapshotSubset(e.g. argv and env vars) to output differently?Process
a
average-vr-56795
11/05/2021, 12:54 AMI suspect people would look at it a bit funny, but probably be ok with it?
👍 1
h
hundreds-father-404
11/05/2021, 12:56 AMDo you know if Bazel suffers from large digests slowing them down too? I'm still filing the ticket for it, but it was ~4 seconds to load the digest from LMDB iirc
w
witty-crayon-22786
11/05/2021, 1:44 AMLarge directory structures, or large files? High counts of either files or directories tends to be a bottleneck moreso than total size
a
average-vr-56795
11/05/2021, 8:02 AMInternally not so much - they've put a lot of work into optimising how they store and manipulate structures to minimise that overhead (e.g. one of the core bazel data structures is a "nested set" - a set which is a union of other sets) so that they can pass around and handle references to things like "the files in a go distribution" from a bunch of actions with very low overhead
Their file transfer code between client and server is pretty unoptimised, which causes some mild woe, but only the first time each file is uploaded somewhere, so it amortises pretty well. But if you're the first person to build some go on a cluster, it's not going to be amazing for you
There's actually also a lot of overhead from the fact that the representations are different - converting nested sets (really optimised for "these things are related") into merkle trees (really optimised for "this representation is canonical and consistent") is also surprisingly overhead-y
And has been discussion about changing the REAPI format to allow a more nested-set-like representation, though it never goes anywhere because there are still optimisations that can be done in the existing implementation
The other big difference for local execution is that actions are all run in symlink forests not places with re-materialised files
(Which probably has similar performance characteristics in practice to the "symlink out to the go toolchain" stuff Tom was designing)
f
fast-nail-55400
11/05/2021, 10:57 AMcould you relocate the cache instead?or maybe run a
rm -rf path/to/cacheProcessh
hundreds-father-404
11/05/2021, 3:46 PMThank you Daniel! And oh, good idea Tom!! We're already using a bash script to run Go code, so that is really easy to do
w
witty-crayon-22786
11/05/2021, 4:56 PM@average-vr-56795: mm, yea. funny you should mention DepSets… we were looking at that yesterday. we’ve used a similar concept in a few places (TransitiveTarget, CoarsenedTarget most recently), but Eric hit a case yesterday with a recursive structure, and it’s becoming clearer why it is generalized in Bazel
➕ 1
and due (in-part) to their docs, i realized the connection to
Directoryso maybe the answer to https://github.com/pantsbuild/pants/issues/13112 is actually a generic recursive ordered set (maybe in Rust, to ease porting the filesystem operations on #13112)
👍 1