https://pantsbuild.org/ logo
#development
Title
# development
w

witty-crayon-22786

02/02/2021, 9:48 PM
according to
vmmap
, we were `mmap`’ing 1.7TB… and… it was right! https://github.com/pantsbuild/pants/pull/11513
👀 1
h

hundreds-father-404

02/02/2021, 9:48 PM
was this...a regression?
w

witty-crayon-22786

02/02/2021, 9:49 PM
no, not as far as i know. we had been overallocating for a long time.
1
h

hundreds-father-404

02/02/2021, 9:49 PM
so we should prob cherry-pick, possibly as far back as 1.30?
w

witty-crayon-22786

02/02/2021, 9:50 PM
we don’t have any known crashers in 1.30/2.0.x/2.1.x, so i don’t think so.
the comment in the change is mostly accurate… this shouldn’t matter… there are just some rare cases where it does, like core dump gathering
👍 1
but will pick to 2.2.x probably
h

hundreds-father-404

02/02/2021, 9:51 PM
Thanks. Somewhat related, 2.1's branch is busted for some reason, and I'm not sure why. I tried creating a fresh PR against it last week and it still failed with the same Git issue that it couldn't find the ref head
a

average-vr-56795

02/02/2021, 10:46 PM
Commented on the PR - I agree that the 1.7TB should be trivial to shrink down to 110GB or so (yay simple bug!), but I'm not sure about the other shrinking...
(Also I'm fairly sure if someone already has a larger-than-now-configured shared, their new Pants version will insta-crash?)
(I generally somewhat regret choosing LMDB, in retrospect, it's a little too inflexible imo...)
w

witty-crayon-22786

02/02/2021, 11:21 PM
makes total sense! thanks for the comment
❤️ 1
yea, the need to shard complicated things. need to re-benchmark at some point.
(Also I’m fairly sure if someone already has a larger-than-now-configured shared, their new Pants version will insta-crash?)
the docs indicate that you can successfully open a larger store… it will only fail if you try to further increase the size. https://docs.rs/lmdb/0.8.0/lmdb/struct.EnvironmentBuilder.html#method.set_map_size
👍 1