hey, we just upgraded from 2.2.0 to 2.2.2 and star...
# generalh
helpful-lunch-92084
04/29/2021, 3:14 PMhey, we just upgraded from 2.2.0 to 2.2.2 and started getting this type of error on some of our build jobs:
Copy code
MDB_MAP_FULL: Environment mapsize limit reached"h
happy-kitchen-89482
04/29/2021, 3:42 PMHmm, I think that means LMDB is "full". What does
du -h -d0 ~/.cache/pants/lmdb_store/happy-kitchen-89482
04/29/2021, 3:42 PMWhat happens if you nuke/move that dir?
h
helpful-lunch-92084
04/29/2021, 3:46 PM Copy code
159M .cache/pants/lmdb_store/helpful-lunch-92084
04/29/2021, 3:57 PMweird, i moved lmdb_store and re-ran and it still failed but a new lmdb_store didn’t exist. I then nuked ~/.cache/pants dir and re-ran and it still failed and again no lmdb_store nor named_caches exist
helpful-lunch-92084
04/29/2021, 3:58 PMwe noticed this on our build server, testing locally now
h
happy-kitchen-89482
04/29/2021, 4:13 PMHmmmm
happy-kitchen-89482
04/29/2021, 4:13 PMoh, maybe you have no permissions to write to that dir or something?
happy-kitchen-89482
04/29/2021, 4:13 PMcan't think why 2.2.2 would make a difference there, will look at the changelog
h
helpful-lunch-92084
04/29/2021, 4:24 PMyah, our build server does a pants test :: so that seems to manifest it. when i revert to 2.2.0 the tests pass
helpful-lunch-92084
04/29/2021, 4:25 PMoh wait, yah, the lmdb_store dir i ran du on was the wrong one, the correct one is 51G
helpful-lunch-92084
04/29/2021, 4:25 PMtrying to move that now and re-test
helpful-lunch-92084
04/29/2021, 4:30 PMalright, yah nuking the 51G lmdb_store fixed the issue
helpful-lunch-92084
04/29/2021, 4:31 PMare we going to have to periodically purge that dir now?
h
hundreds-father-404
04/29/2021, 4:33 PMPantsd is supposed to do garbage cleanup, although I'm honestly not certain it does
h
helpful-lunch-92084
04/29/2021, 4:33 PMwe’re not running pantsd on our ci server
h
hundreds-father-404
04/29/2021, 4:35 PMAh, we used to recommend turning pantsd off in CI but it should be safe to have on and should slightly speed up CI if you have multiple subsequent calls to ./pants
h
helpful-lunch-92084
04/29/2021, 4:36 PMinteresting, ok, we’ll test that out
helpful-lunch-92084
04/29/2021, 4:46 PMcould it have been this commit? https://github.com/pantsbuild/pants/commit/75f984f8abbc5be30522bd34d4b553b642ec6b1c
h
hundreds-father-404
04/29/2021, 4:54 PMI think likely. It sounds like you have a workaround, right?
If so, could you please comment on that PR what is going on? Stu is out on paternity leave. He understands this best, and he'll probably want to know about this when he gets back, so we want to preserve what happened
h
helpful-lunch-92084
04/29/2021, 4:54 PMsure
h
happy-kitchen-89482
04/29/2021, 5:10 PMHow is that directory getting preserved across CI runs? Is it set up to be cached by the CI provider? Or are you running your own CI servers on long-lived instances?
h
helpful-lunch-92084
04/29/2021, 5:11 PMthe latter
helpful-lunch-92084
04/29/2021, 5:18 PMtbh, we noticed some finickyness with pantsd early on when testing it out with our workstations where pants would hang and we’d have to kill pantsd to unwedge stuff and then noted that you didn’t recommend it on ci and just kind of ran with that w/out much thought.
clearly though, on long-lived instances, pantsd does make a lot of sense.
h
hundreds-father-404
04/29/2021, 5:22 PMYeah I think our recommendation in 1.30 and 2.0 was an acknowledgement that pantsd was more finicky than we wanted, and because CI normally only runs 1-2 pants invocations it's not worth the risk of something going wrong.
But we've been incrementally improving things so it is hopefully more safe to use pantsd in CI
h
happy-kitchen-89482
04/29/2021, 7:12 PMYeah, for long-running CI servers I think it makes sense
2 Views