https://pantsbuild.org/ logo
#general
Title
# general
h

helpful-lunch-92084

04/29/2021, 3:14 PM
hey, we just upgraded from 2.2.0 to 2.2.2 and started getting this type of error on some of our build jobs:
Copy code
MDB_MAP_FULL: Environment mapsize limit reached"
Any suggestions?
h

happy-kitchen-89482

04/29/2021, 3:42 PM
Hmm, I think that means LMDB is "full". What does
du -h -d0 ~/.cache/pants/lmdb_store/
show?
What happens if you nuke/move that dir?
h

helpful-lunch-92084

04/29/2021, 3:46 PM
Copy code
159M	.cache/pants/lmdb_store/
weird, i moved lmdb_store and re-ran and it still failed but a new lmdb_store didn’t exist. I then nuked ~/.cache/pants dir and re-ran and it still failed and again no lmdb_store nor named_caches exist
we noticed this on our build server, testing locally now
h

happy-kitchen-89482

04/29/2021, 4:13 PM
Hmmmm
oh, maybe you have no permissions to write to that dir or something?
can't think why 2.2.2 would make a difference there, will look at the changelog
h

helpful-lunch-92084

04/29/2021, 4:24 PM
yah, our build server does a pants test :: so that seems to manifest it. when i revert to 2.2.0 the tests pass
oh wait, yah, the lmdb_store dir i ran du on was the wrong one, the correct one is 51G
trying to move that now and re-test
alright, yah nuking the 51G lmdb_store fixed the issue
are we going to have to periodically purge that dir now?
h

hundreds-father-404

04/29/2021, 4:33 PM
Pantsd is supposed to do garbage cleanup, although I'm honestly not certain it does
h

helpful-lunch-92084

04/29/2021, 4:33 PM
we’re not running pantsd on our ci server
h

hundreds-father-404

04/29/2021, 4:35 PM
Ah, we used to recommend turning pantsd off in CI but it should be safe to have on and should slightly speed up CI if you have multiple subsequent calls to ./pants
h

helpful-lunch-92084

04/29/2021, 4:36 PM
interesting, ok, we’ll test that out
h

hundreds-father-404

04/29/2021, 4:54 PM
I think likely. It sounds like you have a workaround, right? If so, could you please comment on that PR what is going on? Stu is out on paternity leave. He understands this best, and he'll probably want to know about this when he gets back, so we want to preserve what happened
h

helpful-lunch-92084

04/29/2021, 4:54 PM
sure
h

happy-kitchen-89482

04/29/2021, 5:10 PM
How is that directory getting preserved across CI runs? Is it set up to be cached by the CI provider? Or are you running your own CI servers on long-lived instances?
h

helpful-lunch-92084

04/29/2021, 5:11 PM
the latter
tbh, we noticed some finickyness with pantsd early on when testing it out with our workstations where pants would hang and we’d have to kill pantsd to unwedge stuff and then noted that you didn’t recommend it on ci and just kind of ran with that w/out much thought. clearly though, on long-lived instances, pantsd does make a lot of sense.
h

hundreds-father-404

04/29/2021, 5:22 PM
Yeah I think our recommendation in 1.30 and 2.0 was an acknowledgement that pantsd was more finicky than we wanted, and because CI normally only runs 1-2 pants invocations it's not worth the risk of something going wrong. But we've been incrementally improving things so it is hopefully more safe to use pantsd in CI
h

happy-kitchen-89482

04/29/2021, 7:12 PM
Yeah, for long-running CI servers I think it makes sense
2 Views