i would also potentially try `RUST_BACKTRACE=full`...
# developmenta
aloof-angle-91616
10/09/2019, 9:35 PMi would also potentially try
RUST_BACKTRACE=fulle
early-needle-54791
10/09/2019, 9:36 PMthey are all failing with
Copy code
thread 'remote::tests::remote_workunits_are_stored' panicked at 'called `Result::unwrap()` on an `Err` value: Fatal("Error fetching stdout digest (Digest(Fingerprint<693d8db7b05e99c6b7a7c0616456039d89c555029026936248085193559a0b5d>, 16)): \"Error making env for store at \\\"/var/f
olders/cf/z3ktw6dn467gvm24fgx8ft6c0000gp/T/.tmpBlydey/files/0\\\": No space left on device\"")a
aloof-angle-91616
10/09/2019, 9:37 PMgot it
aloof-angle-91616
10/09/2019, 9:37 PMthank you
aloof-angle-91616
10/09/2019, 9:37 PMi'm updating git repositories right now, might be a while
e
early-needle-54791
10/09/2019, 9:37 PMthanks
early-needle-54791
10/09/2019, 9:37 PMFYI i blew the /var/folders/cf/blahblah directory away but the issue persists.
early-needle-54791
10/09/2019, 9:38 PMbefore I blew it away the size was 3.4Gb should be below the 5gb that the caching command runner creates itself with
👍 1
a
aloof-angle-91616
10/09/2019, 9:38 PMgot it
aloof-angle-91616
10/09/2019, 9:41 PMoff of a master i just pulled, i am not reproducing that error when running with the same command except with
../../../src/rust/engineMODE=debugRUST_BACKTRACE=fullaloof-angle-91616
10/09/2019, 9:41 PMi.e. all the tests are passing
aloof-angle-91616
10/09/2019, 9:41 PMand if you blew away that dir then ugh
e
early-needle-54791
10/09/2019, 9:42 PMyeah sorry I do also run from the engine directory as well
a
aloof-angle-91616
10/09/2019, 9:42 PMyeah exactly
aloof-angle-91616
10/09/2019, 9:42 PMi have been stuck in this exact situation for a while before when developing on rust code especially the caching command runner specifically and haven't gotten out of it which is not good but i don't remember what i did the last time
e
early-needle-54791
10/09/2019, 9:42 PMthere wasn't anything in /private/var/folders that looked related at all
a
aloof-angle-91616
10/09/2019, 9:42 PMi might also try blowing away
~/.cache/pantsgit clean -xfdaloof-angle-91616
10/09/2019, 9:43 PMyeah that's why i'm thinking maybe clearing the lmdb store might work
aloof-angle-91616
10/09/2019, 9:43 PMbut not sure
aloof-angle-91616
10/09/2019, 9:43 PMas in, maybe it has pointers to bad filesystem entries
aloof-angle-91616
10/09/2019, 9:43 PM(guessing)
aloof-angle-91616
10/09/2019, 9:43 PMtrying very hard to remember what i did last time, it was extremely frustrating. sorry
e
early-needle-54791
10/09/2019, 9:47 PMno worries for now. I'm cleaning out pants
early-needle-54791
10/09/2019, 9:47 PMand if that doesn't work ill remove the pants cache dir
👍 1
a
aloof-angle-91616
10/09/2019, 9:48 PMok. if we figure it out we should document it this time -- my fault for not following up previously. if this continues to block you later today or tomorrow morning we could pair on it or someone else can pair on it, whoever has context/time
a
average-vr-56795
10/10/2019, 12:42 PM@fancy-motherboard-24956 has run into this in the past and rebooting his actual OS has generally been his fix; can’t remember whether we worked out anything more useful
average-vr-56795
10/10/2019, 12:42 PMIt didn’t seem to actually relate to disk space, but may have been related to number of free inodes on a tmp mount or something?
f
fancy-motherboard-24956
10/10/2019, 12:44 PMYes, I can confirm I’ve hit this a couple of time and Hagai also hit this once and asked me for advice. I don’t have a fix at this point, unfortunately. Rebooting works as a workaround.
fancy-motherboard-24956
10/10/2019, 12:46 PMWhere I would have expected the general OS limits to have something to do with this, nothing I tried setting them (max files, max process handles…) to unlimited did anything.
e
early-needle-54791
10/10/2019, 5:12 PMsame, unlimited open files didnt change anything, and the disk was obviously not full. inodes could have been an issue. Either way a restart fixed it 🤧