https://pantsbuild.org/ logo
#development
Title
# development
a

ancient-vegetable-10556

12/08/2022, 5:28 PM
Is anyone else getting spurious CI test failures that can’t seem to be cleared?
b

bitter-ability-32190

12/08/2022, 5:33 PM
It's been SUPER flaky for me the last few days
I'm really worried its the immut hardlinking 🙈
a

ancient-vegetable-10556

12/08/2022, 5:34 PM
I’m getting:
Exceeded timeout of 450.0 seconds when executing local process: Run Pytest for src/python/pants/init/load_backends_integration_test.py:tests
496
Exceeded timeout of 300.0 seconds when executing local process: Run Pytest for src/python/pants/backend/python/goals/export_integration_test.py:tests
415
which is, I guess, plausible that Pants is getting too big?
@bitter-ability-32190 maybe fire off a draft PR that reverts the change and see how reliably CI passes?
a

ancient-vegetable-10556

12/08/2022, 5:37 PM
hrm, I’m not getting that one
f

fast-nail-55400

12/08/2022, 5:37 PM
(for my earlier report of CI timeouts)
could be different issues here but CI has been problematic for past few days
h

happy-kitchen-89482

12/08/2022, 5:38 PM
Yep, seeing this as well
b

bitter-ability-32190

12/08/2022, 5:38 PM
maybe fire off a draft PR that reverts the change and see how reliably CI passes?
If someone wants to do this, i'm about to be in meeting marathon
a

ancient-vegetable-10556

12/08/2022, 5:39 PM
@bitter-ability-32190 do you have a PR number? I can do it
b

bitter-ability-32190

12/08/2022, 5:40 PM
a

ancient-vegetable-10556

12/08/2022, 5:40 PM
on it
w

wide-midnight-78598

12/08/2022, 5:44 PM
I had the timeout on some unrelated code, with PyOxidizer's tests
a

ancient-vegetable-10556

12/08/2022, 5:45 PM
It’s weird that that one would fail consistently.
I’m seeing that one most of the time
@bitter-ability-32190 Unfortunately: https://github.com/pantsbuild/pants/pull/17751 — all tests passed.
b

bitter-ability-32190

12/08/2022, 6:51 PM
🤮
FYI @witty-crayon-22786
a

ancient-vegetable-10556

12/08/2022, 6:52 PM
I’m going to push a change that will fix linting so that we can merge it
w

witty-crayon-22786

12/08/2022, 6:54 PM
this definitely smells more like network flakes, but who knows.
a

ancient-vegetable-10556

12/08/2022, 6:55 PM
the things that were failing consistently produce large files (pyoxidizer tests, export tests, etc)
w

witty-crayon-22786

12/08/2022, 6:55 PM
and hit the network a lot. *shrug
a

ancient-vegetable-10556

12/08/2022, 6:56 PM
unless we were caching the failures?
w

witty-crayon-22786

12/08/2022, 6:56 PM
pyoxidizer downloads python interpreters, etc
a

ancient-vegetable-10556

12/08/2022, 6:56 PM
well yes, and you’d expect it to fail here under similar circumstances
w

witty-crayon-22786

12/08/2022, 6:58 PM
did we bump the timeouts to see if those tests got slower, rather than having hung?
(royal we, sorry)
a

ancient-vegetable-10556

12/08/2022, 6:58 PM
not yet
@witty-crayon-22786 I’ll do that now, along with the revert on the other branch
https://github.com/pantsbuild/pants/pull/17752 Here’s a branch testing increased timeouts on the consistently-failing tests.
@witty-crayon-22786 Looks like
pyoxidizer
tests are still timing out, even with a timeout of 960s (double what’s on
main
).
Confirmed: Increasing test timeouts does not fix the issue.
w

wide-midnight-78598

12/08/2022, 7:26 PM
Is that on a specific platform that its failing?
a

ancient-vegetable-10556

12/08/2022, 7:26 PM
Linux tests
w

wide-midnight-78598

12/08/2022, 7:26 PM
x86, I guess
a

ancient-vegetable-10556

12/08/2022, 7:27 PM
and only after @bitter-ability-32190’s symlink changes; reverting that PR fixes the failure
I’m getting that reversion PR into a state where it’s mergeable
w

wide-midnight-78598

12/08/2022, 7:27 PM
🤯
a

ancient-vegetable-10556

12/08/2022, 8:29 PM
https://github.com/pantsbuild/pants/pull/17751 @bitter-ability-32190 python tests have passed three times in CI on this branch, while other PRs (including ones that increase timeouts on the consistently problematic tests) continue failing. 😞
@bitter-ability-32190 I saw your comment just before merging; happy to give that a go
b

bitter-ability-32190

12/08/2022, 8:49 PM
Yeah, sorry the OG PR took 2 weeks to work on, and my other rust merge conflicts have left me scarred
a

ancient-vegetable-10556

12/08/2022, 8:49 PM
Yeah, I can see
I can make a PR that sets the limit to 2GB, or you can tell me how to set to max
b
a

ancient-vegetable-10556

12/08/2022, 8:50 PM
k
b

bitter-ability-32190

12/08/2022, 8:50 PM
There's a test that will need disabled too
I think the easiest way to do that is to comment out the
#[tokio::test]
with a comment
a

ancient-vegetable-10556

12/08/2022, 8:52 PM
where?
b

bitter-ability-32190

12/08/2022, 8:53 PM
commented
I appreciate you doing this. Sorry this PR gave us heartache
a

ancient-vegetable-10556

12/08/2022, 8:54 PM
OK, PR incoming in a minute or three
just need to make sure that everything passes over here
f

fast-nail-55400

12/08/2022, 9:09 PM
I appreciate you doing this. Sorry this PR gave us heartache
the joys of our industry ... unknown unknowns 🙂
b

bitter-ability-32190

12/08/2022, 9:11 PM
I'm a bit disheartened. This was supposed to speed up runs... Although @witty-crayon-22786 could this be that with each integration test, we have a new daemon which means a new immut tmpdir?
w

witty-crayon-22786

12/08/2022, 9:12 PM
yes, could be. although how that would make things slower rather than … crashier, i don’t know.
b

bitter-ability-32190

12/08/2022, 9:12 PM
It's be extra read-only'ing and hardlinking
After running some integration tests I DO see a lot of immut tempdirs in my
/tmp
I'm up to 197 right now
a

ancient-vegetable-10556

12/08/2022, 9:17 PM
@bitter-ability-32190 the suggested changed makes rust linting fail on fs/store/src/lib.rs:1403
due to always being false
(I think I can fix this…)
b

bitter-ability-32190

12/08/2022, 9:18 PM
just needs a leading underscore
a

ancient-vegetable-10556

12/08/2022, 9:19 PM
what does?
b

bitter-ability-32190

12/08/2022, 9:19 PM
oh I was thinking of something else
sorry
just subtract 1 from MAX 😂
a

ancient-vegetable-10556

12/08/2022, 9:19 PM
looks like
#[allow(clippy::absurd_extreme_comparisons)]
should fix it
b

bitter-ability-32190

12/08/2022, 9:19 PM
LOLOLOL
❤️
a

ancient-vegetable-10556

12/08/2022, 9:19 PM
oh, subtracting 1 would fix too, I guess
let’s do that
b

bitter-ability-32190

12/08/2022, 9:20 PM
That might still be absurdly extreme
a

ancient-vegetable-10556

12/08/2022, 9:20 PM
but not ‘always false’
I swear, I was planning on getting an early afternoon nap in today 😉
b

bitter-ability-32190

12/08/2022, 9:23 PM
IOU one nap
a

ancient-vegetable-10556

12/08/2022, 9:24 PM
the branch this is preventing seeing the light of day is really cool too
b

busy-vase-39202

12/08/2022, 9:26 PM
more environments stuff?
a

ancient-vegetable-10556

12/08/2022, 9:26 PM
nope 🙂
b

busy-vase-39202

12/08/2022, 9:27 PM
ooh mystery coolness
a

ancient-vegetable-10556

12/08/2022, 9:27 PM
I want it to be a surprise
b

busy-vase-39202

12/08/2022, 9:27 PM
exciting
👍
a

ancient-vegetable-10556

12/08/2022, 10:13 PM
ah, finally passed rust linting. Python tests should be 20 minutes from now
pyo3 tests succeeded; thank etc
OK, change is merged, now re-running CI for my branches with
main
merged in. 🤞