https://pantsbuild.org/ logo
Join Slack
Powered by
Is anyone else getting spurious CI test failures t...
# development
a
Is anyone else getting spurious CI test failures that can’t seem to be cleared?
b
It's been SUPER flaky for me the last few days
I'm really worried its the immut hardlinking 🙈
a
I’m getting:
Exceeded timeout of 450.0 seconds when executing local process: Run Pytest for src/python/pants/init/load_backends_integration_test.py:tests
496
Exceeded timeout of 300.0 seconds when executing local process: Run Pytest for src/python/pants/backend/python/goals/export_integration_test.py:tests
415
which is, I guess, plausible that Pants is getting too big?
@bitter-ability-32190 maybe fire off a draft PR that reverts the change and see how reliably CI passes?
f
a
hrm, I’m not getting that one
f
(for my earlier report of CI timeouts)
could be different issues here but CI has been problematic for past few days
h
Yep, seeing this as well
b
maybe fire off a draft PR that reverts the change and see how reliably CI passes?
If someone wants to do this, i'm about to be in meeting marathon
a
@bitter-ability-32190 do you have a PR number? I can do it
b
a
on it
w
I had the timeout on some unrelated code, with PyOxidizer's tests
a
It’s weird that that one would fail consistently.
I’m seeing that one most of the time
@bitter-ability-32190 Unfortunately: https://github.com/pantsbuild/pants/pull/17751 — all tests passed.
b
🤮
FYI @witty-crayon-22786
a
I’m going to push a change that will fix linting so that we can merge it
w
this definitely smells more like network flakes, but who knows.
a
the things that were failing consistently produce large files (pyoxidizer tests, export tests, etc)
w
and hit the network a lot. *shrug
a
unless we were caching the failures?
w
pyoxidizer downloads python interpreters, etc
a
well yes, and you’d expect it to fail here under similar circumstances
w
did we bump the timeouts to see if those tests got slower, rather than having hung?
(royal we, sorry)
a
not yet
@witty-crayon-22786 I’ll do that now, along with the revert on the other branch
https://github.com/pantsbuild/pants/pull/17752 Here’s a branch testing increased timeouts on the consistently-failing tests.
@witty-crayon-22786 Looks like 
pyoxidizer
tests are still timing out, even with a timeout of 960s (double what’s on 
main
).
Confirmed: Increasing test timeouts does not fix the issue.
w
Is that on a specific platform that its failing?
a
Linux tests
w
x86, I guess
a
and only after @bitter-ability-32190’s symlink changes; reverting that PR fixes the failure
I’m getting that reversion PR into a state where it’s mergeable
w
🤯
a
https://github.com/pantsbuild/pants/pull/17751 @bitter-ability-32190 python tests have passed three times in CI on this branch, while other PRs (including ones that increase timeouts on the consistently problematic tests) continue failing. 😞
@bitter-ability-32190 I saw your comment just before merging; happy to give that a go
b
Yeah, sorry the OG PR took 2 weeks to work on, and my other rust merge conflicts have left me scarred
a
Yeah, I can see
I can make a PR that sets the limit to 2GB, or you can tell me how to set to max
b
a
k
b
There's a test that will need disabled too
I think the easiest way to do that is to comment out the 
#[tokio::test]
with a comment
a
where?
b
commented
I appreciate you doing this. Sorry this PR gave us heartache
a
OK, PR incoming in a minute or three
just need to make sure that everything passes over here
f
I appreciate you doing this. Sorry this PR gave us heartache
the joys of our industry ... unknown unknowns 🙂
b
I'm a bit disheartened. This was supposed to speed up runs... Although @witty-crayon-22786 could this be that with each integration test, we have a new daemon which means a new immut tmpdir?
w
yes, could be. although how that would make things slower rather than … crashier, i don’t know.
b
It's be extra read-only'ing and hardlinking
After running some integration tests I DO see a lot of immut tempdirs in my 
/tmp
I'm up to 197 right now
a
@bitter-ability-32190 the suggested changed makes rust linting fail on fs/store/src/lib.rs:1403
due to always being false
(I think I can fix this…)
b
just needs a leading underscore
a
what does?
b
oh I was thinking of something else
sorry
just subtract 1 from MAX 😂
a
looks like 
#[allow(clippy::absurd_extreme_comparisons)]
should fix it
b
LOLOLOL
❤️
a
oh, subtracting 1 would fix too, I guess
let’s do that
b
That might still be absurdly extreme
a
but not ‘always false’
I swear, I was planning on getting an early afternoon nap in today 😉
b
IOU one nap
a
the branch this is preventing seeing the light of day is really cool too
b
more environments stuff?
a
nope 🙂
b
ooh mystery coolness
a
I want it to be a surprise
b
exciting
👍
a
ah, finally passed rust linting. Python tests should be 20 minutes from now
pyo3 tests succeeded; thank etc
OK, change is merged, now re-running CI for my branches with 
main
merged in. 🤞
Open in Slack
PreviousNext
Powered by