https://pantsbuild.org/ logo
Join Slack
Powered by
WTF is "Forbidden skipped test"? We don't allow `p...
# development
h
WTF is "Forbidden skipped test"? We don't allow 
pytest.mark.skip
?
b
Yes. Not in CI
You can skip things locally, like tests if X isn't on the host. But in CI we want to run all our tests or else we have a gap in testing
h
use 
@pytest.mark.no_error_if_skipped
it was meant to solve the issue of for example us skipping tests locally if you don't have Go installed, so that 
pants test ::
doesn't error. But wanting to ensure CI still ran those tests
this comes from the top level 
conftest.py
-- better error message definitely welcomed
h
Seems like unnecessary complication to me but 🤷‍♂️
What if we have flaky tests that we want to skip in CI until we can fix them
h
then use 
@pytest.mark.no_error_if_skipped
. It's basically a custom way of saying
No, really, skip this in CI too!
h
So we just obfuscated?
This is what 
skip
was already for
b
I don't completely agree.
Since we don't have code coverage of any kind, we need to have some amount of confidence we aren't skipping tests due to CI infrastructure issues
h
But sometimes we want to do that! 🙂
b
Yes, then you use the escape level
But that shouldn't be the default, and you SHOULD have to reach for it
➕ 1
h
I... disagree. But whatever.
I'm unstuck for now.
b
If it makes you feel better I don't think there should even be an escape lever. So if it helps to think I'm on the fringe 😂
h
I mean we literally have tests RIGHT NOW that are flaky in CI, what would you propose?
b
Skip em, remove them, or rerun them are the only options. IMO tests that flake are the same as tests that fail which aren't tolerable in CI
h
"Skip em". Exactly.
b
In this case though, unconditional skips are the same as just removing the test. Except the dead code just sits there
h
It's temporary, though, so you don't actually want to delete it
b
Source code control means nothing is ever truly deleted 😊
h
But it bitrots in the revision history. At least this way mypy can continue to provide some maintenance even if the code isn't executed.
And the code is sitting there, easy to find when you grep for 
skip
If you delete it, nothing urges you to fix it
b
As opposed to the urgency we feel to fix it in it's current form 😛
Either way, escape lever skip is the path forward here
3 Views
Open in Slack
PreviousNext
Powered by