https://pantsbuild.org/ logo
Powered by
h

happy-kitchen-89482

04/03/2023, 10:13 PM
WTF is "Forbidden skipped test"? We don't allow 
pytest.mark.skip
?
b

bitter-ability-32190

04/03/2023, 10:17 PM
Yes. Not in CI
You can skip things locally, like tests if X isn't on the host. But in CI we want to run all our tests or else we have a gap in testing
h

hundreds-father-404

04/03/2023, 10:18 PM
use 
@pytest.mark.no_error_if_skipped
it was meant to solve the issue of for example us skipping tests locally if you don't have Go installed, so that 
pants test ::
doesn't error. But wanting to ensure CI still ran those tests
this comes from the top level 
conftest.py
-- better error message definitely welcomed
h

happy-kitchen-89482

04/04/2023, 5:12 AM
Seems like unnecessary complication to me but 🤷‍♂️
What if we have flaky tests that we want to skip in CI until we can fix them
h

hundreds-father-404

04/04/2023, 5:18 AM
then use 
@pytest.mark.no_error_if_skipped
. It's basically a custom way of saying
No, really, skip this in CI too!
h

happy-kitchen-89482

04/04/2023, 5:25 PM
So we just obfuscated?
This is what 
skip
was already for
b

bitter-ability-32190

04/04/2023, 5:26 PM
I don't completely agree.
Since we don't have code coverage of any kind, we need to have some amount of confidence we aren't skipping tests due to CI infrastructure issues
h

happy-kitchen-89482

04/04/2023, 5:27 PM
But sometimes we want to do that! 🙂
b

bitter-ability-32190

04/04/2023, 5:27 PM
Yes, then you use the escape level
But that shouldn't be the default, and you SHOULD have to reach for it
➕ 1
h

happy-kitchen-89482

04/04/2023, 5:29 PM
I... disagree. But whatever.
I'm unstuck for now.
b

bitter-ability-32190

04/04/2023, 5:33 PM
If it makes you feel better I don't think there should even be an escape lever. So if it helps to think I'm on the fringe 😂
h

happy-kitchen-89482

04/04/2023, 6:01 PM
I mean we literally have tests RIGHT NOW that are flaky in CI, what would you propose?
b

bitter-ability-32190

04/04/2023, 6:21 PM
Skip em, remove them, or rerun them are the only options. IMO tests that flake are the same as tests that fail which aren't tolerable in CI
h

happy-kitchen-89482

04/04/2023, 6:21 PM
"Skip em". Exactly.
b

bitter-ability-32190

04/04/2023, 6:32 PM
In this case though, unconditional skips are the same as just removing the test. Except the dead code just sits there
h

happy-kitchen-89482

04/04/2023, 6:38 PM
It's temporary, though, so you don't actually want to delete it
b

bitter-ability-32190

04/04/2023, 7:05 PM
Source code control means nothing is ever truly deleted 😊
h

happy-kitchen-89482

04/04/2023, 7:12 PM
But it bitrots in the revision history. At least this way mypy can continue to provide some maintenance even if the code isn't executed.
And the code is sitting there, easy to find when you grep for 
skip
If you delete it, nothing urges you to fix it
b

bitter-ability-32190

04/04/2023, 7:18 PM
As opposed to the urgency we feel to fix it in it's current form 😛
Either way, escape lever skip is the path forward here
3 Views
Powered by