What's the canonical Pants way of declaring pytest tests are "manual"? •

skip_tests

• a Pants

tag

plus remembering to set the CLI arg? • A

pytest

mark, and an arg in our

pytest

config to skip? • Something I'm missing?

If the entire test file needs to be skipped, I unfortunately don't think that using a PYtest mark will work because PYtest will complain that no test run which I think is exit code number five

What we did was adopt a convention to name our files something like

my_test_manual.py

which avoids it being caught by the filters of

python_tests

. Then, we had a section like

if __name__ == '__main__':
   unittest.main()

and used the

pex_binary

that

tailor

adds to run the tests.

Thanks for sharing that. So then did you have those files included in the Python_sources target?

yup. but the glob filters already did that for us

Oh VERY interesting. I need to sit on that one

What do you mean a manual option?

oops typo. Just option 🙂

I'm interested in hearing more about the flag idea. How would that be different than tags?

It's like an inverse tag. Tags are opt-out. This would be opt-in

Oh I like that idea of tags being opt in, that's my biggest complaint with tags right now is that you have to remember to do the right filtering. Generally I would bias towards a more generic solution here like using tags, because I suspect this would be useful for other goals like the package goal. Feel free to open a ticket with this idea

Hmmmm....

python_tests(
    name="tests",
    tags=["manual_test"],
)

./pants --tag='-manual_test' test path/to/tests/:

Seems to run the tests

probably https://github.com/pantsbuild/pants/issues/14977 and https://github.com/pantsbuild/pants/issues/11123. We need very badly to fix this

use this

def test_hacky():
    pass

Yeah that's another thing we did

We've also gone back-and-forth on whether to special case exit code 5 so that it no longer causes Pants to fail. There are pros and cons to both approaches, so thus far we have stuck with keeping normal PYtest semantics

Since that came up in the marked case because pants collects things that we normally wouldn't have. So we added a dummy unittest in the same file.

I'm gonna try and whip one up

Will that will only work for manual tests? We have some custom markers for integration tests that it sounds like we'd be right back in the same "make a dummy test to circumvent the deselecting error code" issue.

I think if you skip instead of deselect you're OK

Well we're just using the

pytest.mark.<my_custom_marker>

after specifying custom markers in

pytest.ini

. I'm not sure how to configure the skip/deselect behavior. Is that something supported out of the box?

If all you're doing is marking integration tests for the sake of marking, then those tests should be automatically run, no? Are you filtering tests based on that tag?

No, they're only run if we provide the custom marker in the pytest call.

There has to be some machinery in the middle then. Just marking them doesn't deselect them.

Yes, we have some tooling in

conftest.py

that used

pytest_collection_modifyitems

to control what was collected for running. My theory is that since we currently have modules with one marked test in them, pants collects them and then conftest deselects it, and that pants process thinks that no tests were run.

Just been directed to this thread, my context is...

# pytest.ini

[pytest]
markers =
    integration: mark test as integration test

and various test modules that are marked as integration tests, and others that are just unit tests...

# test_some_integration_tests.py

import pytest

pytestmark = pytest.mark.integration

def test_something():
  # some integration tests that rely on db resource

# test_some_unit_tests.py

import pytest

# no marks

def test_something_else():
  # some unit tests that don't rely on any resources

and getting exit code 5 for both of these attempts to run tests...

./pants test :: -- -m "not integration"

-- i.e. just running "unit" tests All of the modules that contain

pytestmark = pytest.mark.integration

return exit code 5 due to

collecting ... collected x item / x deselected / 0 selected

if the unit tests pass, create db and other "expensive" resources in the CI and run integration tests...

./pants test :: -- -m "integration"

-- i.e. just running integration tests All of the modules that don't contain

pytestmark = pytest.mark.integration

also return exit code 5 for the same reason.

@high-yak-85899 try skipping the test instead of deselecting them? Not sure the pytest incantation, but the experiment shows you won't get error code 5

Yes. And unfortunately

./pants --tag='-integration_test' test ::

doesn't work. And I forget if there is a way to recursively match all test targets with a given name, e.g.

./pants test integration_tests/**::

or something

I've adapted the linked plugin to have less weird behavior. Now tests are skipped if manual unless

--run-manual

is specified:

import pytest


def pytest_configure(config):
    config.addinivalue_line("markers", "manual: mark tests which need a person to execute them")


def pytest_addoption(parser):
    group = parser.getgroup("manual", "configuration of manual tests")
    group.addoption(
        "--run-manual",
        action="store_true",
        default=False,
        help="run manual tests",
    )


@pytest.hookimpl(hookwrapper=True, tryfirst=True)
def pytest_runtest_makereport(item, call):
    outcome = yield
    rep = outcome.get_result()
    if call.excinfo and isinstance(call.excinfo.value, pytest.skip.Exception):
        if call.excinfo.value.msg == "manual":
            rep.outcome = "manual"


@pytest.hookimpl(tryfirst=True)
def pytest_report_teststatus(report):
    if report.outcome == "manual":
        return "manual", "M", "MANUAL"


@pytest.hookimpl(tryfirst=True)
def pytest_collection_modifyitems(config, items):
    run_manual = config.getoption("run_manual")

    for item in items:
        if item.get_closest_marker("manual") is not None and not run_manual:
            item.add_marker(pytest.mark.skip("manual"))