Hi all, as mentioned in the #C01SPQQ2WK1 channel, I'm co-responsible for a large python package at Bosch. One problem we have encountered when trying to switch to pants is the coverage report for our setup. Specifically, we run our tests in our CI pipeline three times, more or less like so:

COVERAGE_FILE=.coverage.1 pytest -rfs -v --cov=source/mypkg --option1 tests/
COVERAGE_FILE=.coverage.2 pytest -rfs -v --cov=source/mypkg --option2 tests/
COVERAGE_FILE=.coverage.3 pytest -rfs -v --cov=source/mypkg --option3 tests/
coverage combine .coverage.*
coverage report
coverage xml -o coverage.xml

The custom options

--option1

,

--option2

and

--option3

affect both the code paths in the tests and the selection of which tests to run. The subsets running for the three options are to a large extent overlapping, so it is not possible to split them into directories. Moving to pants, I additionally used tags for the selection of the tests, to avoid the error when a file has no tests selected due to one of the options. The relevant environment is as follows:

# pants.toml
[coverage-py]
report = ["console", "raw"]

# pyproject.toml
[tool.coverage.report]
ignore_errors = true

And the commands look like this:

./pants --tag='+subset1' test --test-use-coverage --coverage-py-output-dir=dist/coverage/1 :: -- --option1
./pants --tag='+subset2' test --test-use-coverage --coverage-py-output-dir=dist/coverage/2 :: -- --option2
./pants --tag='+subset3' test --test-use-coverage --coverage-py-output-dir=dist/coverage/3 :: -- --option3

Here is the main problem: Even though at the end of each one of the commands, the report on the console is accurate, trying to combine or even just report the coverage from the

dist/coverage/*/.coverage

files results in 0% coverage, which means I cannot export the combined report to xml. Also a minor problem, which might be helpful/related, but is solved on its own: If I leave out the

ignore_errors = true

in

pyproject.toml

, I get the following error, although there is no file with the name

config-3.py

in the project:

ProcessExecutionFailure: Process 'Generate Pytest report coverage report.' failed with exit code 1.
stdout:
No source for code: '/tmp/process-executionSiZHX7/config-3.py'.
Aborting report output, consider using -i.

For pants I used version 2.10.0 for experimenting/debugging and just tried 2.11.0, which results in the same behavior. I would try to create a minimal example of a repo causing this behavior, but since it would take quite a bit of effort to reduce it accordingly, I wanted to ask the community if anything comes to mind. Thanks in advance!

That

config-3.py

issue is odd but apparently not a blocker, so I'll put that to one side for a second and come back to it.

are you able to set the relevant pytest options in

conftest.py

files instead?

@happy-kitchen-89482 actually both. The three

.covarage

files don't work with

coverage report

. They show the listing of all files covered, but 0% coverage for each file and also 0% in total. Combining them also produces the same kind of output, but this is to be expected, since the three files have 0% coverage. By the way, I tried to use a

coverage-py

lockfile with the newest version (6.3.2), or without a lockfile and use for my

report

and

combine

commands the same version that pants uses (5.5). Same result every time.

the options that you’re passing directly to `pytest`… if you were able to pass them via

conftest.py

, you wouldn’t need to invoke pants multiple times

I wonder if this is related to how Pants merges coverage files from each underlying test file

Thanks for the replies and ideas! @happy-kitchen-89482 the merging in pants works correctly, as the "Generate Pytest report coverage report" step right after it prints correct numbers to the console. Also, when running with

--no-process-cleanup

and repeating it by running

/tmp/process-executionqbR3SG/__run.sh

, it prints the same correct numbers again. However, outside of the pants processes, I can't manage to read

dist/coverage/1/.coverage

correctly, it lists all the files and has the correct number of statements, but a coverage of 0% for everything besides empty files. I tried it with one file as you suggested and the behavior is exactly the same. @witty-crayon-22786 the challenge is that the options are conflicting with one another, i.e. you cannot have more than one of them in the same run. For your understanding: our package uses TensorFlow and we need to run the tests independently in eager mode and graph mode, which must be set per process right at the beginning. I can imagine we could write a wrapper to

python_tests

in pants, so that for each test multiple

python_test

targets are created, each in a different mode. This way,

pants test ::

would execute and merge all of them. This is a workaround that would work for us, I'm just not familiar enough with pants to pull it off. Just an idea: Since the "Merge Pytest coverage reports" and "Generate Pytest report coverage report" of pants work as intended, would it be possible for me to manually run

coverage combine

within one of those environments, using them as a kind of venv, just to test? I still haven't been able to fully grasp pex, so I'm struggling there. And it it runs correctly (which I assume it will), to manually create such a target in order to do the combine and report?

Hmm, let me think about that one. I'll take a look at that repo.

Yes, that is exactly the issue. Before pants, we called

pytest --cov=source/mypkg

, which corresponds to the

--source

option in

coverage

and activates this behavior (quoting the docs): "Specifying the source option also enables coverage.py to report on unexecuted files, since it can search the source tree for files that haven’t been measured at all." I've updated the documentation in the small repo to show this. By the way, should we take this discussion to a GitHub issue or do you prefer it here?

Neat!

Sorry, I forgot to update this thread, with this hint. However even with

parametrize

, the dependency resolution fails if any test imports code from another test. I solved this by refactoring the code anyway, so now both the macro and the parametrize solution are possible.

You're welcome! That's what we're here for

Of course, will do! 👍

Hi @purple-umbrella-89891 thanks for the docsite suggested edits. Does that mean you've found a solution to the issue in this thread?

Hi @happy-kitchen-89482, yes this issue is solved for me, thanks again! Just for the people seeing this in the future, the solution for the coverage problem is to use

[coverage-py].global_report = true

I should have been aware of that, silly me...