Hi all, as mentioned in the <#C01SPQQ2WK1|welcome>...
# general
Hi all, as mentioned in the #welcome channel, I'm co-responsible for a large python package at Bosch. One problem we have encountered when trying to switch to pants is the coverage report for our setup. Specifically, we run our tests in our CI pipeline three times, more or less like so:
Copy code
COVERAGE_FILE=.coverage.1 pytest -rfs -v --cov=source/mypkg --option1 tests/
COVERAGE_FILE=.coverage.2 pytest -rfs -v --cov=source/mypkg --option2 tests/
COVERAGE_FILE=.coverage.3 pytest -rfs -v --cov=source/mypkg --option3 tests/
coverage combine .coverage.*
coverage report
coverage xml -o coverage.xml
The custom options
affect both the code paths in the tests and the selection of which tests to run. The subsets running for the three options are to a large extent overlapping, so it is not possible to split them into directories. Moving to pants, I additionally used tags for the selection of the tests, to avoid the error when a file has no tests selected due to one of the options. The relevant environment is as follows:
Copy code
# pants.toml
report = ["console", "raw"]

# pyproject.toml
ignore_errors = true
And the commands look like this:
Copy code
./pants --tag='+subset1' test --test-use-coverage --coverage-py-output-dir=dist/coverage/1 :: -- --option1
./pants --tag='+subset2' test --test-use-coverage --coverage-py-output-dir=dist/coverage/2 :: -- --option2
./pants --tag='+subset3' test --test-use-coverage --coverage-py-output-dir=dist/coverage/3 :: -- --option3
Here is the main problem: Even though at the end of each one of the commands, the report on the console is accurate, trying to combine or even just report the coverage from the
files results in 0% coverage, which means I cannot export the combined report to xml. Also a minor problem, which might be helpful/related, but is solved on its own: If I leave out the
ignore_errors = true
, I get the following error, although there is no file with the name
in the project:
Copy code
ProcessExecutionFailure: Process 'Generate Pytest report coverage report.' failed with exit code 1.
No source for code: '/tmp/process-executionSiZHX7/config-3.py'.
Aborting report output, consider using -i.
For pants I used version 2.10.0 for experimenting/debugging and just tried 2.11.0, which results in the same behavior. I would try to create a minimal example of a repo causing this behavior, but since it would take quite a bit of effort to reduce it accordingly, I wanted to ask the community if anything comes to mind. Thanks in advance!
👀 1
issue is odd but apparently not a blocker, so I'll put that to one side for a second and come back to it.
So the main issue is combining coverage reports, it sounds like?
Pants already combines coverage reports for each process it runs, and that produces the console report at the end and the
files, so I would expect them to align
So are you saying that each of
on its own is wrong, even before you combine the reports? Or is the issue the combining itself?
are you able to set the relevant pytest options in
files instead?
@happy-kitchen-89482 actually both. The three
files don't work with
coverage report
. They show the listing of all files covered, but 0% coverage for each file and also 0% in total. Combining them also produces the same kind of output, but this is to be expected, since the three files have 0% coverage. By the way, I tried to use a
lockfile with the newest version (6.3.2), or without a lockfile and use for my
commands the same version that pants uses (5.5). Same result every time.
@witty-crayon-22786 which options do you mean? To enable/configure coverage, instead of
the options that you’re passing directly to `pytest`… if you were able to pass them via
, you wouldn’t need to invoke pants multiple times
(i.e. the
./pants test … -- $options
I wonder if this is related to how Pants merges coverage files from each underlying test file
So when you run just
./pants --tag='+subset1' test --test-use-coverage --coverage-py-output-dir=dist/coverage/1 :: -- --option1
, pants runs on each file in a separate process and then merges the output into
, and that output is apparently bad in your case
Can you try this: first
rm -rf dist/coverage
, then run
./pants --tag='+subset1' test --test-use-coverage --coverage-py-output-dir=dist/coverage/1 path/to/a/single_test.py -- --option1
and see if the file in
is good for that single file?
Just pick some single test file for
of course...
Thanks for the replies and ideas! @happy-kitchen-89482 the merging in pants works correctly, as the "Generate Pytest report coverage report" step right after it prints correct numbers to the console. Also, when running with
and repeating it by running
, it prints the same correct numbers again. However, outside of the pants processes, I can't manage to read
correctly, it lists all the files and has the correct number of statements, but a coverage of 0% for everything besides empty files. I tried it with one file as you suggested and the behavior is exactly the same. @witty-crayon-22786 the challenge is that the options are conflicting with one another, i.e. you cannot have more than one of them in the same run. For your understanding: our package uses TensorFlow and we need to run the tests independently in eager mode and graph mode, which must be set per process right at the beginning. I can imagine we could write a wrapper to
in pants, so that for each test multiple
targets are created, each in a different mode. This way,
pants test ::
would execute and merge all of them. This is a workaround that would work for us, I'm just not familiar enough with pants to pull it off. Just an idea: Since the "Merge Pytest coverage reports" and "Generate Pytest report coverage report" of pants work as intended, would it be possible for me to manually run
coverage combine
within one of those environments, using them as a kind of venv, just to test? I still haven't been able to fully grasp pex, so I'm struggling there. And it it runs correctly (which I assume it will), to manually create such a target in order to do the combine and report?
I've managed to reduce this to a simple repo that can be found here: https://github.com/dbdbc/pants_coverage This shows the issue with only one run, no multiple runs needed.
🙏 1
I managed to work around this issue by defining macros and targets to allow all tests in all modes to run under a single
./pants test --test-use-coverage ::
call. This solves the issue, because I generate the xml file directly, which is then read by other tools (Jenkins/cobertura and SonarQube). However, I found another issue: The coverage is measured too high, because any python files not imported from the tests are not part of the pants chroots and do not count towards the total. I've added this issue to the small repo (link above). @happy-kitchen-89482 would there be an easy fix for this, e.g. to make a target depend on every target below a directory recursively? I tried to add
to a test, by this does not work.
Hmm, let me think about that one. I'll take a look at that repo.
Can you explain how the macros trick works?
And is the issue that you have source files that are not depended on by any test?
Yes, that is exactly the issue. Before pants, we called
pytest --cov=source/mypkg
, which corresponds to the
option in
and activates this behavior (quoting the docs): "Specifying the source option also enables coverage.py to report on unexecuted files, since it can search the source tree for files that haven’t been measured at all." I've updated the documentation in the small repo to show this. By the way, should we take this discussion to a GitHub issue or do you prefer it here?
The macros trick works as follows: • Changed
to change the mode based on an environment variable
instead of the command line options
• Created a
and added it to
. It contains the following:
Copy code
def python_tests_multimode(name, extra_env_vars=None, **kwargs):
    if not extra_env_vars:
        extra_env_vars = []

    for mode in ('1', '2', '3'):
            extra_env_vars=extra_env_vars + [f"MODE={mode}"],
• Also made this the default for
pants tailor
by adding the following to `pants.toml`:
Copy code
python_tests = "python_tests_multimode"
This way, 3 targets are defined for each test, one with each mode. Then, I can just run
pants test ::
to run them all in one go and produce the xml coverage report directly. The only drawback with this solution is that each test file is owned by multiple targets and the dependency resolution does not work if a test imports code from another. I worked around that by moving commonly used code to non-test files, e.g. as fixtures in
or boilerplate in a
or so. I tried to use
(see here), but was not successful.
Unfortunate that parametrize didn't work, since that's what it's for...
Sorry, I forgot to update this thread, with this hint. However even with
, the dependency resolution fails if any test imports code from another test. I solved this by refactoring the code anyway, so now both the macro and the parametrize solution are possible.
So, only the coverage of unused code remains open for me now. I've come across multiple cases where a "depend on all targets in subtree X" would be useful (this one included), but couldn't find a way to do it,
is not accepted as a dependency. Do you have any other idea how to do it?
And by the way, thanks again for all the help, I really appreciate it! :)
You're welcome! That's what we're here for
Yeah there is no way to depend on a glob
But you've hit on a general problem
What happens to coverage when some of the source is unreachable from any test
Which is exactly what coverage measurement is for in the first place...
Of course, will do! 👍
Hi @purple-umbrella-89891 thanks for the docsite suggested edits. Does that mean you've found a solution to the issue in this thread?
Hi @happy-kitchen-89482, yes this issue is solved for me, thanks again! Just for the people seeing this in the future, the solution for the coverage problem is to use
[coverage-py].global_report = true
I should have been aware of that, silly me...
Thanks for finding it!