https://pantsbuild.org/ logo
#general
Title
# general
p

purple-umbrella-89891

05/03/2022, 7:45 AM
Hi all, as mentioned in the #welcome channel, I'm co-responsible for a large python package at Bosch. One problem we have encountered when trying to switch to pants is the coverage report for our setup. Specifically, we run our tests in our CI pipeline three times, more or less like so:
Copy code
COVERAGE_FILE=.coverage.1 pytest -rfs -v --cov=source/mypkg --option1 tests/
COVERAGE_FILE=.coverage.2 pytest -rfs -v --cov=source/mypkg --option2 tests/
COVERAGE_FILE=.coverage.3 pytest -rfs -v --cov=source/mypkg --option3 tests/
coverage combine .coverage.*
coverage report
coverage xml -o coverage.xml
The custom options
--option1
,
--option2
and
--option3
affect both the code paths in the tests and the selection of which tests to run. The subsets running for the three options are to a large extent overlapping, so it is not possible to split them into directories. Moving to pants, I additionally used tags for the selection of the tests, to avoid the error when a file has no tests selected due to one of the options. The relevant environment is as follows:
Copy code
# pants.toml
[coverage-py]
report = ["console", "raw"]

# pyproject.toml
[tool.coverage.report]
ignore_errors = true
And the commands look like this:
Copy code
./pants --tag='+subset1' test --test-use-coverage --coverage-py-output-dir=dist/coverage/1 :: -- --option1
./pants --tag='+subset2' test --test-use-coverage --coverage-py-output-dir=dist/coverage/2 :: -- --option2
./pants --tag='+subset3' test --test-use-coverage --coverage-py-output-dir=dist/coverage/3 :: -- --option3
Here is the main problem: Even though at the end of each one of the commands, the report on the console is accurate, trying to combine or even just report the coverage from the
dist/coverage/*/.coverage
files results in 0% coverage, which means I cannot export the combined report to xml. Also a minor problem, which might be helpful/related, but is solved on its own: If I leave out the
ignore_errors = true
in
pyproject.toml
, I get the following error, although there is no file with the name
config-3.py
in the project:
Copy code
ProcessExecutionFailure: Process 'Generate Pytest report coverage report.' failed with exit code 1.
stdout:
No source for code: '/tmp/process-executionSiZHX7/config-3.py'.
Aborting report output, consider using -i.
For pants I used version 2.10.0 for experimenting/debugging and just tried 2.11.0, which results in the same behavior. I would try to create a minimal example of a repo causing this behavior, but since it would take quite a bit of effort to reduce it accordingly, I wanted to ask the community if anything comes to mind. Thanks in advance!
👀 1
1
h

happy-kitchen-89482

05/03/2022, 2:17 PM
That
config-3.py
issue is odd but apparently not a blocker, so I'll put that to one side for a second and come back to it.
So the main issue is combining coverage reports, it sounds like?
Pants already combines coverage reports for each process it runs, and that produces the console report at the end and the
dist/coverage/*/.coverage
files, so I would expect them to align
So are you saying that each of
dist/coverage/*/.coverage
on its own is wrong, even before you combine the reports? Or is the issue the combining itself?
w

witty-crayon-22786

05/03/2022, 4:35 PM
are you able to set the relevant pytest options in
conftest.py
files instead?
p

purple-umbrella-89891

05/03/2022, 6:13 PM
@happy-kitchen-89482 actually both. The three
.covarage
files don't work with
coverage report
. They show the listing of all files covered, but 0% coverage for each file and also 0% in total. Combining them also produces the same kind of output, but this is to be expected, since the three files have 0% coverage. By the way, I tried to use a
coverage-py
lockfile with the newest version (6.3.2), or without a lockfile and use for my
report
and
combine
commands the same version that pants uses (5.5). Same result every time.
@witty-crayon-22786 which options do you mean? To enable/configure coverage, instead of
--test-use-coverage
?
w

witty-crayon-22786

05/03/2022, 6:15 PM
the options that you’re passing directly to `pytest`… if you were able to pass them via
conftest.py
, you wouldn’t need to invoke pants multiple times
(i.e. the
./pants test … -- $options
)
h

happy-kitchen-89482

05/03/2022, 9:36 PM
I wonder if this is related to how Pants merges coverage files from each underlying test file
So when you run just
./pants --tag='+subset1' test --test-use-coverage --coverage-py-output-dir=dist/coverage/1 :: -- --option1
, pants runs on each file in a separate process and then merges the output into
dist/coverage/1
, and that output is apparently bad in your case
Can you try this: first
rm -rf dist/coverage
, then run
./pants --tag='+subset1' test --test-use-coverage --coverage-py-output-dir=dist/coverage/1 path/to/a/single_test.py -- --option1
and see if the file in
dist/coverage/1
is good for that single file?
Just pick some single test file for
path/to/a/single_test.py
of course...
p

purple-umbrella-89891

05/04/2022, 8:01 AM
Thanks for the replies and ideas! @happy-kitchen-89482 the merging in pants works correctly, as the "Generate Pytest report coverage report" step right after it prints correct numbers to the console. Also, when running with
--no-process-cleanup
and repeating it by running
/tmp/process-executionqbR3SG/__run.sh
, it prints the same correct numbers again. However, outside of the pants processes, I can't manage to read
dist/coverage/1/.coverage
correctly, it lists all the files and has the correct number of statements, but a coverage of 0% for everything besides empty files. I tried it with one file as you suggested and the behavior is exactly the same. @witty-crayon-22786 the challenge is that the options are conflicting with one another, i.e. you cannot have more than one of them in the same run. For your understanding: our package uses TensorFlow and we need to run the tests independently in eager mode and graph mode, which must be set per process right at the beginning. I can imagine we could write a wrapper to
python_tests
in pants, so that for each test multiple
python_test
targets are created, each in a different mode. This way,
pants test ::
would execute and merge all of them. This is a workaround that would work for us, I'm just not familiar enough with pants to pull it off. Just an idea: Since the "Merge Pytest coverage reports" and "Generate Pytest report coverage report" of pants work as intended, would it be possible for me to manually run
coverage combine
within one of those environments, using them as a kind of venv, just to test? I still haven't been able to fully grasp pex, so I'm struggling there. And it it runs correctly (which I assume it will), to manually create such a target in order to do the combine and report?
I've managed to reduce this to a simple repo that can be found here: https://github.com/dbdbc/pants_coverage This shows the issue with only one run, no multiple runs needed.
🙏 1
I managed to work around this issue by defining macros and targets to allow all tests in all modes to run under a single
./pants test --test-use-coverage ::
call. This solves the issue, because I generate the xml file directly, which is then read by other tools (Jenkins/cobertura and SonarQube). However, I found another issue: The coverage is measured too high, because any python files not imported from the tests are not part of the pants chroots and do not count towards the total. I've added this issue to the small repo (link above). @happy-kitchen-89482 would there be an easy fix for this, e.g. to make a target depend on every target below a directory recursively? I tried to add
dependencies=["source/mypkg::"]
to a test, by this does not work.
h

happy-kitchen-89482

05/05/2022, 4:54 PM
Hmm, let me think about that one. I'll take a look at that repo.
Can you explain how the macros trick works?
And is the issue that you have source files that are not depended on by any test?
p

purple-umbrella-89891

05/06/2022, 8:26 AM
Yes, that is exactly the issue. Before pants, we called
pytest --cov=source/mypkg
, which corresponds to the
--source
option in
coverage
and activates this behavior (quoting the docs): "Specifying the source option also enables coverage.py to report on unexecuted files, since it can search the source tree for files that haven’t been measured at all." I've updated the documentation in the small repo to show this. By the way, should we take this discussion to a GitHub issue or do you prefer it here?
The macros trick works as follows: • Changed
conftest.py
to change the mode based on an environment variable
MODE
instead of the command line options
--option1/...
• Created a
pants-plugins/macros.py
and added it to
pants.toml
. It contains the following:
Copy code
def python_tests_multimode(name, extra_env_vars=None, **kwargs):
    if not extra_env_vars:
        extra_env_vars = []

    for mode in ('1', '2', '3'):
        python_tests(
            name=f"{name}_mode{mode}",
            extra_env_vars=extra_env_vars + [f"MODE={mode}"],
            **kwargs,
        )
• Also made this the default for
pants tailor
by adding the following to `pants.toml`:
Copy code
[tailor]
[tailor.alias_mapping]
python_tests = "python_tests_multimode"
This way, 3 targets are defined for each test, one with each mode. Then, I can just run
pants test ::
to run them all in one go and produce the xml coverage report directly. The only drawback with this solution is that each test file is owned by multiple targets and the dependency resolution does not work if a test imports code from another. I worked around that by moving commonly used code to non-test files, e.g. as fixtures in
conftest.py
or boilerplate in a
common.py
or so. I tried to use
parametrize
(see here), but was not successful.
h

happy-kitchen-89482

05/06/2022, 2:50 PM
Neat!
Unfortunate that parametrize didn't work, since that's what it's for...
p

purple-umbrella-89891

05/06/2022, 3:07 PM
Sorry, I forgot to update this thread, with this hint. However even with
parametrize
, the dependency resolution fails if any test imports code from another test. I solved this by refactoring the code anyway, so now both the macro and the parametrize solution are possible.
So, only the coverage of unused code remains open for me now. I've come across multiple cases where a "depend on all targets in subtree X" would be useful (this one included), but couldn't find a way to do it,
source/mypkg::
is not accepted as a dependency. Do you have any other idea how to do it?
And by the way, thanks again for all the help, I really appreciate it! :)
h

happy-kitchen-89482

05/06/2022, 4:20 PM
You're welcome! That's what we're here for
Yeah there is no way to depend on a glob
But you've hit on a general problem
What happens to coverage when some of the source is unreachable from any test
Which is exactly what coverage measurement is for in the first place...
p

purple-umbrella-89891

05/09/2022, 6:59 AM
Of course, will do! 👍
h

happy-kitchen-89482

05/12/2022, 5:29 AM
Hi @purple-umbrella-89891 thanks for the docsite suggested edits. Does that mean you've found a solution to the issue in this thread?
p

purple-umbrella-89891

05/12/2022, 8:11 AM
Hi @happy-kitchen-89482, yes this issue is solved for me, thanks again! Just for the people seeing this in the future, the solution for the coverage problem is to use
[coverage-py].global_report = true
h

happy-kitchen-89482

05/12/2022, 7:06 PM
I should have been aware of that, silly me...
Thanks for finding it!
21 Views