purple-umbrella-89891
05/03/2022, 7:45 AMCOVERAGE_FILE=.coverage.1 pytest -rfs -v --cov=source/mypkg --option1 tests/
COVERAGE_FILE=.coverage.2 pytest -rfs -v --cov=source/mypkg --option2 tests/
COVERAGE_FILE=.coverage.3 pytest -rfs -v --cov=source/mypkg --option3 tests/
coverage combine .coverage.*
coverage report
coverage xml -o coverage.xml
The custom options --option1
, --option2
and --option3
affect both the code paths in the tests and the selection of which tests to run. The subsets running for the three options are to a large extent overlapping, so it is not possible to split them into directories.
Moving to pants, I additionally used tags for the selection of the tests, to avoid the error when a file has no tests selected due to one of the options. The relevant environment is as follows:
# pants.toml
[coverage-py]
report = ["console", "raw"]
# pyproject.toml
[tool.coverage.report]
ignore_errors = true
And the commands look like this:
./pants --tag='+subset1' test --test-use-coverage --coverage-py-output-dir=dist/coverage/1 :: -- --option1
./pants --tag='+subset2' test --test-use-coverage --coverage-py-output-dir=dist/coverage/2 :: -- --option2
./pants --tag='+subset3' test --test-use-coverage --coverage-py-output-dir=dist/coverage/3 :: -- --option3
Here is the main problem: Even though at the end of each one of the commands, the report on the console is accurate, trying to combine or even just report the coverage from the dist/coverage/*/.coverage
files results in 0% coverage, which means I cannot export the combined report to xml.
Also a minor problem, which might be helpful/related, but is solved on its own: If I leave out the ignore_errors = true
in pyproject.toml
, I get the following error, although there is no file with the name config-3.py
in the project:
ProcessExecutionFailure: Process 'Generate Pytest report coverage report.' failed with exit code 1.
stdout:
No source for code: '/tmp/process-executionSiZHX7/config-3.py'.
Aborting report output, consider using -i.
For pants I used version 2.10.0 for experimenting/debugging and just tried 2.11.0, which results in the same behavior.
I would try to create a minimal example of a repo causing this behavior, but since it would take quite a bit of effort to reduce it accordingly, I wanted to ask the community if anything comes to mind.
Thanks in advance!happy-kitchen-89482
05/03/2022, 2:17 PMconfig-3.py
issue is odd but apparently not a blocker, so I'll put that to one side for a second and come back to it.dist/coverage/*/.coverage
files, so I would expect them to aligndist/coverage/*/.coverage
on its own is wrong, even before you combine the reports? Or is the issue the combining itself?witty-crayon-22786
05/03/2022, 4:35 PMconftest.py
files instead?purple-umbrella-89891
05/03/2022, 6:13 PM.covarage
files don't work with coverage report
. They show the listing of all files covered, but 0% coverage for each file and also 0% in total. Combining them also produces the same kind of output, but this is to be expected, since the three files have 0% coverage.
By the way, I tried to use a coverage-py
lockfile with the newest version (6.3.2), or without a lockfile and use for my report
and combine
commands the same version that pants uses (5.5). Same result every time.--test-use-coverage
?witty-crayon-22786
05/03/2022, 6:15 PMconftest.py
, you wouldn’t need to invoke pants multiple times./pants test … -- $options
)happy-kitchen-89482
05/03/2022, 9:36 PM./pants --tag='+subset1' test --test-use-coverage --coverage-py-output-dir=dist/coverage/1 :: -- --option1
, pants runs on each file in a separate process and then merges the output into dist/coverage/1
, and that output is apparently bad in your caserm -rf dist/coverage
, then run ./pants --tag='+subset1' test --test-use-coverage --coverage-py-output-dir=dist/coverage/1 path/to/a/single_test.py -- --option1
and see if the file in dist/coverage/1
is good for that single file?path/to/a/single_test.py
of course...purple-umbrella-89891
05/04/2022, 8:01 AM--no-process-cleanup
and repeating it by running /tmp/process-executionqbR3SG/__run.sh
, it prints the same correct numbers again. However, outside of the pants processes, I can't manage to read dist/coverage/1/.coverage
correctly, it lists all the files and has the correct number of statements, but a coverage of 0% for everything besides empty files. I tried it with one file as you suggested and the behavior is exactly the same.
@witty-crayon-22786 the challenge is that the options are conflicting with one another, i.e. you cannot have more than one of them in the same run. For your understanding: our package uses TensorFlow and we need to run the tests independently in eager mode and graph mode, which must be set per process right at the beginning. I can imagine we could write a wrapper to python_tests
in pants, so that for each test multiple python_test
targets are created, each in a different mode. This way, pants test ::
would execute and merge all of them. This is a workaround that would work for us, I'm just not familiar enough with pants to pull it off.
Just an idea: Since the "Merge Pytest coverage reports" and "Generate Pytest report coverage report" of pants work as intended, would it be possible for me to manually run coverage combine
within one of those environments, using them as a kind of venv, just to test? I still haven't been able to fully grasp pex, so I'm struggling there. And it it runs correctly (which I assume it will), to manually create such a target in order to do the combine and report?./pants test --test-use-coverage ::
call. This solves the issue, because I generate the xml file directly, which is then read by other tools (Jenkins/cobertura and SonarQube).
However, I found another issue: The coverage is measured too high, because any python files not imported from the tests are not part of the pants chroots and do not count towards the total. I've added this issue to the small repo (link above). @happy-kitchen-89482 would there be an easy fix for this, e.g. to make a target depend on every target below a directory recursively? I tried to add dependencies=["source/mypkg::"]
to a test, by this does not work.happy-kitchen-89482
05/05/2022, 4:54 PMpurple-umbrella-89891
05/06/2022, 8:26 AMpytest --cov=source/mypkg
, which corresponds to the --source
option in coverage
and activates this behavior (quoting the docs): "Specifying the source option also enables coverage.py to report on unexecuted files, since it can search the source tree for files that haven’t been measured at all." I've updated the documentation in the small repo to show this.
By the way, should we take this discussion to a GitHub issue or do you prefer it here?conftest.py
to change the mode based on an environment variable MODE
instead of the command line options --option1/...
• Created a pants-plugins/macros.py
and added it to pants.toml
. It contains the following:
def python_tests_multimode(name, extra_env_vars=None, **kwargs):
if not extra_env_vars:
extra_env_vars = []
for mode in ('1', '2', '3'):
python_tests(
name=f"{name}_mode{mode}",
extra_env_vars=extra_env_vars + [f"MODE={mode}"],
**kwargs,
)
• Also made this the default for pants tailor
by adding the following to `pants.toml`:
[tailor]
[tailor.alias_mapping]
python_tests = "python_tests_multimode"
This way, 3 targets are defined for each test, one with each mode. Then, I can just run pants test ::
to run them all in one go and produce the xml coverage report directly.
The only drawback with this solution is that each test file is owned by multiple targets and the dependency resolution does not work if a test imports code from another. I worked around that by moving commonly used code to non-test files, e.g. as fixtures in conftest.py
or boilerplate in a common.py
or so.
I tried to use parametrize
(see here), but was not successful.happy-kitchen-89482
05/06/2022, 2:50 PMpurple-umbrella-89891
05/06/2022, 3:07 PMparametrize
, the dependency resolution fails if any test imports code from another test. I solved this by refactoring the code anyway, so now both the macro and the parametrize solution are possible.source/mypkg::
is not accepted as a dependency. Do you have any other idea how to do it?happy-kitchen-89482
05/06/2022, 4:20 PMpurple-umbrella-89891
05/09/2022, 6:59 AMhappy-kitchen-89482
05/12/2022, 5:29 AMpurple-umbrella-89891
05/12/2022, 8:11 AM[coverage-py].global_report = true
happy-kitchen-89482
05/12/2022, 7:06 PM