https://pantsbuild.org/ logo
Join Slack
Powered by
Specifically, I want my pytest to use coverage to ...
# general
g
Specifically, I want my pytest to use coverage to enforce 100% coverage, and mypy to enforce 100% typing.
h
MyPy runs against every Python file you specify, so :: should get you 100% coverage. (So long as Pants knows about those python files by having a BUILD file / target for them) ./pants test :: runs against every python_tests target. We tell pytest-cov to report on all code used in each distinct test. This means you get 100% coverage, but only if every source file is imported (directly or transitively) by your tests. Independent of Pants, I wasn’t able to figure out how to get Coverage to report on files never used by pytest. Do you know of a way? I spent ~1-2 full days trying to figure it out and couldn’t get it
g
Is there a way to specify pytest config file location like there is the other plugins?
h
You can specify confest.py files, which automatically get used for all tests in and below the directory. And you can permanently set pass through arguments by setting [pytest].args No support for a config file, but not for any particular reason. We could add it. And it’s not too complex of a change if you’d be interested in adding it - I could write up some instructions
g
Sorry, I meant how do I set the threshold of coverage to 100%. So anything less than 100% fails tests.
h
Oh, I see. For both coverage and MyPy, they have a config file option. I think you can use their options to set the threshold I’m not 100% certain that the error code would propagate properly for test coverage though. If not, that’s an easy bug fix https://www.pantsbuild.org/docs/reference-coverage-py https://www.pantsbuild.org/docs/reference-mypy
g
Oh, that shows where I can specify the coverage rc file, that should be enough. Thank you.
Really, I just need to explore this sub-system part a bit further.
h
Cool. Let us know if the error code isn’t propagating like you’d like. Also FYI on how coverage works: each individual test process uses Pytest cov to get data on every file encountered in the test run. Then, after all tests are done, we run coverage combine to get back a single result
g
I'm not getting consistent results back.
👀 1
Screen Shot 2020-11-01 at 4.58.34 PM.png
I'm showing both a 91% and a 7.08%
Same thing, but includes a printout of my BUILD file
h
Interesting, it looks like Pytest Cov is what’s failing with the 7%? Because it’s failing for that specific test, rather than the final coverage report Are you running multiple test files in this command, or only the one?
g
./pants test ::
But this one test file is all I have.
image.png
👍 1
h
And what’s your coverage rc file?
g
The fail on less than 100%.
Which is my goal, I want 100% forced code coverage.
I also tried putting in as a pytest argument, same thing (without the crash in the middle).
h
Are you able to push this repo to public? It’d be helpful for us to debug what’s going on
g
The repo is public, but this branch isn't yet. 1 sec.
👍 1
h
No rush. I’m afk today and OOO until Wednesday because of working at the election. I won’t be able to get to this till then, but others might tomorrow In the meantime, you might be able to use a tool to post process the coverage data and have the error come from that step in CI. Such as coveralls. That's how Toolchain does it But we definitely should fix the approach you’re trying to use
Hm looks like that branch doesn’t have the coverage file, that’s the part I’m interested in to be able to reproduce
g
Oh, sorry, I just removed it after your comment.
This commit has them.
Lol, now I'm just getting this and I don't know why..
h
Hm, this is a broken window with pantsd that we’re planning to fix. The issue is that once that specific run was cached/memoized, we don’t log agin. Which is bad UX. We should always log the tool’s output To force it to rerun, in the meantime, you can either add a trivial white space change to a file, or use 
--no-pantsd
Okay I reproduce. Thanks for posting that reprository
I see the issue. We tell 
pytest-cov
to report on 
.
, meaning everything, whereas we tell 
coverage.py
to only report on the specific files being used iiuc.
Copy code
▶ ./pants test src/python/greeter/test_main.py --coverage-py-filter='greeter'
...
Required test coverage of 80.0% reached. Total coverage: 90.91%

...

✓ src/python/greeter/test_main.py:test succeeded.

Name                              Stmts   Miss  Cover
-----------------------------------------------------
src/python/greeter/__init__.py        0      0   100%
src/python/greeter/main.py            4      1    75%
src/python/greeter/test_main.py       7      0   100%
-----------------------------------------------------
TOTAL                                11      1    91%
Ah, this is what Pants is essentially running:
Copy code
▶ ./pytest.pex src/python/greeter/test_main.py --cov . --no-header
======================================= test session starts ========================================
collected 3 items

src/python/greeter/test_main.py ...                                                          [100%]

---------- coverage: platform darwin, python 3.8.5-final-0 -----------
Name                                                Stmts   Miss  Cover
-----------------------------------------------------------------------
pytest.pex/.bootstrap/pex/third_party/__init__.py     234    227     3%
src/python/greeter/__init__.py                          0      0   100%
src/python/greeter/main.py                              4      1    75%
src/python/greeter/test_main.py                         7      0   100%
-----------------------------------------------------------------------
TOTAL                                                 245    228     7%
The data is messed up by that 
pytest.pex
file, which is being included by 
.
Another ah ha, this was a regression in 2.0.x. Thanks for finding it! Will try to get a fix out today and cherry-pick into 2.0
Fix #1: https://github.com/pantsbuild/pants/pull/11110 Another fix coming soon to not throw an exception when 
coverage.py
fails when writing a report
2.1.0rc0 is now out, which has the fix for coverage. See https://www.pantsbuild.org/v2.1/docs/release-notes-2-1 for release notes and https://www.pantsbuild.org/v2.1/docs/upgrade-tips for tips on upgrading We’ll release 2.0.1rc0 soon too
6 Views
Open in Slack
PreviousNext
Powered by