https://pantsbuild.org/ logo
Join Slack
Powered by
Hello folks! I have a question about using covera...
# general
o
Hello folks! I have a question about using coverage.py through pants. We recently noticed that pants doesn’t generate test coverage report for files that are not encountered in the transitive closure of the test run which seems to match what the documentation says here -
Coverage will not report on unencountered files
Coverage will only report on files encountered during the tests’ run. This means that your coverage score may be misleading; even with a score of 100%, you may have files without any tests.
This is a shortcoming of Coverage itself.
https://www.pantsbuild.org/docs/python-test-goal#coverage However, it would be good to get some sense of which files are missing documentation in our monorepo. As the repo gets larger, it becomes more difficult to track where the coverage is lacking. So, my question is - Are there any known workarounds to generate coverage for files that are not covered by unit tests?
FWIW, coverage.py’s documentation specifies that it is possible by providing the source files using the 
source
option in the run configuration -
Specifying the source option also enables coverage.py to report on unexecuted files, since it can search the source tree for files that haven’t been measured at all. Only importable files (ones at the root of the tree, or in directories with a 
__init__.py
 file) will be considered.
https://coverage.readthedocs.io/en/coverage-5.1/source.html#execution
For reference, this is what our 
pants.toml
and 
.coveragerc
files look like -
Copy code
[test]
use_coverage = true

[coverage-py]
report = ["raw", "xml", "json", "console"]
config = ".coveragerc"
.coveragerc
Copy code
[run]
branch = True
relative_files = True
source =
    # include all files under src
    src/

[report]
show_missing = True
e
So, you can get this to work with pants only if you artificially make your tests depend on all src/ files. The downsides to that include more boilerplate to set that up and maintain it, but probably worse: tests will be forced to re-run even when there are unrelated (in reality) src/ changes. I just re-read coverage docs and .coveragerc does support environment variable interpolation. That seems like the crease we need to support this without adding the artificial blanket dependency. Your .coveragerc would then say 
source = $BUILDROOT/src
If it's not clear, the fundamental issue here is that Pants runs all processes in a sandbox directory off in /tmp. We only copy the files an action needs (depends on) into the sandbox. The env var trick would allow coverage to see outside that sandbox.
And also to be clear, this env var trick does not work today, I'm just brainstorming.
j
Wouldn't a completely different goal be an acceptable way to solve this? 
--use-coverage
is answering how well is the code I am testing covered? To answer what is the coverage of this target, then maybe 
./pants coverage :target
could be called (and target could be 
::
to get the full repo report). If the mechanics for getting coverage info is too entangled with tests, then maybe 
./pants test --get-complete-coverage
would take various targets (e.g. 
python_library
, 
python_tests
etc) to build up the source list.
h
@jolly-midnight-72759 I think you may be on to something... That is an interesting idea.
j
I like John's idea of 
pants
orchestrating the environment in which the tools it calls on operate. This ability would probably make it easier to have pants manage more languages. (My "don't build it until you need it" alarm just went off as I typed that, so maybe that isn't a good reason to do it.)
e
Actually @jolly-midnight-72759 the existing test goal is sort of set up this way. The results of test are (optionally) fed into a coverage rule set. I just experimented and you can have the rule set get a --global flag which triggers running ~:`coverage run --source src/root dummy.py`, combining that runs resulting noop 
.coverage
(which is fast to generate!) with the test run 
.coverage
and now suddenly your report includes the missing coverage for all python sources in the src/root too without ever re-running the tests.
👖 2
So I'll file an issue with the sketch, but we can and should support this.
And it doesn;t use any of my env var hack sketched above. This is honest and performant and all the good stuff we try to deliver with Pants.
The only slow thing is the generation of the --global report itself. But that's exactly what you ordered with --global - so you pay for what you ask (only).
j
Sweet!
e
Probably our biggest current and future issue with Pants development will be getting people who really know how to use the underlying tools hacking on Pants. Its missing that knowledge that leaves features on the floor for any particular tool, In this case all the regular Pants developers were just too pytest-cov / coverage uninformed. It took me a good bit of reading and experimenting to become informed this morning. So! If you know a tool really well we'd love your help adding Pants support for it. We know Pants really well and the two can complement each other.
💯 1
👍 1
o
Wow! Just came back from the weekend and saw that the issue is already solved! Thanks so much for the super fast turnaround on this 😄. Is this expected to be released in the 2.4.2 release?
e
There was no plan to (we haven't backported it to 2.5.x either). I just re-read https://www.pantsbuild.org/docs/release-strategy#stable-releases though and it appears you asking for the feature backport triggers a clause that allows this. I think the rest of the conditions are met (easy backport, low risk, no deprecations) but I'd like a second opinion on this. @hundreds-father-404 has a better feel for our rules here.
o
oh I didn’t realize that 2.5.x was already out. We are fine with it being released in the next patch of 2.5.x. We are currently on 2.2 and are planning to upgrade to the latest release very soon.
h
Yeah I think 2.5 is more appropriate I agree it's low risk because it's gated by an option
o
@hundreds-father-404 - is there a suggested way to upgrade pants versions? We are trying to upgrade from 2.2 to 2.5 (and also give 2.6.0.dev0 a try). Do the latest versions of pants need python 3.9 to run or do I need to clear some cache? I get the following error while following the steps mentioned here -
Copy code
$ printf '[GLOBAL]\npants_version = "2.5.0"\n' > pants.toml

$ curl -L -o ./pants <https://pantsbuild.github.io/setup/pants> && \
chmod +x ./pants

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   162  100   162    0     0    686      0 --:--:-- --:--:-- --:--:--   686
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100 11047  100 11047    0     0  20012      0 --:--:-- --:--:-- --:--:--  399k


$ ./pants --version

Using base prefix '/usr/local/opt/python@3.9/Frameworks/Python.framework/Versions/3.9'
Traceback (most recent call last):
  File "/Users/paritosh/.cache/pants/setup/bootstrap-Darwin-x86_64/virtualenv-16.4.3/virtualenv.py", line 2567, in <module>
    main()
  File "/Users/paritosh/.cache/pants/setup/bootstrap-Darwin-x86_64/virtualenv-16.4.3/virtualenv.py", line 783, in main
    create_environment(
  File "/Users/paritosh/.cache/pants/setup/bootstrap-Darwin-x86_64/virtualenv-16.4.3/virtualenv.py", line 1071, in create_environment
    install_python(home_dir, lib_dir, inc_dir, bin_dir, site_packages=site_packages, clear=clear, symlink=symlink)
  File "/Users/paritosh/.cache/pants/setup/bootstrap-Darwin-x86_64/virtualenv-16.4.3/virtualenv.py", line 1361, in install_python
    copy_required_modules(home_dir, symlink)
  File "/Users/paritosh/.cache/pants/setup/bootstrap-Darwin-x86_64/virtualenv-16.4.3/virtualenv.py", line 1254, in copy_required_modules
    dst_filename = change_prefix(filename, dst_prefix)
  File "/Users/paritosh/.cache/pants/setup/bootstrap-Darwin-x86_64/virtualenv-16.4.3/virtualenv.py", line 1186, in change_prefix
    assert False, "Filename {} does not start with any of these prefixes: {}".format(filename, prefixes)
AssertionError: Filename /usr/local/Cellar/python@3.9/3.9.4/Frameworks/Python.framework/Versions/3.9/lib/python3.9/os.py does not start with any of these prefixes: ['/usr/local/opt/python@3.9/Frameworks/Python.framework/Versions/3.9/Extras/lib/python', '/usr/local/opt/python@3.9/Frameworks/Python.framework/Versions/3.9', '/usr/local/opt/python@3.9/Frameworks/Python.framework/Versions/3.9', '/Users/paritosh/Library/Python/3.9/lib/python/site-packages', '/Users/paritosh/.local/lib/python/3.9/site-packages', '/Users/paritosh/Library/Python/3.9/site-packages', '/Library/Python/3.9/site-packages']
./pants: line 282: /Users/paritosh/.cache/pants/setup/bootstrap-Darwin-x86_64/2.5.0_py39/bin/python: No such file or directory
h
Huh, I would not expect that...thanks for the report. Will look in the morning. What happens with 2.3.0 and 2.4.0?
o
Interestingly both 2.3 and 2.4 work! I tried 2.5 again but still the same issue.
Copy code
$ ./pants --version

Using base prefix '/usr/local/opt/pyenv/versions/3.7.7'
New python executable in /Users/paritosh/.cache/pants/setup/bootstrap-Darwin-x86_64/pants.gOMWvL/install/bin/python3.7
Also creating executable in /Users/paritosh/.cache/pants/setup/bootstrap-Darwin-x86_64/pants.gOMWvL/install/bin/python
Installing setuptools, pip, wheel...
done.
Collecting pip
  Using cached <https://files.pythonhosted.org/packages/cd/6f/43037c7bcc8bd8ba7c9074256b1a11596daa15555808ec748048c1507f08/pip-21.1.1-py3-none-any.whl>
Installing collected packages: pip
  Found existing installation: pip 19.0.3
    Uninstalling pip-19.0.3:
      Successfully uninstalled pip-19.0.3
Successfully installed pip-21.1.1
Collecting pantsbuild.pants==2.3.0
  Downloading pantsbuild.pants-2.3.0-cp37-cp37m-macosx_10_11_x86_64.whl (7.0 MB)

Collecting PyYAML<5.5,>=5.4
  Using cached PyYAML-5.4.1-cp37-cp37m-macosx_10_9_x86_64.whl (249 kB)
Collecting psutil==5.7.0
  Using cached psutil-5.7.0-cp37-cp37m-macosx_10_15_x86_64.whl
Collecting typing-extensions==3.7.4.2
  Using cached typing_extensions-3.7.4.2-py3-none-any.whl (22 kB)
Collecting pex==2.1.34
  Downloading pex-2.1.34-py2.py3-none-any.whl (2.6 MB)

Collecting fasteners==0.15.0
  Using cached fasteners-0.15-py2.py3-none-any.whl (23 kB)
Collecting setproctitle==1.2
  Using cached setproctitle-1.2-cp37-cp37m-macosx_10_15_x86_64.whl
Collecting ansicolors==1.1.8
  Using cached ansicolors-1.1.8-py2.py3-none-any.whl (13 kB)
Collecting toml==0.10.2
  Using cached toml-0.10.2-py2.py3-none-any.whl (16 kB)
Collecting setuptools<54.0,>=50.3.0
  Using cached setuptools-53.1.0-py3-none-any.whl (784 kB)
Collecting packaging==20.9
  Using cached packaging-20.9-py2.py3-none-any.whl (40 kB)
Collecting monotonic>=0.1
  Using cached monotonic-1.6-py2.py3-none-any.whl (8.2 kB)
Collecting six
  Using cached six-1.16.0-py2.py3-none-any.whl (11 kB)
Collecting pyparsing>=2.0.2
  Using cached pyparsing-2.4.7-py2.py3-none-any.whl (67 kB)
Installing collected packages: six, pyparsing, monotonic, typing-extensions, toml, setuptools, setproctitle, PyYAML, psutil, pex, packaging, fasteners, ansicolors, pantsbuild.pants
  Attempting uninstall: setuptools
    Found existing installation: setuptools 40.8.0
    Uninstalling setuptools-40.8.0:
      Successfully uninstalled setuptools-40.8.0
Successfully installed PyYAML-5.4.1 ansicolors-1.1.8 fasteners-0.15 monotonic-1.6 packaging-20.9 pantsbuild.pants-2.3.0 pex-2.1.34 psutil-5.7.0 pyparsing-2.4.7 setproctitle-1.2 setuptools-53.1.0 six-1.16.0 toml-0.10.2 typing-extensions-3.7.4.2
New virtual environment successfully created at /Users/paritosh/.cache/pants/setup/bootstrap-Darwin-x86_64/2.3.0_py37.
10:14:25.51 [INFO] initializing pantsd...
10:14:25.63 [INFO] pantsd initialized.
2.3.0
./pants --version  7.84s user 2.41s system 60% cpu 17.053 total
h
Okay, thanks! What's fishy to me is that 2.5 is the first version to work with Python 3.9 I'm afk, but in the pants bash script, there should be a function like default_python or something. In it, try changing 3.8 to come before 3.9. I'm curious if that would fix it
o
that seems to have worked. Thanks for the suggestion Eric!
Getting the following error now. It worked when I ran it against a target that had 2 test files but failed against a target that had 13 test files.
Copy code
$ ./pants --tag='-integration_tests'  test  path/to/my/test_target::

12:48:03.91 [ERROR] Exception caught: (pants.engine.internals.scheduler.ExecutionError)
  File "/Users/paritosh/.cache/pants/setup/bootstrap-Darwin-x86_64/2.6.0.dev0_py37/lib/python3.7/site-packages/pants/bin/local_pants_runner.py", line 234, in _run_inner
    return self._perform_run(goals)
  File "/Users/paritosh/.cache/pants/setup/bootstrap-Darwin-x86_64/2.6.0.dev0_py37/lib/python3.7/site-packages/pants/bin/local_pants_runner.py", line 173, in _perform_run
    return self._perform_run_body(goals, poll=False)
  File "/Users/paritosh/.cache/pants/setup/bootstrap-Darwin-x86_64/2.6.0.dev0_py37/lib/python3.7/site-packages/pants/bin/local_pants_runner.py", line 195, in _perform_run_body
    poll_delay=(0.1 if poll else None),
  File "/Users/paritosh/.cache/pants/setup/bootstrap-Darwin-x86_64/2.6.0.dev0_py37/lib/python3.7/site-packages/pants/init/engine_initializer.py", line 136, in run_goal_rules
    goal_product, params, poll=poll, poll_delay=poll_delay
  File "/Users/paritosh/.cache/pants/setup/bootstrap-Darwin-x86_64/2.6.0.dev0_py37/lib/python3.7/site-packages/pants/engine/internals/scheduler.py", line 530, in run_goal_rule
    self._raise_on_error([t for _, t in throws])
  File "/Users/paritosh/.cache/pants/setup/bootstrap-Darwin-x86_64/2.6.0.dev0_py37/lib/python3.7/site-packages/pants/engine/internals/scheduler.py", line 494, in _raise_on_error
    wrapped_exceptions=tuple(t.exc for t in throws),

Exception message: 1 Exception encountered:

Engine traceback:
  in select
  in pants.core.goals.test.run_tests
  in pants.backend.python.goals.coverage_py.generate_coverage_reports
  in pants.backend.python.goals.coverage_py.merge_coverage_data
Traceback (most recent call last):
  File "/Users/paritosh/.cache/pants/setup/bootstrap-Darwin-x86_64/2.6.0.dev0_py37/lib/python3.7/site-packages/pants/engine/internals/selectors.py", line 654, in native_engine_generator_send
    res = func.send(arg)
  File "/Users/paritosh/.cache/pants/setup/bootstrap-Darwin-x86_64/2.6.0.dev0_py37/lib/python3.7/site-packages/pants/backend/python/goals/coverage_py.py", line 410, in merge_coverage_data
    *coverage_digest_gets,
TypeError: MultiGet() takes from 1 to 10 positional arguments but 14 were given

Traceback (most recent call last):
  File "/Users/paritosh/.cache/pants/setup/bootstrap-Darwin-x86_64/2.6.0.dev0_py37/lib/python3.7/site-packages/pants/bin/local_pants_runner.py", line 234, in _run_inner
    return self._perform_run(goals)
  File "/Users/paritosh/.cache/pants/setup/bootstrap-Darwin-x86_64/2.6.0.dev0_py37/lib/python3.7/site-packages/pants/bin/local_pants_runner.py", line 173, in _perform_run
    return self._perform_run_body(goals, poll=False)
  File "/Users/paritosh/.cache/pants/setup/bootstrap-Darwin-x86_64/2.6.0.dev0_py37/lib/python3.7/site-packages/pants/bin/local_pants_runner.py", line 195, in _perform_run_body
    poll_delay=(0.1 if poll else None),
  File "/Users/paritosh/.cache/pants/setup/bootstrap-Darwin-x86_64/2.6.0.dev0_py37/lib/python3.7/site-packages/pants/init/engine_initializer.py", line 136, in run_goal_rules
    goal_product, params, poll=poll, poll_delay=poll_delay
  File "/Users/paritosh/.cache/pants/setup/bootstrap-Darwin-x86_64/2.6.0.dev0_py37/lib/python3.7/site-packages/pants/engine/internals/scheduler.py", line 530, in run_goal_rule
    self._raise_on_error([t for _, t in throws])
  File "/Users/paritosh/.cache/pants/setup/bootstrap-Darwin-x86_64/2.6.0.dev0_py37/lib/python3.7/site-packages/pants/engine/internals/scheduler.py", line 494, in _raise_on_error
    wrapped_exceptions=tuple(t.exc for t in throws),
pants.engine.internals.scheduler.ExecutionError: 1 Exception encountered:

Engine traceback:
  in select
  in pants.core.goals.test.run_tests
  in pants.backend.python.goals.coverage_py.generate_coverage_reports
  in pants.backend.python.goals.coverage_py.merge_coverage_data
Traceback (most recent call last):
  File "/Users/paritosh/.cache/pants/setup/bootstrap-Darwin-x86_64/2.6.0.dev0_py37/lib/python3.7/site-packages/pants/engine/internals/selectors.py", line 654, in native_engine_generator_send
    res = func.send(arg)
  File "/Users/paritosh/.cache/pants/setup/bootstrap-Darwin-x86_64/2.6.0.dev0_py37/lib/python3.7/site-packages/pants/backend/python/goals/coverage_py.py", line 410, in merge_coverage_data
    *coverage_digest_gets,
TypeError: MultiGet() takes from 1 to 10 positional arguments but 14 were given

./pants --tag='-integration_tests' test  --print-stacktrace  10.01s user 3.58s system 124% cpu 10.910 total
h
Oh @happy-kitchen-89482 Paritosh has the same issue as Brett iiuc with the ./pants script and Py39
Re this new error, I know exactly what it is. Will cherry-pick a fix. Thanks for the report!
h
Did the fix in that issue work?
h
Let's see: @orange-beach-75711 could you please try the advice in https://github.com/pantsbuild/pants/issues/12114?
o
yup, that seems to have fixed the issue! Thanks guys!
🙌 1
@hundreds-father-404 @happy-kitchen-89482 - weirdly the issue reappeared this morning.
Copy code
09:38:04.73 [ERROR] Exception caught: (pants.engine.internals.scheduler.ExecutionError)
  File "/Users/paritosh/.cache/pants/setup/bootstrap-Darwin-x86_64/2.6.0.dev0_py37/lib/python3.7/site-packages/pants/bin/local_pants_runner.py", line 234, in _run_inner
    return self._perform_run(goals)
  File "/Users/paritosh/.cache/pants/setup/bootstrap-Darwin-x86_64/2.6.0.dev0_py37/lib/python3.7/site-packages/pants/bin/local_pants_runner.py", line 173, in _perform_run
    return self._perform_run_body(goals, poll=False)
  File "/Users/paritosh/.cache/pants/setup/bootstrap-Darwin-x86_64/2.6.0.dev0_py37/lib/python3.7/site-packages/pants/bin/local_pants_runner.py", line 195, in _perform_run_body
    poll_delay=(0.1 if poll else None),
  File "/Users/paritosh/.cache/pants/setup/bootstrap-Darwin-x86_64/2.6.0.dev0_py37/lib/python3.7/site-packages/pants/init/engine_initializer.py", line 136, in run_goal_rules
    goal_product, params, poll=poll, poll_delay=poll_delay
  File "/Users/paritosh/.cache/pants/setup/bootstrap-Darwin-x86_64/2.6.0.dev0_py37/lib/python3.7/site-packages/pants/engine/internals/scheduler.py", line 530, in run_goal_rule
    self._raise_on_error([t for _, t in throws])
  File "/Users/paritosh/.cache/pants/setup/bootstrap-Darwin-x86_64/2.6.0.dev0_py37/lib/python3.7/site-packages/pants/engine/internals/scheduler.py", line 500, in _raise_on_error
    wrapped_exceptions=tuple(t.exc for t in throws),

Exception message: 1 Exception encountered:

  TypeError: MultiGet() takes from 1 to 10 positional arguments but 116 were given


(Use --print-stacktrace to see more error details.)
./pants --tag='-integration_tests' test ::  434.55s user 119.02s system 580% cpu 1:35.33 total
My pants & pants.toml are here - https://gitlab.com/-/snippets/2127757 Note that the pants file has this diff applied and contains the following change that I had added to reorder python versions 3.7-3.9 -
Copy code
# (25th May 2021) Changed the ordering of the versions based on the suggestion here -
    # <https://pantsbuild.slack.com/archives/C046T6T9U/p1621589507090400?thread_ts=1620995740.080000&cid=C046T6T9U>
    supported_python_versions_decimal=('3.7' '3.8' '3.9')
    supported_python_versions_int=('37' '38' '39')
    supported_message='3.7, 3.8, or 3.9'
e
Ah - my bad. I think this just got lost in the shuffle above since two issues were being discussed and handled - the 
pants
script issue and the coverage issue: https://github.com/pantsbuild/pants/issues/12143
👍 1
o
oh got it! Will keep an eye on this issue. Thanks John!
h
Ah yeah. Thanks John! I'll do a 2.5 release today with that cherry picked
o
@hundreds-father-404 Thanks Eric!
@hundreds-father-404 - was this released last week?
h
Oh yeah, I forgot to announce it. 2.5.1rc1
4 Views
Open in Slack
PreviousNext
Powered by