https://pantsbuild.org/ logo
#general
Title
# general
c

crooked-tent-69497

02/06/2020, 5:25 PM
within the test ? I’ve tried two ways -
psutil.Popen('./pants run {target}', shell=True)
and
psutil.Popen([os.getcwd() + '/pants', 'run', '{target}'])
h

hundreds-father-404

02/06/2020, 5:26 PM
For the sake of debugging, it might help to start by trying to get it working directly on the command line, rather than using
psutil.Popen
. Once you get that working, then switch to
psutil.Popen
Here,
psutil.Popen
is a little distracting from the underlying issue
c

crooked-tent-69497

02/06/2020, 5:27 PM
the binary runs fine, are you suggesting i try to run coverage on the executable manually ?
d

dry-analyst-73584

02/06/2020, 5:27 PM
To clarify: is the
Copy code
Exception: ModuleNotFoundError("No module named '__pants_backend_python_tasks_pytest_prep_coverage_plugin__'",)
error happening within your subprocess, or just during the test which uses the subprocess?
c

crooked-tent-69497

02/06/2020, 5:29 PM
it’s happening in the test, I believe I’ve tried doing what pytest-cov does manually (as far as I understand, set
$COVERAGE_PROCESS_START
and then in my initialization routine invoke
coverage.process_startup()
), but I can try that again and see what happens
h

hundreds-father-404

02/06/2020, 5:30 PM
are you suggesting i try to run coverage on the executable manually ?
I mean try running
./pants run target
directly on the command line. Is that working? If so, but it doesn’t work with
subprocess.Popen
, then we know the issue has something to do with
subprocess.Popen
rather than an underlying issue with the call to Pants
c

crooked-tent-69497

02/06/2020, 5:31 PM
the target standalone runs without any issues, the issue is when pytest-cov tries to start running coverage on the subprocess, which is what i could try explicitly doing myself
d

dry-analyst-73584

02/06/2020, 5:31 PM
You should not need to call coverage directly.
c

crooked-tent-69497

02/06/2020, 5:32 PM
that’s true if the process is started within a situation where coverage is already running, if i just
./pants run
, I believe coverage doesn’t get invoked
d

dry-analyst-73584

02/06/2020, 5:32 PM
Are you trying to get coverage to run when running
./pants run target
inside of a test?
I would not expect that to work.
c

crooked-tent-69497

02/06/2020, 5:33 PM
sort of, to clarify, i’m running an integration test, and need to run the code being profiled in a separate process outside the test itself
according to pytest-cov and coverage itself, as long as the environment is carried over from process to process, and coverage is available in the environment of the target, it “should work”
so i guess that leaves two questions, is the environment carried over from the environment of
./pants run
to the executable it creates ?
and secondly, is there any reason to believe the target environment shouldn’t include the coverage module
d

dry-analyst-73584

02/06/2020, 5:35 PM
The target environment likely does not include the coverage module as it is not a test.
c

crooked-tent-69497

02/06/2020, 5:35 PM
to be clear, running the test in this fashion and running coverage on it does work correctly if I use pytest
I tried setting it as a dependency in the BUILD file
i think that’s when i ended up with version mismatch issues
how do i set what version of coverage/pytest-cov is used? I have coverage and pytest-cov as 3rdparty requirements, and I’m also pinning their versions in pants.ini [plugins]
sorry if these are all bizarre questions, been staring at this for too long and on little sleep now!
d

dry-analyst-73584

02/06/2020, 5:39 PM
It's all good, I'm just trying to understand what you're doing and how we can help.
I would not expect pants' custom coverage plugin to be available when you run
./pants run target
, which explains the
"No module named '__pants_backend_python_tasks_pytest_prep_coverage_plugin__'"
error
For the version matching issue, I know how to set that in V2 (in a PR that hasn't landed yet) but I'm trying to figure out how to do it in V1. I believe it may be dictated by the version of pytest you're using.
c

crooked-tent-69497

02/06/2020, 5:44 PM
since this an inherited codebase i’m working on, i’ve noticed there’s some settings in pants.ini that i don’t quite understand and can’t find docs for -
Copy code
[GLOBAL]
pants_version: 1.21.0

plugins: +[
    "pantsbuild.pants.contrib.node==%(pants_version)s",
    "pantsbuild.pants.contrib.python.checks==%(pants_version)s",
    "pantsbuild.pants.contrib.mypy==%(pants_version)s",
    "pylint==1.9.3",
  ]
I’ve tried setting packages here (had run into a no module coverage issue, and if i include it here it seems to work) and here -
Copy code
[DEFAULT]
[pytest]
requirements: pytest==3.6.4
cov_requirements: pytest-cov==2.5.1
i’ll admit these are out of date packages, but i’ve also tried updating pants, pytest and coverage separately and still have the issue of coverage not being initialized properly on the subprocess
so here’s another question, what version of pytest, coverage etc. is set in the “default pants environment”, aka, can i mirror those package versions within a 3rd party dependency and just manually include/invoke it in the subprocess
are those the pants.ini settings i posted above ?
d

dry-analyst-73584

02/06/2020, 5:51 PM
Hah
When you run this with pytest and it works, are you still executing pants in a subprocess?
c

crooked-tent-69497

02/06/2020, 5:55 PM
when it works with pytest, i’m not using pants in the subprocess.
when i tried with pytest -> pants subprocess it timed out
but i can extend that possibly and see what happens, but i think the “no module named coverage” error happens then also
yeah, so it looks like the most reliable error is that the custom pants wrapper around coverage is not available in the environment of the executable
hence this error -
pytest-cov: Failed to setup subprocess coverage. Environ: {'COV_CORE_SOURCE': 'modules:cc:cc.imagen_cc.devices.middleware:cc.imagen_cc.devices.lib:<http://cc.imagen_cc.devices.cc|cc.imagen_cc.devices.cc>', 'COV_CORE_CONFIG': '/Users/sameermoidu/code/mordor/.pants.d/test/pytest/cc.imagen_test.devices.middleware.test_e2e/tmp9bgkk2hp', 'COV_CORE_DATAFILE': '/Users/sameermoidu/code/mordor/.coverage'} Exception: ModuleNotFoundError("No module named '__pants_backend_python_tasks_pytest_prep_coverage_plugin__'",)
d

dry-analyst-73584

02/06/2020, 6:02 PM
I have some answers, I think. The pants.ini values you're seeing should be the ones that are being used. You can also pass these on the command line: with
--pytest-version=<str>
and
--pytest-pytest-plugins=<str>
BUT: these things are only available when you are running
./pants test
they are not available in the environment when you
./pants run
The custom pants coverage plugin is loaded only when pants sets up a pytest environment
c

crooked-tent-69497

02/06/2020, 6:04 PM
makes sense generally, so if i use a pex or something i’ll run into the same issue
d

dry-analyst-73584

02/06/2020, 6:04 PM
I believe so.
c

crooked-tent-69497

02/06/2020, 6:04 PM
ok, i was trying to figure out how the pants integration tests run
but it seems that they may not use coverage ?
or at least i can’t figure out if they do/how they do
d

dry-analyst-73584

02/06/2020, 6:05 PM
I'm not sure that what you're attempting is currently possible with pants. If you can run your binary with something other than
./pants run
from within your integration test you may get the behavior you're looking for.
c

crooked-tent-69497

02/06/2020, 6:05 PM
hmm, ok
h

happy-kitchen-89482

02/06/2020, 6:21 PM
To clarify for myself, it sounds like you have an integration test that happens to fork out to
./pants run
and you want to run coverage on that
./pants run
invocation?
c

crooked-tent-69497

02/06/2020, 6:22 PM
yes, as supported by pytest-cov and coverage
h

happy-kitchen-89482

02/06/2020, 6:22 PM
I see, so you're actually not directly using the coverage support in
./pants test
but setting up coverage manually yourself in your
./pants run
invocation
c

crooked-tent-69497

02/06/2020, 6:23 PM
well, i’m setting the options in ./pants test, i just want that to include the data from the subprocess
h

happy-kitchen-89482

02/06/2020, 6:23 PM
But you want the resulting coverage results to be merged back in with pants's regular
./pants test
coverage?
c

crooked-tent-69497

02/06/2020, 6:23 PM
yeah, basically
h

happy-kitchen-89482

02/06/2020, 6:23 PM
Hmmm
Test runs are very isolated, by design, so I don't think you can make this happen with env vars.
👍 1
c

crooked-tent-69497

02/06/2020, 6:24 PM
yeah, pytest-cov does some crazy crap to get this to work, but i think pants does somethings to further isolate than standard python
h

happy-kitchen-89482

02/06/2020, 6:24 PM
And are you able to get coverage data when you run your process, but you can't make
./pants test
pick it up and merge it into the test's coverage? Is that an accurate representation of the problem?
c

crooked-tent-69497

02/06/2020, 6:25 PM
yes, something is generated, i haven’t taken a look at what exactly to be honest
h

happy-kitchen-89482

02/06/2020, 6:25 PM
I.e., your forked process does in fact produce a coverage file, but Pants doesn't do anything with it, and you want it to?
We could add support for that in v2, I suspect
c

crooked-tent-69497

02/06/2020, 6:25 PM
i’ve mostly been focused on trying to get the merged thing to work since it seems easiest
but i could probably create a separate .coverage file and merge myself
h

happy-kitchen-89482

02/06/2020, 6:25 PM
Have your process drop coverage files in some well-known location
c

crooked-tent-69497

02/06/2020, 6:26 PM
but pants doesn’t seem to generate .coverage files, i just see html and xml
or i’m looking in the wrong place ?
h

happy-kitchen-89482

02/06/2020, 6:26 PM
It generates them, but only internally
They are buried in
.pants.d
somewhere
c

crooked-tent-69497

02/06/2020, 6:26 PM
ok
perhaps the best thing to do is try and find those files, do the merge myself with coverage and then use that
first thing is either running the process outside ./pants or somehow getting coverage included in that environment
h

happy-kitchen-89482

02/06/2020, 6:27 PM
In the short term that's probably right
c

crooked-tent-69497

02/06/2020, 6:27 PM
ok, that’s a good starting point, thanks
h

happy-kitchen-89482

02/06/2020, 6:30 PM
But this is great feedback, we can probably provide this as a feature in v2
d

dry-analyst-73584

02/06/2020, 6:31 PM
Of note: running
coverage combine
on
.coverage
files will delete the
.coverage
files. I suggest copying them to some other location before doing your merge
👍 1
c

crooked-tent-69497

02/06/2020, 6:31 PM
ok, is there some failsafe way to find the .coverage files ?
aka, scriptable
d

dry-analyst-73584

02/06/2020, 6:32 PM
In v1 I am not sure, sorry 😞
c

crooked-tent-69497

02/06/2020, 8:25 PM
another coverage question … for a non subprocess/crazy scenario, what i’m seeing is that when i run
./pants test {target} — --cov=targetdir --
pretty much nothing is recorded (init files are 100% and everything else is more or less 0) Then, I switched to trying to supply coverage targets via --test-pytest-coverage=modules:, and then frequently modules that I know for a fact are included don’t get any hits. Any guidance on the best approach here ?
I’ve also noticed that if I have some modules that should be detected, depending on how i specify the --coverage param, they may or may not be properly captured. for example
--coverage=modules:path.inner.module_1,path.inner.module_2
vs
--coverage=modules:path.inner
produces different results
4 Views