So presumably you're running Pants with some command line flags. If you could run including `-ldebug`in those flags as well as using

test --force

and include your full command line and full output, that might be enlightening.

The `-ldebug`will give more log info and the

test --force

will force the tests to re-run even if they have cached success.

So, high level, Pants runs nothing in your code tree. It builds a sandbox in a /tmp dir with just what's needed (including tools) and runs from there.

This option keeps the sandboxes from getting cleaned up so you can inspect them.

There will be a `__run.sh`script at the root of each sandbox that simulates exactly what Pants actually runs using its Rust engine code.

It can be useful to poke around the sandbox to see file layout and examine and run

__run.sh

to debug.

And, correcting my suggestion this has to do with coverage collection - I read wrongly - it does not: https://github.com/pantsbuild/pants/blob/a69b8f0ef4e14451dbf7f39fd23adaf3f2700535/src/python/pants/backend/python/goals/pytest_runner.py#L370-L376

Do you use remote caching?

Yeah, so without a sandbox from the issue reproducing this is hard to diagnose. When Pants runs pytest it add command line flag to have it emit the JUnit XML and then snapshots all the files output during the pytest run. For the snapshot to exist (there would have been an ugly backtrace if it did not), but the JUnit XML file not to be in the snapshot is unexpected.

From my debug advice above, the `-ldebug`would have proved / disproved the pytest JUnit XML flags were being passed and the sandbox would have allowed further diagnosis.

We would be looking for

--junitxml-...

from here: https://github.com/pantsbuild/pants/blob/a69b8f0ef4e14451dbf7f39fd23adaf3f2700535/src/python/pants/backend/python/goals/pytest_runner.py#L286-L292

I just want to step back and confirm this is flaky generally. Is that true? I.E.: Before adding the extra flags to debug all this, would this happen sporadically or all the time?

Ok, is there a doc on how to use the

sandboxe

and

__run.sh

to troubleshoot?

If you want to share the

__run.sh

or even the full sandbox tarball I'm happy to take a look.

I don't see

--junitxml-...

No, I'm not running in debug mode...

My Nautobot tests set the

PYTEST_ADDOPTS

extra_env_vars

in the

python_tests

target. It seems to override the content of this variable set by pants...

This variable is used by Pants to pass the options for junitxml.

Working report:

❯ cat __run.sh

#!/bin/bash

# This command line should execute the same process as pants did internally.

export PANTS_EXECUTION_SLOT=1 PEX_EXTRA_SYS_PATH=$'.:src/python' PYTEST_ADDOPTS=$'--color=yes --junitxml=tests.python.nornir_tests.nornir_greet_tests.test_greeting_tasks.py@tests.xml -o junit_family=xunit2'

cd /tmp/pants-sandbox-vjs2JG

./pytest_runner.pex_pex_shim.sh -c build-support/python/tools/pytest.ini -vv --no-header -s $'--cov-report=' $'--cov-config=build-support/python/tools/.coveragerc' $'--cov=.' $'--cov=src/python' tests/python/nornir_tests/nornir_greet_tests/test_greeting_tasks.py

Not working:

❯ cat __run.sh

#!/bin/bash

# This command line should execute the same process as pants did internally.

export DJANGO_SETTINGS_MODULE=nautobot.core.tests.nautobot_config NAUTOBOT_CONFIG=./src/python/netops_nautobot/nautobot_config.py NAUTOBOT_REDIS_PASSWORD=decinablesprewad PANTS_EXECUTION_SLOT=7 PEX_EXTRA_SYS_PATH=$'.:src/python' PYTEST_ADDOPTS=$'-p pytest_nautobot --reuse-db'

cd /tmp/pants-sandbox-SHdrIJ

./pytest_runner.pex_pex_shim.sh -c build-support/python/tools/pytest.ini -vv --no-header -s $'--cov-report=' $'--cov-config=build-support/python/tools/.coveragerc' $'--cov=.' $'--cov=src/python' tests/python/nautobot_tests/nautobot_animal_sounds_tests/test_api.py

My BUILD:

python_tests(
    name="tests",
    dependencies=[
        "3rdparty/python:nautobot-plugin-reqs",
        "3rdparty/python:reqs#pytest-django",
        "src/python/nautobot_animal_sounds",
        "src/python/netops_nautobot",
        "src/python/pytest_nautobot",
    ],
    extra_env_vars=[
        "DJANGO_SETTINGS_MODULE=nautobot.core.tests.nautobot_config",
        "NAUTOBOT_CONFIG=./src/python/netops_nautobot/nautobot_config.py",
        "NAUTOBOT_REDIS_PASSWORD=decinablesprewad",
        "PYTEST_ADDOPTS=-p pytest_nautobot --reuse-db",
    ],
    sources=["**/test_*.py"],
)

Definitely straight up bug: https://github.com/pantsbuild/pants/blob/a69b8f0ef4e14451dbf7f39fd23adaf3f2700535/src/python/pants/backend/python/goals/pytest_runner.py#L315-L322 Your BUILD env vars are that last splat which overwrites our internal use. We do not try to merge.

I think merging is the right thing to do, but this needs experimentation to see what happens when you specify conflicting pytest options via the merge, etc.

@rhythmic-glass-66959 I assume only applying these pytest options for these tests is very intentional and that's why their not configured globally via https://www.pantsbuild.org/docs/reference-pytest#section-args

I think the explicit

"3rdparty/python:nautobot-plugin-reqs",

dep pretty much answers my question.

Thank you for your help, very appreciated!