Hi, I have a question about using

xdist

. It does not appear possible to set

xdist_concurrency="auto"

in a pytest target, eg:

python_tests(
    name="my_tests",
    sources=["test_*.py"],
    resolve="main",
    xdist_concurrency="auto",
)

However if you remove

xdist_concurrency

then you can run this successfully:

pants test test_this: -- -n auto

I wanted to confirm whether or not setting

xdist_concurrency="auto"

was possible or not.

Looks like that target field is of

int

type, so yeah,

auto

will not be accepted. However it also looks like “If pytest-xdist is enabled and this field is unset, Pants will attempt to derive the concurrency for test sources by counting the number of tests in each file.”

Ok yes that is what I understood from the documentation. However I found that if I don’t set it explicitly, a parameterized unit-test ran on a single worker, but if I set it explicitly in the

target

, the parallelization is invoked. I managed to work around this by assigning

nproc

to an environmental variable

PANTS_XDIST_CONCURRENCY

and and then using that in the BUILD file:

xdist_concurrency=int(env("PANTS_XDIST_CONCURRENCY", 20)),

Ah nice. Just checking - did you set

[pytest].xdist_enabled = True

?

yep!

For future readers, this is how the number of tests are counted: https://github.com/pantsbuild/pants/blob/977112d372e67070f7b47e3d572bb8736654dc9e/src/python/pants/backend/python/goals/pytest_runner.py#L230

_TEST_PATTERN = re.compile(b"def\\s+test_")

def _count_pytest_tests(contents: DigestContents) -> int:
    return sum(len(_TEST_PATTERN.findall(file.content)) for file in contents)

So parameterized tests are counted incorrectly.

Thanks for the update