Hi, I have a question about using `xdist`. It does...
# general
a
Hi, I have a question about using
xdist
. It does not appear possible to set
xdist_concurrency="auto"
in a pytest target, eg:
Copy code
python_tests(
    name="my_tests",
    sources=["test_*.py"],
    resolve="main",
    xdist_concurrency="auto",
)
However if you remove
xdist_concurrency
then you can run this successfully:
Copy code
pants test test_this: -- -n auto
I wanted to confirm whether or not setting
xdist_concurrency="auto"
was possible or not.
h
Looks like that target field is of
int
type, so yeah,
auto
will not be accepted. However it also looks like “If pytest-xdist is enabled and this field is unset, Pants will attempt to derive the concurrency for test sources by counting the number of tests in each file.”
So maybe that is helpful?
You generally want to let Pants make concurrency decisions rather than delegating them to underlying tools, because Pants is already running multiple processes that will confound auto-concurrency detectors in just one of them
a
Ok yes that is what I understood from the documentation. However I found that if I don’t set it explicitly, a parameterized unit-test ran on a single worker, but if I set it explicitly in the
target
, the parallelization is invoked. I managed to work around this by assigning
nproc
to an environmental variable
PANTS_XDIST_CONCURRENCY
and and then using that in the BUILD file:
Copy code
xdist_concurrency=int(env("PANTS_XDIST_CONCURRENCY", 20)),
h
Ah nice. Just checking - did you set
[pytest].xdist_enabled = True
?
a
yep!