Hey @witty-crayon-22786 / @bitter-ability-32190, I saw that https://github.com/pantsbuild/pants/issues/15026 had the good first issue tag. Since this is a feature that I’m interested in, I was wondering if either of you had any suggestions on how I could get started with investigating how to do this

Admittedly, I'm not familiar with

pytest-xdist

. Generally though I'd personally cargo cult the wiring needed to make this happen. On the Pants side, in the relevant

pytest

rules, you can cargo-cult the

black

rule code for "concurrency_available". https://github.com/pantsbuild/pants/blob/511691657d00827e5e44ab5118688b99c221d744/src/python/pants/backend/python/lint/black/rules.py#L80

cargo cult

I’m not familiar with this term. What does this mean?

Another phrase might be "monkey see, monkey do". Essentially just copying what others do without thinking too hard about it

thanks @cold-soccer-63228: i can sketch out a suggested implementation on the ticket

@witty-crayon-22786 that’d be awesome! Thank you both!

So, trying to replace some setup_pytest_for_target code with the following:

process = await Get(
    Process,
    VenvPexProcess(
        pytest_runner_pex,
        argv=(
            *request.prepend_argv,
            "-n",
            "{pants_concurrency}",
            *pytest.args,
            *coverage_args,
            *field_set_source_files.files,
        ),
        concurrency_available=4,
        ...
    ),
)

I’m receiving errors of the following format.

12:34:46.18 [ERROR] Completed: Run Pytest - path/to/test.py:target failed (exit code 4).
ERROR: usage: pex [options] [file_or_dir] [file_or_dir] [...]
pex: error: unrecognized arguments: -n

Taking a step back and looking at PyTest.args, I’m wondering: is this error happening because I’ve tried passing

-n {pants_concurrency}

to

pex

, as opposed to passing it to

pytest

? Would the correct way be for me to somehow pass these arguments via

*pytest.args

? If so, how would I go about injecting

-n {pants_concurrency}

into the

*pytest.args

?

hm…. that is surprising, given that the very next line is

pytest.args

Ahhhh, I see. Without the extras plugin,

pytest

doesn’t have that

-n

argument. Thanks so much for explaining this! (And feel free to ignore my messages and answer when you’re on vacation 🙂)

looks like an error in the xdist config maybe…? https://github.com/pytest-dev/pytest-xdist/issues/435

Yeah, seems strange. I wonder if perhaps the workers that

pytest-xdist

need are taken up by the the cores that are responsible for Pants’ built-in parallelism? Could that be possible?

Shouldn't be... reserving cores isn't really a thing afaik.

Yeah, here’s one example of a test failure.

21:46:42.69 [ERROR] Completed: Run Pytest - src/python/pants/backend/codegen/soap/tailor_test.py:tests failed (exit code 3).
============================= test session starts ==============================
gw0 I
INTERNALERROR> def worker_internal_error(self, node, formatted_error):
INTERNALERROR>         """
INTERNALERROR>         pytest_internalerror() was called on the worker.
INTERNALERROR>
INTERNALERROR>         pytest_internalerror() arguments are an excinfo and an excrepr, which can't
INTERNALERROR>         be serialized, so we go with a poor man's solution of raising an exception
INTERNALERROR>         here ourselves using the formatted message.
INTERNALERROR>         """
INTERNALERROR>         self._active_nodes.remove(node)
INTERNALERROR>         try:
INTERNALERROR> >           assert False, formatted_error
INTERNALERROR> E           AssertionError: Traceback (most recent call last):
INTERNALERROR> E             File "/Users/hughhan/.cache/pants/named_caches/pex_root/venvs/50028e433a7454962e443452efe7d7385c9e75e2/128c89477b00adb22c6d0dd28e14bbaa7ea24512/lib/python3.7/site-packages/_pytest/main.py", line 266, in wrap_session
INTERNALERROR> E               config.hook.pytest_sessionstart(session=session)
INTERNALERROR> E             File "/Users/hughhan/.cache/pants/named_caches/pex_root/venvs/50028e433a7454962e443452efe7d7385c9e75e2/128c89477b00adb22c6d0dd28e14bbaa7ea24512/lib/python3.7/site-packages/pluggy/_hooks.py", line 265, in __call__
INTERNALERROR> E               return self._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult)
INTERNALERROR> E             File "/Users/hughhan/.cache/pants/named_caches/pex_root/venvs/50028e433a7454962e443452efe7d7385c9e75e2/128c89477b00adb22c6d0dd28e14bbaa7ea24512/lib/python3.7/site-packages/pluggy/_manager.py", line 80, in _hookexec
INTERNALERROR> E               return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
INTERNALERROR> E             File "/Users/hughhan/.cache/pants/named_caches/pex_root/venvs/50028e433a7454962e443452efe7d7385c9e75e2/128c89477b00adb22c6d0dd28e14bbaa7ea24512/lib/python3.7/site-packages/pluggy/_callers.py", line 60, in _multicall
INTERNALERROR> E               return outcome.get_result()
INTERNALERROR> E             File "/Users/hughhan/.cache/pants/named_caches/pex_root/venvs/50028e433a7454962e443452efe7d7385c9e75e2/128c89477b00adb22c6d0dd28e14bbaa7ea24512/lib/python3.7/site-packages/pluggy/_result.py", line 60, in get_result
INTERNALERROR> E               raise ex[1].with_traceback(ex[2])
INTERNALERROR> E             File "/Users/hughhan/.cache/pants/named_caches/pex_root/venvs/50028e433a7454962e443452efe7d7385c9e75e2/128c89477b00adb22c6d0dd28e14bbaa7ea24512/lib/python3.7/site-packages/pluggy/_callers.py", line 39, in _multicall
INTERNALERROR> E               res = hook_impl.function(*args)
INTERNALERROR> E             File "/private/var/folders/yw/gqb55c1j1y354lwwlksln1mw0000gp/T/pants-sandbox-MfnfNz/src/python/pants/conftest.py", line 37, in pytest_sessionstart
INTERNALERROR> E               f"In order for `pants` to be a namespace package, {namespace_init_path} must not "
INTERNALERROR> E           Exception: In order for `pants` to be a namespace package, src/python/pants/__init__.py must not exist on disk. See the explanation in /private/var/folders/yw/gqb55c1j1y354lwwlksln1mw0000gp/T/pants-sandbox-MfnfNz/src/python/pants/conftest.py.
INTERNALERROR>
INTERNALERROR> /Users/hughhan/.cache/pants/named_caches/pex_root/venvs/50028e433a7454962e443452efe7d7385c9e75e2/128c89477b00adb22c6d0dd28e14bbaa7ea24512/lib/python3.7/site-packages/xdist/dsession.py:192: AssertionError
INTERNALERROR> Traceback (most recent call last):
INTERNALERROR>   File "/Users/hughhan/.cache/pants/named_caches/pex_root/venvs/50028e433a7454962e443452efe7d7385c9e75e2/128c89477b00adb22c6d0dd28e14bbaa7ea24512/lib/python3.7/site-packages/_pytest/main.py", line 268, in wrap_session
INTERNALERROR>     session.exitstatus = doit(config, session) or 0
INTERNALERROR>   File "/Users/hughhan/.cache/pants/named_caches/pex_root/venvs/50028e433a7454962e443452efe7d7385c9e75e2/128c89477b00adb22c6d0dd28e14bbaa7ea24512/lib/python3.7/site-packages/_pytest/main.py", line 322, in _main
INTERNALERROR>     config.hook.pytest_runtestloop(session=session)
INTERNALERROR>   File "/Users/hughhan/.cache/pants/named_caches/pex_root/venvs/50028e433a7454962e443452efe7d7385c9e75e2/128c89477b00adb22c6d0dd28e14bbaa7ea24512/lib/python3.7/site-packages/pluggy/_hooks.py", line 265, in __call__
INTERNALERROR>     return self._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult)
INTERNALERROR>   File "/Users/hughhan/.cache/pants/named_caches/pex_root/venvs/50028e433a7454962e443452efe7d7385c9e75e2/128c89477b00adb22c6d0dd28e14bbaa7ea24512/lib/python3.7/site-packages/pluggy/_manager.py", line 80, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
INTERNALERROR>   File "/Users/hughhan/.cache/pants/named_caches/pex_root/venvs/50028e433a7454962e443452efe7d7385c9e75e2/128c89477b00adb22c6d0dd28e14bbaa7ea24512/lib/python3.7/site-packages/pluggy/_callers.py", line 60, in _multicall
INTERNALERROR>     return outcome.get_result()
INTERNALERROR>   File "/Users/hughhan/.cache/pants/named_caches/pex_root/venvs/50028e433a7454962e443452efe7d7385c9e75e2/128c89477b00adb22c6d0dd28e14bbaa7ea24512/lib/python3.7/site-packages/pluggy/_result.py", line 60, in get_result
INTERNALERROR>     raise ex[1].with_traceback(ex[2])
INTERNALERROR>   File "/Users/hughhan/.cache/pants/named_caches/pex_root/venvs/50028e433a7454962e443452efe7d7385c9e75e2/128c89477b00adb22c6d0dd28e14bbaa7ea24512/lib/python3.7/site-packages/pluggy/_callers.py", line 39, in _multicall
INTERNALERROR>     res = hook_impl.function(*args)
INTERNALERROR>   File "/Users/hughhan/.cache/pants/named_caches/pex_root/venvs/50028e433a7454962e443452efe7d7385c9e75e2/128c89477b00adb22c6d0dd28e14bbaa7ea24512/lib/python3.7/site-packages/xdist/dsession.py", line 117, in pytest_runtestloop
INTERNALERROR>     self.loop_once()
INTERNALERROR>   File "/Users/hughhan/.cache/pants/named_caches/pex_root/venvs/50028e433a7454962e443452efe7d7385c9e75e2/128c89477b00adb22c6d0dd28e14bbaa7ea24512/lib/python3.7/site-packages/xdist/dsession.py", line 129, in loop_once
INTERNALERROR>     raise RuntimeError("Unexpectedly no active workers available")
INTERNALERROR> RuntimeError: Unexpectedly no active workers available

There is an another error stashed in the middle there: "In order for Pants to" ... so that's actually something specific to the Pants repo

Okay, thanks @witty-crayon-22786! I appreciate it

so, yea: that error comes from here: https://github.com/pantsbuild/pants/blob/699087bad9110dceca3fdcd6650cb7f2caaa0f8a/src/python/pants/conftest.py#L34-L40 … which is specific to the

pantsbuild/pants

repo. but all that that means in this case is that

pantsbuild/pants

itself cannot use this feature yet… which is a good demonstration of why the feature needs to be disabled by default

Thanks for sharing that. I’m a bit confused. If the error comes from enabling the option, it means that it can’t be implemented right now, right? Does disabling by default, and then enabling somewhere later down the line cause this error to not be thrown for some reason?

If the error comes from enabling the option, it means that it can’t be implemented right now, right?

no: it can be implemented.

pantsbuild/pants

uses

pants

to build itself. all the error means is that when testing itself,

pants

cannot use the option. but an integration test (see the link at the end of the ticket comment) will create a test harness made up anonymous files, and so it can.