https://pantsbuild.org/ logo
#general
Title
# general
c

cold-soccer-63228

07/25/2022, 7:31 PM
Hey @witty-crayon-22786 / @bitter-ability-32190, I saw that https://github.com/pantsbuild/pants/issues/15026 had the good first issue tag. Since this is a feature that I’m interested in, I was wondering if either of you had any suggestions on how I could get started with investigating how to do this
b

bitter-ability-32190

07/25/2022, 7:46 PM
Admittedly, I'm not familiar with
pytest-xdist
. Generally though I'd personally cargo cult the wiring needed to make this happen. On the Pants side, in the relevant
pytest
rules, you can cargo-cult the
black
rule code for "concurrency_available". https://github.com/pantsbuild/pants/blob/511691657d00827e5e44ab5118688b99c221d744/src/python/pants/backend/python/lint/black/rules.py#L80
So you only really need to figure out the
pytest-xdist
side of things. (What the value of
concurrency_available
should be (line 93 of that file)) and how that value is turned on/off/populated
c

cold-soccer-63228

07/25/2022, 8:03 PM
cargo cult
I’m not familiar with this term. What does this mean?
b

bitter-ability-32190

07/25/2022, 8:04 PM
Another phrase might be "monkey see, monkey do". Essentially just copying what others do without thinking too hard about it
😂 1
It has a real-world definition the software-y usage stems from: https://en.wikipedia.org/wiki/Cargo_cult
w

witty-crayon-22786

07/26/2022, 4:53 PM
thanks @cold-soccer-63228: i can sketch out a suggested implementation on the ticket
c

cold-soccer-63228

07/26/2022, 5:06 PM
@witty-crayon-22786 that’d be awesome! Thank you both!
c

cold-soccer-63228

07/27/2022, 4:41 PM
So, trying to replace some setup_pytest_for_target code with the following:
Copy code
process = await Get(
    Process,
    VenvPexProcess(
        pytest_runner_pex,
        argv=(
            *request.prepend_argv,
            "-n",
            "{pants_concurrency}",
            *pytest.args,
            *coverage_args,
            *field_set_source_files.files,
        ),
        concurrency_available=4,
        ...
    ),
)
I’m receiving errors of the following format.
Copy code
12:34:46.18 [ERROR] Completed: Run Pytest - path/to/test.py:target failed (exit code 4).
ERROR: usage: pex [options] [file_or_dir] [file_or_dir] [...]
pex: error: unrecognized arguments: -n
Taking a step back and looking at PyTest.args, I’m wondering: is this error happening because I’ve tried passing
-n {pants_concurrency}
to
pex
, as opposed to passing it to
pytest
? Would the correct way be for me to somehow pass these arguments via
*pytest.args
? If so, how would I go about injecting
-n {pants_concurrency}
into the
*pytest.args
?
w

witty-crayon-22786

07/27/2022, 4:59 PM
hm…. that is surprising, given that the very next line is
pytest.args
1
oh, actually. it’s possible that that error message is misleading. the “pex” there might just
argv[0]
, and you’re instead getting that error from
pytest
itself…. because: i missed a step in the instructions. you’ll also need to include the
python-xdist
plugin in the default extra requirements, here: https://github.com/pantsbuild/pants/blob/258ab2602a05b1450540d8d3d89f51407aa416fa/src/python/pants/backend/python/subsystems/pytest.py#L75
the
-n
flag is added by
python-xdist
c

cold-soccer-63228

07/27/2022, 5:04 PM
Ahhhh, I see. Without the extras plugin,
pytest
doesn’t have that
-n
argument. Thanks so much for explaining this! (And feel free to ignore my messages and answer when you’re on vacation 🙂)
Hmm, seeing a different error now 😞
Copy code
RuntimeError: Unexpectedly no active workers available
Will probably just drop this for now and work with a workaround instead
Thanks for all of your help @witty-crayon-22786 and @bitter-ability-32190!
w

witty-crayon-22786

07/27/2022, 5:32 PM
looks like an error in the xdist config maybe…? https://github.com/pytest-dev/pytest-xdist/issues/435
maybe
-n
has more semantics than i was aware of.
c

cold-soccer-63228

07/28/2022, 1:09 AM
Yeah, seems strange. I wonder if perhaps the workers that
pytest-xdist
need are taken up by the the cores that are responsible for Pants’ built-in parallelism? Could that be possible?
w

witty-crayon-22786

07/28/2022, 1:36 AM
Shouldn't be... reserving cores isn't really a thing afaik.
It seems more likely that PEX affects how xdist spawns workers. Was there any additional output with the error?
c

cold-soccer-63228

07/28/2022, 1:47 AM
Yeah, here’s one example of a test failure.
Copy code
21:46:42.69 [ERROR] Completed: Run Pytest - src/python/pants/backend/codegen/soap/tailor_test.py:tests failed (exit code 3).
============================= test session starts ==============================
gw0 I
INTERNALERROR> def worker_internal_error(self, node, formatted_error):
INTERNALERROR>         """
INTERNALERROR>         pytest_internalerror() was called on the worker.
INTERNALERROR>
INTERNALERROR>         pytest_internalerror() arguments are an excinfo and an excrepr, which can't
INTERNALERROR>         be serialized, so we go with a poor man's solution of raising an exception
INTERNALERROR>         here ourselves using the formatted message.
INTERNALERROR>         """
INTERNALERROR>         self._active_nodes.remove(node)
INTERNALERROR>         try:
INTERNALERROR> >           assert False, formatted_error
INTERNALERROR> E           AssertionError: Traceback (most recent call last):
INTERNALERROR> E             File "/Users/hughhan/.cache/pants/named_caches/pex_root/venvs/50028e433a7454962e443452efe7d7385c9e75e2/128c89477b00adb22c6d0dd28e14bbaa7ea24512/lib/python3.7/site-packages/_pytest/main.py", line 266, in wrap_session
INTERNALERROR> E               config.hook.pytest_sessionstart(session=session)
INTERNALERROR> E             File "/Users/hughhan/.cache/pants/named_caches/pex_root/venvs/50028e433a7454962e443452efe7d7385c9e75e2/128c89477b00adb22c6d0dd28e14bbaa7ea24512/lib/python3.7/site-packages/pluggy/_hooks.py", line 265, in __call__
INTERNALERROR> E               return self._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult)
INTERNALERROR> E             File "/Users/hughhan/.cache/pants/named_caches/pex_root/venvs/50028e433a7454962e443452efe7d7385c9e75e2/128c89477b00adb22c6d0dd28e14bbaa7ea24512/lib/python3.7/site-packages/pluggy/_manager.py", line 80, in _hookexec
INTERNALERROR> E               return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
INTERNALERROR> E             File "/Users/hughhan/.cache/pants/named_caches/pex_root/venvs/50028e433a7454962e443452efe7d7385c9e75e2/128c89477b00adb22c6d0dd28e14bbaa7ea24512/lib/python3.7/site-packages/pluggy/_callers.py", line 60, in _multicall
INTERNALERROR> E               return outcome.get_result()
INTERNALERROR> E             File "/Users/hughhan/.cache/pants/named_caches/pex_root/venvs/50028e433a7454962e443452efe7d7385c9e75e2/128c89477b00adb22c6d0dd28e14bbaa7ea24512/lib/python3.7/site-packages/pluggy/_result.py", line 60, in get_result
INTERNALERROR> E               raise ex[1].with_traceback(ex[2])
INTERNALERROR> E             File "/Users/hughhan/.cache/pants/named_caches/pex_root/venvs/50028e433a7454962e443452efe7d7385c9e75e2/128c89477b00adb22c6d0dd28e14bbaa7ea24512/lib/python3.7/site-packages/pluggy/_callers.py", line 39, in _multicall
INTERNALERROR> E               res = hook_impl.function(*args)
INTERNALERROR> E             File "/private/var/folders/yw/gqb55c1j1y354lwwlksln1mw0000gp/T/pants-sandbox-MfnfNz/src/python/pants/conftest.py", line 37, in pytest_sessionstart
INTERNALERROR> E               f"In order for `pants` to be a namespace package, {namespace_init_path} must not "
INTERNALERROR> E           Exception: In order for `pants` to be a namespace package, src/python/pants/__init__.py must not exist on disk. See the explanation in /private/var/folders/yw/gqb55c1j1y354lwwlksln1mw0000gp/T/pants-sandbox-MfnfNz/src/python/pants/conftest.py.
INTERNALERROR>
INTERNALERROR> /Users/hughhan/.cache/pants/named_caches/pex_root/venvs/50028e433a7454962e443452efe7d7385c9e75e2/128c89477b00adb22c6d0dd28e14bbaa7ea24512/lib/python3.7/site-packages/xdist/dsession.py:192: AssertionError
INTERNALERROR> Traceback (most recent call last):
INTERNALERROR>   File "/Users/hughhan/.cache/pants/named_caches/pex_root/venvs/50028e433a7454962e443452efe7d7385c9e75e2/128c89477b00adb22c6d0dd28e14bbaa7ea24512/lib/python3.7/site-packages/_pytest/main.py", line 268, in wrap_session
INTERNALERROR>     session.exitstatus = doit(config, session) or 0
INTERNALERROR>   File "/Users/hughhan/.cache/pants/named_caches/pex_root/venvs/50028e433a7454962e443452efe7d7385c9e75e2/128c89477b00adb22c6d0dd28e14bbaa7ea24512/lib/python3.7/site-packages/_pytest/main.py", line 322, in _main
INTERNALERROR>     config.hook.pytest_runtestloop(session=session)
INTERNALERROR>   File "/Users/hughhan/.cache/pants/named_caches/pex_root/venvs/50028e433a7454962e443452efe7d7385c9e75e2/128c89477b00adb22c6d0dd28e14bbaa7ea24512/lib/python3.7/site-packages/pluggy/_hooks.py", line 265, in __call__
INTERNALERROR>     return self._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult)
INTERNALERROR>   File "/Users/hughhan/.cache/pants/named_caches/pex_root/venvs/50028e433a7454962e443452efe7d7385c9e75e2/128c89477b00adb22c6d0dd28e14bbaa7ea24512/lib/python3.7/site-packages/pluggy/_manager.py", line 80, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
INTERNALERROR>   File "/Users/hughhan/.cache/pants/named_caches/pex_root/venvs/50028e433a7454962e443452efe7d7385c9e75e2/128c89477b00adb22c6d0dd28e14bbaa7ea24512/lib/python3.7/site-packages/pluggy/_callers.py", line 60, in _multicall
INTERNALERROR>     return outcome.get_result()
INTERNALERROR>   File "/Users/hughhan/.cache/pants/named_caches/pex_root/venvs/50028e433a7454962e443452efe7d7385c9e75e2/128c89477b00adb22c6d0dd28e14bbaa7ea24512/lib/python3.7/site-packages/pluggy/_result.py", line 60, in get_result
INTERNALERROR>     raise ex[1].with_traceback(ex[2])
INTERNALERROR>   File "/Users/hughhan/.cache/pants/named_caches/pex_root/venvs/50028e433a7454962e443452efe7d7385c9e75e2/128c89477b00adb22c6d0dd28e14bbaa7ea24512/lib/python3.7/site-packages/pluggy/_callers.py", line 39, in _multicall
INTERNALERROR>     res = hook_impl.function(*args)
INTERNALERROR>   File "/Users/hughhan/.cache/pants/named_caches/pex_root/venvs/50028e433a7454962e443452efe7d7385c9e75e2/128c89477b00adb22c6d0dd28e14bbaa7ea24512/lib/python3.7/site-packages/xdist/dsession.py", line 117, in pytest_runtestloop
INTERNALERROR>     self.loop_once()
INTERNALERROR>   File "/Users/hughhan/.cache/pants/named_caches/pex_root/venvs/50028e433a7454962e443452efe7d7385c9e75e2/128c89477b00adb22c6d0dd28e14bbaa7ea24512/lib/python3.7/site-packages/xdist/dsession.py", line 129, in loop_once
INTERNALERROR>     raise RuntimeError("Unexpectedly no active workers available")
INTERNALERROR> RuntimeError: Unexpectedly no active workers available
w

witty-crayon-22786

07/28/2022, 1:50 AM
There is an another error stashed in the middle there: "In order for Pants to" ... so that's actually something specific to the Pants repo
I'll take a look tomorrow
c

cold-soccer-63228

07/28/2022, 3:31 PM
Okay, thanks @witty-crayon-22786! I appreciate it
w

witty-crayon-22786

07/28/2022, 3:31 PM
so, yea: that error comes from here: https://github.com/pantsbuild/pants/blob/699087bad9110dceca3fdcd6650cb7f2caaa0f8a/src/python/pants/conftest.py#L34-L40 … which is specific to the
pantsbuild/pants
repo. but all that that means in this case is that
pantsbuild/pants
itself cannot use this feature yet… which is a good demonstration of why the feature needs to be disabled by default
so would recommend looping back around to step 1 from the ticket and adding the option to disable this by default
c

cold-soccer-63228

07/28/2022, 3:34 PM
Thanks for sharing that. I’m a bit confused. If the error comes from enabling the option, it means that it can’t be implemented right now, right? Does disabling by default, and then enabling somewhere later down the line cause this error to not be thrown for some reason?
w

witty-crayon-22786

07/28/2022, 3:36 PM
If the error comes from enabling the option, it means that it can’t be implemented right now, right?
no: it can be implemented.
pantsbuild/pants
uses
pants
to build itself. all the error means is that when testing itself,
pants
cannot use the option. but an integration test (see the link at the end of the ticket comment) will create a test harness made up anonymous files, and so it can.
5 Views