https://pantsbuild.org/ logo
Join Slack
Powered by
I have a python build that uses pytest and a globa...
# general
n
I have a python build that uses pytest and a global 
conftest.py
in my root directory. That sets up some custom markers and associated commnd line options like 
--run_large
. It was all working just fine but then I upgraded to Python 3.12 which wasn't compatible with the pytest version Pants uses by default so I setup a resolve for the test tool and specified a newer version of pytest like:
Copy code
[pytest]
args = ['--log-cli-level=INFO']
install_from_resolve = "tools"
That is now using the newer version of pytest but it's no longer running my 
conftest.py
even if I add:
Copy code
[python-infer]
conftests = true
I can verify that because (1) I added print statements to things like my 
pytest_addoption
method and (2) the command line arguments like 
--run_large
are no longer recognized. How can I fix it?
h
To start with, run with 
--keep-sandboxes=always
, identify the pytest sandbox and se if 
conftest.py
has been copied into it.
n
Thanks for the advice @happy-kitchen-89482. I tried:
Copy code
$ pants --keep-sandboxes=always test --debug --test-force --test-output=all  ./src/logutil/:: -- --run_large
ERROR: usage: pex [options] [file_or_dir] [file_or_dir] [...]
pex: error: unrecognized arguments: --run_large
  inifile: None
  rootdir: /tmp/pants-sandbox-2bbw6E/src/logutil

((default/3.12.10) ) oliver@astro:~/Documents/code/main/py$ ls /tmp/pants-sandbox-2bbw6E
ls: cannot access '/tmp/pants-sandbox-2bbw6E': No such file or directory
but, as you can see, it appears the andbaox was not retained. What did I do wrong?
h
To be sure you're looking for the right thing, are you getting the sandbox location from that error message? That obviously looks like it should be the relevant sandbox, but look for it in the log lines preceding the invocation just to be sure.
n
That's the complete output - there aren't any log lines that preceed that error message.
h
And if you remove the 
-- --run_large
which pex is choking on?
n
FWIW, the 
--run_large
works when my 
conftest.py
is run - it adds that as a pytest flag and then the test runs and passes. If I remove that the test passes but I still don't get any extra logging:
Copy code
$ pants --keep-sandboxes=always test --debug --test-force --test-output=all  ./src/logutil/:: 
=========================================================================================================== test session starts ============================================================================================================
platform linux -- Python 3.12.10, pytest-8.4.2, pluggy-1.6.0
rootdir: /tmp/pants-sandbox-6Hw7P9/src/logutil
collected 1 item                                                                                                                                                                                                                           

src/logutil/test_json_fmt.py::test_json_fmt PASSED                                                                                                                                                                                   [100%]

============================================================================================================ 1 passed in 0.01s =============================================================================================================
$ ls /tmp/pants-sandbox-6Hw7P9
ls: cannot access '/tmp/pants-sandbox-6Hw7P9': No such file or directory
I thought maybe it was re-using sandboxes from prior runs but if I make a change to that test and re-run it still doesn't log anything. So I tried deleting all the sandboxes and re-ran and it doesn't create any new ones:
Copy code
$ rm -rf /tmp/pants-sandbox-*
$ pants --keep-sandboxes=always test --debug --test-force --test-output=all  ./src/logutil/:: 
17:45:00.76 [WARN] A plugin is calling `await Effect(InteractiveProcessResult, InteractiveProcess, process)` directly. This will cause restarting logic not to be applied. Use `await run_interactive_process(process)` or `await run_interactive_process_in_environment(process, environment_name)` instead. See <https://github.com/pantsbuild/pants/blob/release_2.28.0/src/python/pants/engine/intrinsics.py> for more details.
=========================================================================================================== test session starts ============================================================================================================
platform linux -- Python 3.12.10, pytest-8.4.2, pluggy-1.6.0
rootdir: /tmp/pants-sandbox-LI1JB5/src/logutil
collected 1 item                                                                                                                                                                                                                           

src/logutil/test_json_fmt.py::test_json_fmt PASSED                                                                                                                                                                                   [100%]

============================================================================================================ 1 passed in 0.01s =============================================================================================================
$ ls /tmp/pants*
ls: cannot access '/tmp/pants*': No such file or directory
However, my test does depend on conftest according to pants:
Copy code
$ pants dependencies ./src/logutil/test_json_fmt.py 
src/conftest.py:conftest
src/logutil/__init__.py:lib
src/logutil/json_fmt.py:lib
I'm stumped.
Even stranger: at some point I ran the exact same command and it did log the sandbox directories. Then I did some exploration and could no longer see the full list of sandboxes so I re-ran the same exact command (up-arrow) and it did not display the sandboxes again and I can't get it to display them again. Per above, I can't figure out what triggers it to use a sandbox and/or display the sandbox.
h
Well, the sandboxes are only created (and therefore displayed) when a process actually runs, but at the very least your pytest process should run, due to 
--test-force
and to the fact that failed processes aren't cached.
So you should be seeing a log line telling you where the sandbox is created
weird
I'm also not sure what `[WARN] A plugin is calling 
await Effect(InteractiveProcessResult, InteractiveProcess, process)
directly. This will cause restarting logic not to be applied. Use 
await run_interactive_process(process)
or 
await run_interactive_process_in_environment(process, environment_name)
instead. See https://github.com/pantsbuild/pants/blob/release_2.28.0/src/python/pants/engine/intrinsics.py for more details.` is about.
Is there a custom plugin in the mix?
n
Is there a custom plugin in the mix?
I don't think so.
BTW: all the help is much appreciated @happy-kitchen-89482
h
Try adding 
--no-pantsd
n
OK - gotta switch branches to get back to that state - hold please...
h
You should see 
[INFO] Preserving local process execution dir /blah/blah/pants-sandbox-blah for Run Pytest for blah
n
OK, that looks more promising:
Copy code
$ pants --keep-sandboxes=always --no-pantsd test --debug --test-force --test-output=all   ./src/logutil/:: 
11:11:47.18 [INFO] Preserving local process execution dir /tmp/pants-sandbox-iKtiIJ for Find interpreter for constraints: CPython==3.12.10
11:11:48.39 [INFO] Completed: Scheduling: Find interpreter for constraints: CPython==3.12.10
11:11:48.40 [INFO] Completed: Scheduling: Building 1 requirement for requirements.pex from the 3rdparty/python/default.lock resolve: pytest
11:11:48.40 [INFO] Completed: Scheduling: Building pytest.pex
11:11:48.40 [INFO] Completed: Scheduling: Building local_dists.pex
11:11:48.40 [INFO] Preserving local process execution dir /tmp/pants-sandbox-CnzwJP for Test binary /usr/bin/bash.
11:11:48.40 [INFO] Preserving local process execution dir /tmp/pants-sandbox-H6NPSK for Test binary /bin/bash.
11:11:48.41 [INFO] Completed: Scheduling: Test binary /usr/bin/bash.
11:11:48.41 [INFO] Completed: Scheduling: Test binary /bin/bash.
11:11:48.42 [INFO] Completed: Scheduling: Building pytest_runner.pex
=========================================================================================================== test session starts ============================================================================================================
platform linux -- Python 3.12.10, pytest-8.4.2, pluggy-1.6.0
rootdir: /tmp/pants-sandbox-SiQm3u/src/logutil
collected 1 item                                                                                                                                                                                                                           

src/logutil/test_json_fmt.py::test_json_fmt PASSED
and I do see a pex in that sandbox:
Copy code
$ ls -l /tmp/pants-sandbox-iKtiIJ
total 4728
-r-xr-xr-x 2 oliver oliver 4833957 Oct 19 11:08 pex
-rwxr-xr-x 1 oliver oliver    1564 Oct 19 11:11 __run.sh
I assume I should 
unzip -l
it to see what's in there?
unzip -l /tmp/pants-sandbox-iKtiIJ/pex | rg test_json_fmt.py
returns nothing - if I look at what's in the pex I don't see any of my code - just 3rd party dependencies and things.
The other sandboxes just contain a single 
__run.sh
file.
And the one where it says it's going to run the test got cleaned up:
Copy code
$ ls /tmp/pants-sandbox-SiQm3u
ls: cannot access '/tmp/pants-sandbox-SiQm3u': No such file or directory
I'm quite confused: • There 4 total sandboxes but only 3 are mentioned in the log lines that preceed the test run • Of the 3 that were logged 2 contain just a single 
__run.sh
file and one contains a 
pex
file but that pex doesn't contain any of my code. • The 4th sandbox, the one that wasn't included in a log message, has been deleted
@happy-kitchen-89482 if it helps I'd be more than happy to jump on a video chat and share my screen...
h
Your source files should be in the sandbox, including your conftest.py
Under src/ if I remember correctly
Not in the pex
And we specifically want the sandbox that ran pytest
The logs tell you which is which
n
Per above, they're not. The log messages mention 3 sandboxes. 2 contain only a single 
__run.sh
file. One contains a 
__run.sh
and a 
pex
file. Nothing else. The test itself logs, 
rootdir: /tmp/pants-sandbox-SiQm3u/src/logutil
but as you can see above, 
/tmp/pants-sandbox-SiQm3u
doesn't exist and it isn't mentioned in the log lines before the test ran. Also note that 
/tmp/pants-sandbox-SiQm3u
is not mentioned in any of the log messages from before the test ran.
h
None of those are the pytest sandbox
It should be like the log line I posted above
So that is weird
Oh
n
Agreed.
h
Right,take out the debug flag
That runs pytest in an interactive process so you can attach a debugger
If you're not doing that then remove that flag
n
AHA! Yes, that resulted in a lot more output. FWIW, the reason I added 
--debug
is it's the only way I know of to get "live output" as the test runs rather than a dump of stdout/stderr after the test is complete. I'm just being impatient 🙂
Anyway - with the extra output lemme see what I can see...
OK - the 
conftest.py
file is in the sandbox. Full ouptut:
Copy code
$ pants --keep-sandboxes=always --no-pantsd test  --test-force --test-output=all   ./src/logutil/:: 
12:00:25.05 [INFO] Preserving local process execution dir /tmp/pants-sandbox-kJcLqQ for Find interpreter for constraints: CPython==3.12.10
12:00:26.54 [INFO] Completed: Scheduling: Find interpreter for constraints: CPython==3.12.10
12:00:26.55 [INFO] Preserving local process execution dir /tmp/pants-sandbox-44C6Io for Building pytest.pex
12:00:26.55 [INFO] Preserving local process execution dir /tmp/pants-sandbox-9DwrBl for Building 1 requirement for requirements.pex from the 3rdparty/python/default.lock resolve: pytest
12:00:26.55 [INFO] Canceled: Building pytest.pex
12:00:26.55 [INFO] Preserving local process execution dir /tmp/pants-sandbox-7HYWSV for Building pytest.pex
12:00:27.45 [INFO] Completed: Building pytest.pex
12:00:27.45 [INFO] Completed: Scheduling: Building pytest.pex
12:00:27.51 [INFO] Completed: Building 1 requirement for requirements.pex from the 3rdparty/python/default.lock resolve: pytest
12:00:27.51 [INFO] Completed: Scheduling: Building 1 requirement for requirements.pex from the 3rdparty/python/default.lock resolve: pytest
12:00:27.51 [INFO] Preserving local process execution dir /tmp/pants-sandbox-0hWG4I for Building local_dists.pex
12:00:28.29 [INFO] Completed: Building local_dists.pex
12:00:28.30 [INFO] Completed: Scheduling: Building local_dists.pex
12:00:28.30 [INFO] Preserving local process execution dir /tmp/pants-sandbox-dgvdMK for Test binary /bin/bash.
12:00:28.30 [INFO] Preserving local process execution dir /tmp/pants-sandbox-6DCiDC for Test binary /usr/bin/bash.
12:00:28.32 [INFO] Completed: Scheduling: Test binary /bin/bash.
12:00:28.32 [INFO] Completed: Scheduling: Test binary /usr/bin/bash.
12:00:28.32 [INFO] Preserving local process execution dir /tmp/pants-sandbox-A2d9LR for Building pytest_runner.pex
12:00:29.16 [INFO] Completed: Building pytest_runner.pex
12:00:29.16 [INFO] Completed: Scheduling: Building pytest_runner.pex
12:00:29.17 [INFO] Preserving local process execution dir /tmp/pants-sandbox-lNudhT for Run Pytest for src/logutil/test_json_fmt.py:tests
12:00:29.33 [INFO] Completed: Scheduling: Run Pytest for src/logutil/test_json_fmt.py:tests
12:00:29.34 [INFO] Completed: Run Pytest - src/logutil/test_json_fmt.py:tests - succeeded.
============================= test session starts ==============================
platform linux -- Python 3.12.10, pytest-8.4.2, pluggy-1.6.0
rootdir: src/logutil
collected 1 item

src/logutil/test_json_fmt.py::test_json_fmt PASSED                       [100%]

- generated xml file: src.logutil.test_json_fmt.py.tests.xml -
$ $ ls /tmp/pants-sandbox-lNudhT/src
conftest.py  logutil
But the 
conftest.py
doesn't appear to be run.
(BTW: I did check the contents of 
conftest.py
and it's as expected - it's my code that adds the 
--run_large
flag which used to work before I upgraded pytest)
h
You can remove the --no-pantsd flag now, the --debug flag (which I previously missed noticing) explains all the sandbox issues
n
OK, thanks. It seems the main mystery remains: why my 
conftest.py
is being ignored. Any thoughts there?
h
OK, so now you can play in that sandbox (run/modify 
__run.sh
or the scripts it points to) to figure it out. But the good news is that conftest.py is in the sandbox, so now you get to debug this at the pytest level
n
I see. Thanks.
@happy-kitchen-89482 I think I've figured this out. The issue appears to be this: https://github.com/pytest-dev/pytest/issues/11261 Specifically, in more recent versions of pytest (that bug says 8.5 but StackOverflow said this started with 7.4 and I'm seeing it with 8.3) if you don't have some kind of project config file (e.g. 
pytest.ini
, 
pyproject.toml
, etc.) in the sandbox root pytest no longer walks up the directory tree to find 
conftest.py
files. There's 2 possible workarounds: 1. put a file like 
pytest.ini
in the root 2. pass the 
--confcutdir
flag I have verified that both work. Letting you know as I assume other Pants users are going to hit this issue so I'm thinking Pants might want want to add one of these fixes as part of the standard 
test
task. Thoughts?
In the meantime, this workaround did the trick:
Copy code
files(
   name = 'pytest_marker',
   sources = ['pytest.ini']
)

python_test_utils(
   name='conftest',
   sources=['conftest.py'],
   dependencies=[':pytest_marker'],
)
h
Oooof that is annoying!! Glad you found a solution.
n
@happy-kitchen-89482 do you agree the fix should be part of Pants? I suspect others will hit this. If so I'd be happy to file a bug...
h
Hmm, I'm not sure. How would pants do this? It seems too magical. Probably better to document the solution?
In which case a PR with the documentation change would be most welcome!
n
The 
--confcutdir
solution doesn't feel too magic to me. You just pass that as an argument to pytest setting it to 
src
in the sandbox. There's already magic to automatically add the conftest.py to the set of dependencies for all tests, so it feels in keeping with the current level of magic. But I can also look into sending you a PR with documentation if that still feels too magic.
h
I guess, I'm just afraid of unintended consequences of setting 
--confcutdir
What does that do exactly?
n
From https://docs.pytest.org/en/6.2.x/reference.html#confval-confcutdir:
Sets a directory where search upwards for 
conftest.py
files stops. By default, pytest will stop searching for 
conftest.py
files upwards from `pytest.ini`/`tox.ini`/`setup.cfg` of the project if any, or up to the file-system root.
So I think setting that to 
src
in the sandbox restores the previous behavior.
h
I don't understand then why that flag is necessary in our case. If there is no 
pytest.ini
or similar then shouldn't it keep going to the filesystem root?
And find the conftest.py?
I am missing something here
n
If there is no 
pytest.ini
or similar then shouldn't it keep going to the filesystem root?
Not anymore. That was the change in 8.5 (or 7.4 depending on who you believe). I think it's a security thing. If there's no 
pytest.ini
they can't figure out where they've left your project and entered the uncharted domain of the filesystem as a whole so they they don't want to run code that's outside your project. So, if they can't find a project marker file they stop traversing up.
From https://github.com/pytest-dev/pytest/issues/11261:
This is due to this change from the changelog:
#11043: When --confcutdir is not specified, and there is no config file present, the conftest cutoff directory (--confcutdir) is now set to the rootdir. Previously in such cases, conftest.py files would be probed all the way to the root directory of the filesystem. If you are badly affected by this change, consider adding an empty config file to your desired cutoff directory, or explicitly set --confcutdir.
Note that if you click on that link "rootdir" is not 
/
. Here's part of the (long) writeup about what the rootdir is:
pytest determines a 
rootdir
for each test run which depends on the command line arguments (specified test files, paths) and on the existence of configuration files. The determined 
rootdir
and 
configfile
are printed as part of the pytest header during startup.
Here’s a summary what 
pytest
uses 
rootdir
for:
• Construct nodeids during collection; each test is assigned a unique nodeid which is rooted at the 
rootdir
and takes into account the full path, class name, function name and parametrization (if any).
• Is used by plugins as a stable location to store project/test run specific information; for example, the internal cache plugin creates a 
.pytest_cache
subdirectory in 
rootdir
to store its cross-test run state.
•
h
Got it. Makes sense from a security standpoint, for sure.
So sounds like we should set 
--confcutdir
to the root of the sandbox, and actually doing so unconditionally is never wrong.
Since we never want to discover conftest.py outside the sandbox
n
Agreed.
Would you like me to file a bug for that?
h
Sure, thanks
And if you feel like tackling it, it would be pretty straightforward, to update 
pytest_args
in 
src/python/pants/backend/python/goals/pytest_runner.py
--confcutdir=./src
should do it
n
Bug created here: https://github.com/pantsbuild/pants/issues/22773 I might have the time to tackle this tonight - if not I likely won't have time to get to it for quite some time.
9 Views
Open in Slack
PreviousNext
Powered by