https://pantsbuild.org/ logo
Join Slack
Powered by
Q. Does pants-executed pex (pytest) targets use a ...
# general
r
Q. Does pants-executed pex (pytest) targets use a private tmp directory? Could I change the behavior?
✅ 1
c
Yes. If you invoke 
./pants --no-process-cleanup …
then those tmp directories are preserved so you can inspect them and re-run the commands interactively again using the 
__run.sh
script which simulates the original process invoked by Pants as closely as possible (some env leaks through, iirc) https://www.pantsbuild.org/v2.12/docs/reference-global#section-process-cleanup
And this option allows you to set where those temp directories are rooted: https://www.pantsbuild.org/v2.12/docs/reference-global#section-local-execution-root-dir
r
Thanks,
I have another question: is it allowed to create any directories or files there that should persist during a pytest session, when running pytest via pants?
i'm experiencing a strange error that all docker container creation fails due to missing bind source directories which are created in pytest fixtures...
calling and printing 
os.listdir()
looks fine...
i'm creating directories like 
/tmp/process-execution4vBpVR/ipc/ipc-testing-bcDf-rUQkL4/container
but with 
--no-process-cleanup
, it only shows 
/tmp/process-execution4vBpVR/ipc
directory....
c
I think you can, but keep in mind that Pants invokes one pytest process per test file, so if you previously relied on a session fixture to setup something once for all tests across multiple files, that may not work as expected any more.
r
the 
ipc
directory is created via pytest's session-level fixture
hmmm
c
Pants only logs the temp directories used for the sandboxes where pytest is executed, if you create other temp directories, it will be up to you to preserve/log them etc…
r
hmmm
it may be a race condition between tests
i'm currently filtering to run a single test case using 
-- -k
but if pytest process is fired up multiple times for different files and terminates immediately after "selecting" tests
then i think created-but-missing directories may happen
running tests in parallel is a very promising goal, but it seems that i need to rewrite almost all fixtures as well as letting each test to use unique TCP port numbers, etc.
i think it would be nice if there is a structured guide/description on how to migrate existing test suites to work well with pants
i will try the global option to limit the test parallelism to 1
hmm still same
I've just tried: 
--process-execution-local-parallelism=1
image.png
ls -lR
shows the directory is there and immediately creating a docker container that mounts it fails.. 😞
(of course all these tests ran fine before migrating to pants)
E               aiodocker.exceptions.DockerError: DockerError(400, 'invalid mount config for type "bind": bind source path does not exist: /tmp/process-executionHj4oGH/ipc/ipc-testing-C4tYCSqV64E/container')
ahhhhhhhhhhhhhhhhhh
i got to know the reason
my docker is installed via Snap, and the docker daemon has its own private tmp and cannot access 
/tmp/...
in the host
i've changed 
[GLOBAL].local_execution_root_dir
in pants.toml to 
%(buildroot)s/tmp
and it now works with the mounts
..........
i think it would be good if the pants documentation has a note about this in the common troubleshooting page
i've submitted a suggested edit to https://www.pantsbuild.org/docs/troubleshooting
please feel free to edit/clean up it and apply 😉
c
Thanks 🙏
Open in Slack
PreviousNext
Powered by