https://pantsbuild.org/ logo
#general
Title
# general
r

rhythmic-morning-87313

05/16/2022, 7:41 AM
Q. Does pants-executed pex (pytest) targets use a private tmp directory? Could I change the behavior?
✅ 1
c

curved-television-6568

05/16/2022, 8:43 AM
Yes. If you invoke
./pants --no-process-cleanup …
then those tmp directories are preserved so you can inspect them and re-run the commands interactively again using the
__run.sh
script which simulates the original process invoked by Pants as closely as possible (some env leaks through, iirc) https://www.pantsbuild.org/v2.12/docs/reference-global#section-process-cleanup
And this option allows you to set where those temp directories are rooted: https://www.pantsbuild.org/v2.12/docs/reference-global#section-local-execution-root-dir
r

rhythmic-morning-87313

05/16/2022, 9:06 AM
Thanks,
I have another question: is it allowed to create any directories or files there that should persist during a pytest session, when running pytest via pants?
i'm experiencing a strange error that all docker container creation fails due to missing bind source directories which are created in pytest fixtures...
calling and printing
os.listdir()
looks fine...
i'm creating directories like
/tmp/process-execution4vBpVR/ipc/ipc-testing-bcDf-rUQkL4/container
but with
--no-process-cleanup
, it only shows
/tmp/process-execution4vBpVR/ipc
directory....
c

curved-television-6568

05/16/2022, 9:10 AM
I think you can, but keep in mind that Pants invokes one pytest process per test file, so if you previously relied on a session fixture to setup something once for all tests across multiple files, that may not work as expected any more.
r

rhythmic-morning-87313

05/16/2022, 9:10 AM
the
ipc
directory is created via pytest's session-level fixture
hmmm
c

curved-television-6568

05/16/2022, 9:11 AM
Pants only logs the temp directories used for the sandboxes where pytest is executed, if you create other temp directories, it will be up to you to preserve/log them etc…
r

rhythmic-morning-87313

05/16/2022, 9:13 AM
hmmm
it may be a race condition between tests
i'm currently filtering to run a single test case using
-- -k
but if pytest process is fired up multiple times for different files and terminates immediately after "selecting" tests
then i think created-but-missing directories may happen
running tests in parallel is a very promising goal, but it seems that i need to rewrite almost all fixtures as well as letting each test to use unique TCP port numbers, etc.
i think it would be nice if there is a structured guide/description on how to migrate existing test suites to work well with pants
i will try the global option to limit the test parallelism to 1
hmm still same
I've just tried:
--process-execution-local-parallelism=1
image.png
ls -lR
shows the directory is there and immediately creating a docker container that mounts it fails.. 😞
(of course all these tests ran fine before migrating to pants)
E               aiodocker.exceptions.DockerError: DockerError(400, 'invalid mount config for type "bind": bind source path does not exist: /tmp/process-executionHj4oGH/ipc/ipc-testing-C4tYCSqV64E/container')
ahhhhhhhhhhhhhhhhhh
i got to know the reason
my docker is installed via Snap, and the docker daemon has its own private tmp and cannot access
/tmp/...
in the host
i've changed
[GLOBAL].local_execution_root_dir
in pants.toml to
%(buildroot)s/tmp
and it now works with the mounts
..........
i think it would be good if the pants documentation has a note about this in the common troubleshooting page
i've submitted a suggested edit to https://www.pantsbuild.org/docs/troubleshooting
please feel free to edit/clean up it and apply 😉
c

curved-television-6568

05/16/2022, 10:58 AM
Thanks 🙏