Hi team, this is a question on how Pants run tests...
# general
Hi team, this is a question on how Pants run tests in parallel. I have multiple shards of unit tests and I run the command like this "`./pants -ldebug test test_dags_shard_1, test_dags_shard_2, ...`". What I see from the log is:
22:36:33.91 [DEBUG] Running Run Pytest for airflow2/test/python/data_dag_tests/dag_tests/test_dags_shard_6.py:tests under semaphore with concurrency id: 8, and concurrency: 1
22:36:33.91 [DEBUG] Starting: Run Pytest for airflow2/test/python/data_dag_tests/dag_tests/test_dags_shard_6.py:tests
22:36:33.91 [DEBUG] Starting: setup_sandbox
The message "`concurrency: 1`" makes me think there is actually no concurrency. Am I correct? Thanks!
by default, pants will run multiple test files in parallel, but it will not run multiple test methods in parallel
what you are seeing there is Pants spawning 1 pytest process, and assigning it 1 core
if you want to run methods in parallel, you can use https://www.pantsbuild.org/docs/reference-pytest#xdist_enabled … but note that it requires that your test methods be concurrency safe
Thanks Stu!
That'd make a cool Pants tip of the Day. How's this?
Tip of the Day:
Pants runs multiple test files in parallel by default. Want to run multiple test methods in parallel? Sure, no problem! https://www.pantsbuild.org/docs/reference-pytest#xdist_enabled (just remember to first make your test methods concurrency-safe...)
👍 1
Yea it is really cool!