We have some heavy tests which consume a lot of memory for t Pants #general

We have some heavy tests which consume a lot of me...

gentle-gigabyte-52115

11/21/2022, 2:11 PM

We have some heavy tests which consume a lot of memory, for these tests we want to restrict the parallelism. Is there a way to reduce parallelism for some tests? While letting other tests run in parallel.

wide-midnight-78598

11/21/2022, 2:18 PM

This might be a starting point: https://www.pantsbuild.org/docs/troubleshooting#controlling-test-parallelism

gentle-gigabyte-52115

11/21/2022, 2:19 PM

Indeed I was looking at this.

Copy code

--process-execution-local-parallelism=1

I wouldn’t want to do because it will reduce performance.

[pytest].execution_slot_var

I don’t really understand how this could be used to control parallelism.

happy-kitchen-89482

11/21/2022, 2:27 PM

execution_slot_var

is not used to control parallelism - it names an env var that will contain a “slot number” that your tests can use to assign unique databases or other resources

happy-kitchen-89482

11/21/2022, 2:31 PM

I don’t think there is a way to restrict parallelism for specific tests in a single pants run. But one thing you can do is create separate

python_tests

targets for those heavy tests (and be sure to exclude those tests from the sources of the

python_tests

target that owns the other tests in the same dir) . Then you can tag the heavy test targets. Then you can run Pants twice, once with

./pants test --process-execution-local-parallelism=1 --tag=heavy ::

and again with

./pants test --tag=-heavy ::

✅ 1

gentle-gigabyte-52115

11/21/2022, 2:35 PM

Thanks Benjy I think that would work indeed.

ambitious-actor-36781

11/21/2022, 10:18 PM

If you use the pytest-xdist parallelizer , you could use xdists segmentation functionality.

5 Views

Open in Slack

Previous Next