Hi, I have a question regarding running pytests with pants test: I noticed when I run the tests with

./pants test foo/tests::

, pants apparently runs pytest individually on every test file inside that folder. I guess this is to allow parallel test execution. However, then the output also looks like running pytest individually, e.g. I have one pytest summary printed to terminal per test file and one pytest html report generated per test file if I use the pytest-html plugin. I would rather prefer it to behave like pytest is being run on all files at once, even if I lose parallel execution. I have found a screenshot of an older version of v2 here https://github.com/pantsbuild/pants/pull/9594 where the output looks exactly like I expect it (one test summary for multiple files), but I'm not sure if/how I can achieve that in the current pants version?

Hi, It seems that it is not currently possible, from what I can see. (it might be possible by adding dependencies between test files… but that would be more of a hackish way of going about it) There’s an ongoing discussion regarding the trade-offs how to best partition jobs (this for lint/check I think, but could apply to some extent to test as well).

I'm on mobile so it's hard to find links, but you could look into how the html plugin works and see if it takes in the xml report to generate the html one. Pants isn't unique in running tests in parallel, I know pytest-xdist does something similar. So it might be possible to find a script or some code to merge the xml reports and then generate the html from that.

That’s good advice, Joshua. I diged up this link: https://github.com/pytest-dev/pytest-html/issues/183#issuecomment-738195470

Hi, you're correct that Pants runs pytest once per file, both for parallel execution and so that each test execution can be cached separately, for better cache hit rates.

Thanks, I was indeed running with a test-output=all setting in the pants.toml I forgot about 🙂 Without this, the terminal output is much cleaner. Regardless, I think it'd be great to have the option to run it on all files and have the output accordingly. The pants test summary contains all important information, but I somehow just got so used to the pytest output and its customization options 🙂

Good feedback! Would you mind opening a ticket for this request at https://github.com/pantsbuild/pants/issues/new/choose ?

Yes, for me it doesn't matter how the tests are executed by pants. For me it's just that I'm coming from a non-pants setup where I could run pytest and customize the test summary/report using pytest options/plugins. Here, pants feels like a step back because I get a more basic summary that I cannot customize and I would have to invest effort to merge per-test-file reports (html, json) into a single report.

Great, so a ticket with that detail would be really helpful