Another question for my evaluation: In the python test section (https://www.pantsbuild.org/v2.8/docs/python-test-goal) it is written, that

./pants test ::

only reruns tests needed if I change e.g. a single file. This would then also be true on CI with a cache enabled right? So if I want up to date coverage on the main branch, I would need to remove the test result cache (

$HOME/.cache/pants/lmdb_store

) on the main branch so that all tests run? Right now we have some grouping of modules into clearly seperated dependency groups and to save time on the main branch we only run the tests for groups that are changed and then combine coverage to save a bit of execution time on the main branch), would this also be possible with pants?

I’m not 100% sure I understand your question fully, so will simply point out this option (in case you haven’t seen it): https://www.pantsbuild.org/v2.8/docs/reference-test#section-force That will allow you to run over all tests, regardless of what’s in the cache.

Yes, that answers the first part - which I hadn’t seen before (so I could keep the cache, but simply pass the force flag to rerun all tests). Basically we have different products (each comprises many modules) and a group of common modules. Each product is in a different source root (let’s call them X, Y and Z) and the same for the common modules (C). Now we know, that

X/...

never depends on targets in

Y/...

or

Z/...

but only on targets in

X

or in

C

. What we currently do, on our main branch to save CI time is that we look if there was a change only in

X,Y or Z

we run only the tests for

X

and merge coverage with the coverage of the last runs for

Y,Z and C

. This can miss coverage in

C

only coming from

X

- but that is something we would not want to have anyway. So my question was how to do this with pants, and I think the answer would also be using the force flag on the main branch and use change detection to limit to certain targets (e.g

X/**

) and keep our current coverage merging logic as pants does not keep or merge coverage in any way, right?

Ah, right. Pants does merge coverage data as the tests are partitioned, but I don’t think they merge between runs, only for a single run. So I guess you could use something like

./pants --changed-since=<some tag/sha/branch…> test

and then (hopefully) merge the coverage data from that as you do now..

Right now we do not use changed since logic for merging coverage because that is always a bit messy (that’s why we tried to identify these larger “dependency clusters” to make coverage merging very close to actual coverage). But maybe with the fine grained information and some dependees logic this could actually work. Thanks for the input!

I think you understand this but just to be clear

./pants test ::

would run all tests in the app. Some of those tests may have cached results and Pants will know those cached results remain valid. These tests will have been 'run' but actually just used cached results (

--test-force

would override this).

./pants test --changed-since=HEAD^ --changed-dependees=transitive

will run only tests that have changed since the provided git refer (

HEAD^

in this case). Any tests that do not depend indirectly on a changed file will not be run at all. It is still possible that some of these test runs will have a cache hit.

Yes that was my understanding, however with the caveat that cached test do not contribute to coverage (which is basically the problem I want to solve, how to get accurate coverage with the least tests that need to run).

Yeah, Gordon is right. Cached tests should feed into coverage!

./pants test ::

is like you had run every single test, only some of them use the cached result. And it will only use the cache if none of the inputs for that test have changed, including every transitive dependency of it - so you can be confident the coverage data is accurate Are you seeing differently, that coverage is not including cached results?

Ah ok, I just understood it differently from the documentation where it states that “Coverage will not report on unencountered files” - I somehow read this as coverage is not cached. I think that was a misunderstanding on my part then.

Ah, I see how that could be interpreted that way. Also, that advice is stale! We added a feature to force reporting on the entire repo's code, even if it's not transitively used by a test: https://www.pantsbuild.org/v2.8/docs/reference-coverage-py#section-global-report Will update the docs this week

sounds like this is answered: but at a fundamental level, if toggling the coverage flag required that the test process have different settings/inputs or produced different outputs, it would miss the cache and re-run anyway. any other behavior would be a bug!