I’m curious about how coverage and `--changed-sinc...
# general
I’m curious about how coverage and
interact in testing. Or, rather, I can see that coverage only shows results for the files that actually ran during a
./pants test --changed-since
run. Are there any best practices for using that coverage information in CI? I’d like to use
in our CI to help speed up test runs, but obviously only showing coverage of the files that have been changed doesn’t allow us to gate MRs on increasing overall coverage
I think you'd need a server that "hosts" the overall coverage and is smart enough to be fed a subset of coverage (and carry the remaining metrics form the merge-base)
IIRC https://about.codecov.io/ has something like that
Ah this seems relevant, albeit wouldn't work with `--changed`: https://docs.codecov.com/docs/carryforward-flags
It's probably not prohibitively hard to whip something up, assuming you have a server to host the commit -> full coverage mapping. But I digress...
Thanks @bitter-ability-32190! Unfortunately, the nature of our work is that we can’t integrate third party services like CodeCov although that’s a great suggestion in general. The last few comments in the thread here seem like they were meant for a different conversation, though 😄
Oh ooooooops lol I'll delete those 😂
Alternatively, a remote caching server would presumably allow you to run
test ::
and see similar perf to
Yep, unfortunately we’re working on defense related stuff so external services are generally hard to get approval for 🙂
Ah yeah makes sense. Hosting a coverage metadata server is likely your option then if you absolutely need to use changed
Test time isn’t a problem yet, but this is good to keep in mind for when it does become a problem
Thank you!