Hi! I'm trying to turn on pants test coverage report on CI (buildkite) for the purposes of: 1. viewing the diff coverage of each PR 2. viewing the global coverage of each branch 3. blocking PRs whose diff coverage is below a certain threshold (and hopefully can set this threshold as customizable by directory) Right now I'm able to get test coverage report working very nicely on my local machine, but it's not clear how to hook that up with CI and github. Is there any resources that'll be helpful? Thanks a ton 🙏

Hmmm, @polite-garden-50641 do we run coverage in our internal CI? (it's on GitHub Actions but the same principles may apply)

Yah ideally on github like https://about.codecov.io/ but i understand that might be hard to achieve... so as long as it can output somewhere that allows a bot to pick it up and say "hey, your diff coverage is too low, you can't merge this PR!" that'll be good

Yes we do. And we get GH status checks from codecov.io (those status checks can be used to block merging PRs if coverage is too low)

My organization does not use codecov. Is there a way I can set it up directly between buildkite, GH and pants? I also have the config you screenshot. IIUC it dumps the file to the CI machine's local dir? How does it show up on GH then?

In our circle CI config we have this:

yes, the only thing is that's global coverage, not diff 🙁

Since pants runs the “coverage report” command, this option should be configured via pants

should be in the

.coveragerc

, no?

Probably

that's true

Oh, right 😞 Options set on

pytest

itself (or its plugins) will apply to each run of

pytest

. Which isn't what you want in this case. So it looks like you might need a little custom logic in the

test

goal - you want to fail the Pants

test

run as a whole, not any individual test in it (because, which test would that even be?)

@ambitious-student-81104 I recently made this, but haven't publicized it yet... it may be useful to you: https://github.com/grapl-security/codecov-buildkite-plugin

awesome stuff!

@happy-kitchen-89482 Will do!

@ambitious-student-81104 You'll need to have enough of a relationship with codecov to have an API token from them. Open source usage is free, if that's relevant to your usecase.

I will need enterprise access which means my company will need to pay for it, and they'll also need a security review... so anything that requires codecov integration will be no-go for me right now 🙁

oh, so you're looking to find something like codecov that will let you perform historical code coverage statistic comparisons in your Github PR, but that's not a separate service; is that accurate?

Yes, precisely! And it needs to work with buildkite as well; buildkite doesn't have as ready of a plugin as circleCI.

@ambitious-student-81104 Sounds like you'd have to build something yourself. After reading through Codecov's documentation, however, reliable historical tracking of these kind of statistics is surprisingly more complicated than I would have initially guessed. There are lots of thorny edge cases 😕

Thanks! Will take a look and bring in @hundreds-father-404, who knows way more about coverage than I do 🙂

@brash-baker-91190 Thanks for the heads-up! For now I'm gonna be happy with just failing CI build on <min test coverage, or > max drop in cov allowed, which I guess shouldn't be hard!

@ambitious-student-81104 Yup, that's definitely a step up 👍 Good luck!

Closing the loop on this: This was added as a feature in https://github.com/pantsbuild/pants/pull/12778, which will go into the next dev release, at the end of the week.

Thanks a lot for following up on this @happy-kitchen-89482!