https://pantsbuild.org/ logo
#general
Title
# general
a

ambitious-student-81104

08/30/2021, 9:51 PM
Hi! I'm trying to turn on pants test coverage report on CI (buildkite) for the purposes of: 1. viewing the diff coverage of each PR 2. viewing the global coverage of each branch 3. blocking PRs whose diff coverage is below a certain threshold (and hopefully can set this threshold as customizable by directory) Right now I'm able to get test coverage report working very nicely on my local machine, but it's not clear how to hook that up with CI and github. Is there any resources that'll be helpful? Thanks a ton 🙏
h

happy-kitchen-89482

08/30/2021, 10:04 PM
Hmmm, @polite-garden-50641 do we run coverage in our internal CI? (it's on GitHub Actions but the same principles may apply)
@ambitious-student-81104 Re your purposes 1. and 2. : where do you want to view these? on GitHub, say?
a

ambitious-student-81104

08/30/2021, 10:07 PM
Yah ideally on github like https://about.codecov.io/ but i understand that might be hard to achieve... so as long as it can output somewhere that allows a bot to pick it up and say "hey, your diff coverage is too low, you can't merge this PR!" that'll be good
p

polite-garden-50641

08/30/2021, 10:14 PM
Yes we do. And we get GH status checks from codecov.io (those status checks can be used to block merging PRs if coverage is too low)
In our pants CI config file we have:
a

ambitious-student-81104

08/30/2021, 10:16 PM
My organization does not use codecov. Is there a way I can set it up directly between buildkite, GH and pants? I also have the config you screenshot. IIUC it dumps the file to the CI machine's local dir? How does it show up on GH then?
p

polite-garden-50641

08/30/2021, 10:17 PM
In our circle CI config we have this:
Codecov provides an orb (circle CI plugin ) to help upload the coverage data to their services
So without using an external service , you will need the CI process itself (one of the steps) to fail the build if coverage is too low
Coverage.py has a “—fail-under” option https://coverage.readthedocs.io/en/coverage-5.5/cmd.html
1
That can be used to do just that
a

ambitious-student-81104

08/30/2021, 10:22 PM
yes, the only thing is that's global coverage, not diff 🙁
p

polite-garden-50641

08/30/2021, 10:22 PM
Since pants runs the “coverage report” command, this option should be configured via pants
a

ambitious-student-81104

08/30/2021, 10:23 PM
should be in the
.coveragerc
, no?
p

polite-garden-50641

08/30/2021, 10:23 PM
Probably
You can’t do a diff coverage that easily in a single CI run.. since a diff needs to compare the current coverage to a previous one.. the previous coverage data is typically not available in CI
a

ambitious-student-81104

08/30/2021, 10:24 PM
that's true
unfortunately i didn't find as good of a plugin for buildkite like circleci does...
Interesting. The
fail-under
configuration seems to be applied to every single file when used with pants test, not global
Is this intended? I only want to enforce the fail-under on the global test cov, which is consistent with what coverage's doc says, but isn't what happens when I run
./pants test
Looks like it has to do with each test file having its own separate test session... that's a side effect of running tests concurrently? Can this be fixed so I have both concurrency and global
fail-under
?
h

happy-kitchen-89482

08/31/2021, 9:13 AM
Oh, right 😞 Options set on
pytest
itself (or its plugins) will apply to each run of
pytest
. Which isn't what you want in this case. So it looks like you might need a little custom logic in the
test
goal - you want to fail the Pants
test
run as a whole, not any individual test in it (because, which test would that even be?)
Let's see...
@ambitious-student-81104 Do you mind opening a feature request ticket for this at https://github.com/pantsbuild/pants/issues/new/choose ?
So we can track it less ephemerally than on Slack?
b

brash-baker-91190

08/31/2021, 1:17 PM
@ambitious-student-81104 I recently made this, but haven't publicized it yet... it may be useful to you: https://github.com/grapl-security/codecov-buildkite-plugin
🎉 3
We've been using it at work for the past few weeks
We use Pants, so the defaults for that plugin are Pants-friendly 😂
❤️ 2
w

witty-crayon-22786

08/31/2021, 3:44 PM
awesome stuff!
a

ambitious-student-81104

08/31/2021, 3:47 PM
@happy-kitchen-89482 Will do!
@brash-baker-91190 Awesome! Do I need to be a customer of codecov to use this?
b

brash-baker-91190

08/31/2021, 3:50 PM
@ambitious-student-81104 You'll need to have enough of a relationship with codecov to have an API token from them. Open source usage is free, if that's relevant to your usecase.
I don't seem to have documented it explicitly yet (🤦‍♂️), but the plugin assumes the token is in the environment as
CODECOV_TOKEN
(e.g., https://github.com/grapl-security/codecov-buildkite-plugin/blob/5a6e2d5a6b6f2812d4d04c0c2f5fe67b453df0f1/hooks/post-command#L27)
(which is what the
codecov
uploader looks for by default)
a

ambitious-student-81104

08/31/2021, 3:55 PM
I will need enterprise access which means my company will need to pay for it, and they'll also need a security review... so anything that requires codecov integration will be no-go for me right now 🙁
b

brash-baker-91190

08/31/2021, 6:50 PM
oh, so you're looking to find something like codecov that will let you perform historical code coverage statistic comparisons in your Github PR, but that's not a separate service; is that accurate?
1
a

ambitious-student-81104

09/01/2021, 3:30 PM
Yes, precisely! And it needs to work with buildkite as well; buildkite doesn't have as ready of a plugin as circleCI.
@happy-kitchen-89482 Opened feature request here
b

brash-baker-91190

09/01/2021, 3:38 PM
@ambitious-student-81104 Sounds like you'd have to build something yourself. After reading through Codecov's documentation, however, reliable historical tracking of these kind of statistics is surprisingly more complicated than I would have initially guessed. There are lots of thorny edge cases 😕
h

happy-kitchen-89482

09/01/2021, 3:59 PM
Thanks! Will take a look and bring in @hundreds-father-404, who knows way more about coverage than I do 🙂
a

ambitious-student-81104

09/01/2021, 4:00 PM
@brash-baker-91190 Thanks for the heads-up! For now I'm gonna be happy with just failing CI build on <min test coverage, or > max drop in cov allowed, which I guess shouldn't be hard!
b

brash-baker-91190

09/01/2021, 4:01 PM
@ambitious-student-81104 Yup, that's definitely a step up 👍 Good luck!
h

happy-kitchen-89482

09/08/2021, 7:29 PM
Closing the loop on this: This was added as a feature in https://github.com/pantsbuild/pants/pull/12778, which will go into the next dev release, at the end of the week.
Thanks for the feature request!
a

ambitious-student-81104

09/10/2021, 3:02 PM
Thanks a lot for following up on this @happy-kitchen-89482!
5 Views