Have anyone used mutation tests before?
# generalf
fresh-architect-74417
10/12/2020, 3:47 PMHave anyone used mutation tests before?
c
calm-ambulance-65371
10/12/2020, 3:48 PMWe've talked about it but have not implemented it
h
hundreds-father-404
10/12/2020, 4:04 PMLet us know if we can improve Pants to help facilitate mutation testing, e.g. helping to write a plugin to integrate with Mutmut
👍 2
f
fresh-architect-74417
10/13/2020, 7:53 PMThanks for offering it. TBH, I don't think mumut is going to work. You will need a new way of running the unittests that pants itself need to be aware of it.
The first step is to run each individual unittest function with coverage. Then take the code coverage and run mutation on each line. For each mutation, you will need to publish it to test runner such as buildfarm. You will need a way to report false positive by converting the impacted code to AST then generalize in an AI model that are used in NLP.
fresh-architect-74417
10/13/2020, 7:54 PMFrom what I remember when I worked as a contractor at Google, they run the mutation tests after the CI test pass, and they report the mutated code in the PR itself. I'm not sure how they implemented the false positive reporting.
h
hundreds-father-404
10/13/2020, 7:56 PMThe first step is to run each individual unittest function with coverage.This is how Pants does test coverage atm. Each test file is a distinct process, and we use
--pytest-covcoverage combineThen take the code coverage and run mutation on each lineBecause each test runs in a chroot, I think this will be possible. That is, mutating the code will not impact your actual build root. It only mutates the chroot. It’s safe.
For each mutation, you will need to publish it to test runner such as buildfarm.Is the remote execution only for performance, or some other motivation?
You will need a way to report false positive by converting the impacted code to AST then generalize in an AI model that are used in NLP.That part we might not be as able to help with 😅
f
fresh-architect-74417
10/13/2020, 7:58 PMIs the remote execution only for performanceIt's mainly for performance. Otherwise, running the same mutated test multiple times is not going to be practical
fresh-architect-74417
10/13/2020, 7:59 PMDo you have a developer on boarding docs? If I want to understand how pants work, where do I start?
h
hundreds-father-404
10/13/2020, 8:05 PMThe best way to understand Pants is to look at the Plugin API, which is a bit of a misnomer. It’s the exact same API we use to build core Pants functionality. https://www.pantsbuild.org/docs/plugins-overview
Deeper down, there’s the Rust engine. That should be entirely abstracted away, though, unless you want to change how the engine itself works.
hundreds-father-404
10/13/2020, 8:11 PMIt’s mainly for performance. Otherwise, running the same mutated test multiple times is not going to be practicalCool. The way the Plugin API is set up, remote execution works with no extra effort required by the plugin author. Build logic is broken up into several composable steps via “rules”. Things like subprocesses are hermetic. The engine will send relevant work to the remote workers for you. The only challenge is for the end-user to hook up remote execution, i.e. get their config and env working. Tom is working on some docs for that.
f
fresh-architect-74417
10/13/2020, 8:33 PMThanks for the document. I will go over it.
h
hundreds-father-404
10/13/2020, 8:35 PMLet us know if you have any questions. A couple folks have written plugins so far, and we know we’re going to want to do a revamp of those plugin docs based on that feedback to include more guided examples.
Also check out the #C01CQHVDMMW Slack channel