Anyone have some thoughts on how to make the test ...
# general
b
Anyone have some thoughts on how to make the test output blob more legible?
Copy code
✓ path/to/my/project/and/my/module/test_that_my_module_works.py:tests succeeded in 54.06s (cached locally).
At the very least I'm thinking we could apply colors to a few select atoms of data here (like the test name and the process metadata source) Maybe
succeeded
and
failed
could also be color at least to break the monocolor line. (The check or X on the waaaay left doesn't help split the line visually)
h
I really like bazel's way of doing this
They cut down on the clutter by not reporting the individual status of all tests and only break it down for failed tests
Haven't had time to dig through the plugin to see how feasible it is for me to do a pull request, but I do find this especially noisy for our build systems that run everything.
b
In this case I think the output is hardwired into Pants'
test
goal handling, so it'd be a change to Pants itself...... Although making this customizable outside of Pants (via a config or pluggable) is quite interesting
h
The benefits I'm aware of for seeing success output is 1) verifying no false positive where your test wasn't actually collected/executed 2) seeing how long the test took to know if interfaces are being mocked properly
The idea of separating test execution/report generation and report rendering/visualization is something I've seen and worked with in other tools so there's some precedent there.
w
formatting changes should be very straightforward: https://github.com/pantsbuild/pants/blob/c0ba9ffeeb98e9ec17ad4bf69f98470ef113a1d5/src/python/pants/core/goals/test.py#L523-L545 … most of the effort would probably be around finding good defaults, but more options (or a threshold on test count before summarizing, etc) could work too.
👍 2
b
@witty-crayon-22786 what do you think about the feasibility and "does-it-make-sense"ability for making the output pluggable?
FWIW I originally started editing
_format_test_summary
before making this post because I thought it was worth debate from the community what the ouptut should look like
w
not sure about pluggability… for a case like this, structured output feels like a more generally useful capability. e.g. json.
probably a question of how many folks who would write a plugin would end up writing json with their plugin =P
i generally think that plugins make sense when you cannot anticipate usecases. but this seems like a relatively bounded domain, where we could improve the output to catch 95% of cases without plugins (and leave the rest to structured output)
👍 1
b
A third facet here is optionability. Like @high-yak-85899’s ability to "turn off" reporting the passed tests individually
w
yea. i feel like that could be a threshold with a default (more than 50 tests defaults to a summary)
🤔 2
h
Yeah something to capture the "if I have to scroll to see all the output, it's too much" sentiment. Totally on board with a setting like that.
👍 1
h
I like that idea. My fear the whole time has been "I'm trying to run a single test -- did it run??"
h
Yeah definitely. I think Stu's idea captures that fine. If I'm developing locally, I'll be running just that test and feel confident Pants isn't skipping it because it will show the full output. In practice, are people regularly auditing this test output against what is expected?
w
having said that, summarizing also moves the problem to a degree… need to decide what a good summary looks like. i can imagine summary plugins, although whether they needed to be thirdparty is another question
b
FWIW the OP asked about splitting a single line, visually. So let's not forget that 😂
h
I personally find it legible. Just overbearing sometimes.
w
@bitter-ability-32190: yea, sorry. i suspect that that would be uncontroversial… both color and indentation would be great.
👍 1
see https://github.com/pantsbuild/pants/issues/13588#issuecomment-991863653 for the last discussion of that topic i think