Hi folks. Wondering if anyone has tried to parallelize tests in buildkite? I'm getting weird coverage report failures. I'm planning the tests by filtering for all the test targets and

split -l $block_size

. I suspect that I need to toposort and dedupe test targets?

you shouldn’t need to unless you have test targets depending on other test targets (which is discouraged, but not banned, afaik)

yes we are using pants 2.x. I'm getting coverage report failures that I can't repro locally

what is the coverage failure?

I get a log line like

22:20:50.31 [WARN] Failed to generate coverage data for some/package/test/python/my/test.py:tests

interesting… do all targets report that warning, or only some of them? and to confirm, you’re not running with

test --debug

, are you?

just one of them. Deterministically. Let me add

--debug

flag and run again

sorry: would not recommend adding

--debug

.

oh no, I'm not adding --debug in CI

do you have a separate

toml

config for CI by any chance?

yes

does it set

[test] debug

?

no

[test]
use_coverage = true

[coverage-py]
report = ["json", "html"]
global_report = true

ok. since it’s just one target, i do suspect something fishy related to test targets depending on other test targets

is there a query I can run to find test depending on tests? Bazel style? 😂

not exactly 😃 …

./pants dependees $target

should tell you if that target is being pulled in elsewhere

$ ./pants filter --target-type=python_tests :: | xargs ./pants dependees             13s
16:39:26.19 [INFO] Initialization options changed: reinitializing scheduler...
16:39:26.89 [INFO] Scheduler initialized.
16:39:28.07 [INFO] Initialization options changed: reinitializing scheduler...
16:39:29.14 [INFO] Scheduler initialized.

No such targets found. One more interesting observation, the ordering of files in the test report is different in CI and locally, when I run the same command.

mm, that’s not ideal.

Local is Mac, and CI is Ubuntu 18

good idea. you can pass

test --output=all

(or set

[test] output = "all"

) to have the tests dump their output, which might include some more information about why the

.coverage

file wasn’t written

While I wait for CI, just wanna get a sanity check on my test script. Maybe I wrote something silly

./pants -l=error filter --filter-target-type=python_tests :: | sort > /tmp/all_tests

total_tests=$(cat /tmp/all_tests | wc -l)

total_blocks=$BUILDKITE_PARALLEL_JOB_COUNT
if test -z "$total_blocks"; then
    total_blocks=1
fi
current_block=$BUILDKITE_PARALLEL_JOB
if test -z "$current_block"; then
    current_block=0
fi

block_size=$(((total_tests/total_blocks)+1))

size as well
split -l $block_size -a 1 /tmp/all_tests test_target_set_

extra_space=$((block_size*total_blocks-total_tests))
empty_blocks=$((extra_space/block_size))

alphabet=({a..z})
block_index=${alphabet[current_block]}

# Skip empty blocks
if [ "$current_block" -ge "$((total_blocks-empty_blocks))" ]; then
    exit 0
fi

tests_to_run=$(cat test_target_set_$block_index | tr "\n" " ")

./pants test --output=all $tests_to_run

do you have the full Pants run for that?

Do you expect the tests to be using lots of memory?

Ohhhhhh, maybe. Its loading BERT, LOL

might be worth playing with https://www.pantsbuild.org/docs/using-pants-in-ci#tuning-resource-consumption-advanced also

I see. I guess, is there a way to print OOM killed instead of just exiting -9? I'd be happy to contribute code for this

Hmmm maybe, yeah. You can't get Pytest itself to do that, it gets killed abruptly without being able to clean up The exit code line comes from https://github.com/pantsbuild/pants/blob/824b8d70dbdfb86cc89b1bebfaea9711fca4a715/src/python/pants/core/goals/test.py#L127. Maybe we should augment it to detect exit code's of 9 and -9? Or in general when running any Process, detect that exit code and always log a warning? That would catch things like resolving requirements, running a linter, etc, rather than just running the test I'm not sure, always tricky to know when to add special casing like this vs. when it's distracting, including the risk of leading people astray. What do you think?

Let me read a bit on linux OOM killer. Maybe there is some stable interface we can depend on to print OOM killed?

Negative exit codes are always signals. So we could convert that into something like "received signal SIG$X"

so

exit -x

is -->

kill -x

?

Yes, although it looks like this is a bit process-API specific. I believe it's always true in our usage of the Rust process APIs.

I see, I was just reading about

128 + SIG

as the standard convention

nice! I'll open an issue and send a PR. + a test that make sure

SIGINT

is correctly caught and surfaced as

Neg(sig)

<-- does this suffice?

Yea, maybe... I just realized that it might not be iron clad, because when we remote execute a process, we won't be able to get the signal. So maybe the better thing to do would be to change the rust code near where I linked above to append some information about the signal to stderr...

I see. so ?

exit_status.signal().map(tee(print)).map(Neg::neg))

But do feel free to file the issue: if you're interested in fixing it I can add more info about how. If not, I can take a look.

I don't rust at all. Just guess there has to be some thing like

tee

To clarify my understanding of this entire thread, when you say "parallelize tests in buildkite" do you mean sharding across multiple buildkite machines? Or running tests in parallel on a single machine? Pants will happily take care of the latter