https://pantsbuild.org/ logo
Join Slack
Powered by
I notice that `src/python/pants/init/load_backends...
# development
h
I notice that 
src/python/pants/init/load_backends_integration_test.py
is very slow (almost 400 seconds on my machine, almost 700 in CI) and it causes its CI shard to lag considerably.
And it’s not surprising, considering it runs the solver dozens of times, once per backend.
Seems like this could run nightly or post-merge, and not have every CI run gated on it
w
Isn't this what we were discussing on Monday? The tradeoff between "faster CI" and "what if it breaks for someone else"? Similar in concept to the --changed-since + --transitive
Copy code
@pytest.mark.parametrize("backend", discover_backends())
def test_each_distinct_backend_loads(backend) -> None:
    """This should catch graph incompleteness errors, i.e. when a required rule is not
    registered."""
    # The `typescript` backend uses rules from the `javascript` backend, and it therefore
    # should be loaded together with it for the relevant rules to be discovered.
    if "typescript" in backend:
        backend = ["pants.backend.experimental.javascript", "pants.backend.experimental.typescript"]
    else:
        backend = [backend]
    assert_backends_load(backend)
I don't like that one bit
f
We could move the heavyweight tests to that process in addition to any other (periodic or aggregate) settings it should run in.
h
It is related, but this will always run under any changes, since it depends on all of Pants. But this is an example of a test that is a good backstop to have but is unlikely to break in any specific change.
I guess we could finesse this specific test to only run on a backend when it has had code changes, and I guess run on a sampling of backends when core changes?
f
longer term idea (not for now): Pants is able to ask pytest for the test names in each file and can understand test-level dependencies inside a file. for now: create a subpackage with one file per backend with boilerplate to just call a common test helper with the name of the backend to test
and some magic or boilerplate in the BUILD file to set up the proper dependencies for each 
python_test
target
Then Pants will "know" that there is a test for each backend and can invalidate at the file level as usual.
w
Yeah, to Tom's point - wouldn't it make more sense for there to be a per-backend integration test to confirm this - which would let us be fancier about which backends load what
👆 1
b
For that specific test, could we split that test into small parts (but still run all the variants it currently runs)? Aiming to get more parallelism/make it not such a long pole. Eg shard it, or run it under xdist?
p
That test has been very helpful when introducing a new backend. It caught several of my screw ups 😛
Of course, we don't introduce a new backend every day, so ...
I like the idea of splitting the test up so that it can be cached per-file.
Could we do some kind of code-generation here to generate the per-backend test files on the fly?
f
Similar idea: Would the following work? Use a BUILD macro to generate multiple 
python_test
targets for a single source file but each with a different 
test_extra_env
with the backend to test in an environment variable.
So you end up with multiple `python_test`targets so the target-level caching should work, but you don't need to generate or create multiple files.
😎 1
> target-level caching should work Caching is per target and not per file? It is usually only per file only because 
python_tests
makes one 
python_test
per test file?
p
What a fascinating idea! I hope someone has time to experiment with this.
9 Views
Open in Slack
PreviousNext
Powered by