Hello 👋 Recently, I’ve noticed that a target generator is always a dependent of a target. For example, suppose I have the following files in directory `src`:

src/BUILD
    foo.py
    bar.py

and inside

BUILD

I have the target generator

python_sources(
    name="lib",
    sources=["*.py"]
)

In this case,

src:lib

depends on

src/foo.py:lib

and

src/bar.py:lib

Intuitively, it makes sense that a generator had a dependency on the individual targets. A change to any individual target means a change within the generator. However, there are undesirable consequences. Extending that example, suppose there are also these files in the directory `test`:

test/BUILD
     test_foo.py
     test_bar.py

and inside

BUILD

I have the target generator

python_tests(
    name="test_lib",
    sources=["test_*.py"]
)

like before,

test:test_lib

depends on

test/test_foo.py:test_lib

and

test/test_bar.py:test_lib

. Now suppose

test/test_foo.py:test_lib

exclusively depends on

src/foo.py

and

test/test_bar.py:test_lib

exclusively depends on

src/bar.py

and neither

src/foo.py

depends on

src/bar.py

nor the reverse. If I modify

src/foo.py

on

HEAD

and only that file, then I would like to run all tests dependent on

src/foo.py

and all tests dependent on any dependents of

src/foo.py

at any depth. It seems like the right choice for that should be

pants test --changed-since=HEAD --changed-dependents=transitive

but as I noted earlier: •

test/test_foo.py:test_lib

is dependent on

src/foo.py

- this is correct, we want to run this test •

test:test_lib

depends on

test/test_foo.py:test_lib

• Running

test

on

test:test_lib

also runs

test/test_bar.py:test_lib

- this is incorrect In other words, when using

pants test --changed-since=HEAD --changed-dependents=transitive

tests will always run at the granularity of the

python_tests

target generator. Is there a way to make this more granular? Is there a way to exclude generators when recursively finding transitive dependencies? Is there something wrong in how I have unit tests setup?

Huh, good catch. I don't have a solution but just commenting to get notifications and also to link to https://github.com/pantsbuild/pants/issues/18888 which is potentially caused by these dependencies.

Hmm, yes. I think the issue may be that a target generator is also a target. We should probably treat it as something more like a macro that generates targets but is not itself part of the dependency graph.

The solution I’ve been using so far has been to use

pants list --changed-since=... --changed-dependents=transitive --filter-target-type=python_test --list-output-file=spec_file
pants --spec-files=spec_file test

Are you using

[python-infer].imports = false

? https://www.pantsbuild.org/docs/reference-python-infer#imports I would think that this: https://github.com/pantsbuild/pants/blob/bfb795036e13fea5512dfa8bc9e7925972f9f7ba/src/python/pants/backend/python/target_types_rules.py#L81-L86 would avoid this issue by default… will need to look at it a little more to be sure.

> Are you using

[python-infer].imports = false

? No, in this example, I believe that inferring imports allows the dependency of

test/test_foo.py:test_lib

on

src/foo.py

to be detected. The problem is the dependency of

test:test_lib

on

test/test_foo.py:test_lib

ah, right yes of course. So I think in some cases this makes sense, when you want to treat a collection of (source or other) targets as one unit you can add a dependency to in order to get them all. But when relying on granular dependency inference, it may not be. So it seems like we’d want a new target field to be able to toggle this behaviour.

> So it seems like we’d want a new target field to be able to toggle this behaviour. Is this something that should be set on the target generator definition? Or more like a flag that’s passed when resolving dependencies - e.g. something like

--changed-generators=direct

so you could run

pants test --changed-since=HEAD --changed-dependents=transitive --changed-generators=direct

so that generators are only transitive if they were directly changed?

I imagine this to be a property of the target generator, as it may be confusing otherwise I think.