Directory specs question #2: do we privilege recursive semantics by default? 🧵

My vote is for a directory spec to not be recursive without some additional sign (

::

 or 

…

) that the user wants recursion. Otherwise, we will likely need a way to signify not recursing, for example, when the user just wants to run a test in a single Go package.  (Also, note that Go is not recursive by default.)

I agree with Tom on not defaulting to recursion. I think it's sensible for Pants to default to doing the minimum work possible, such as only running tests for that directory If you do want to do more work, then add 2-3 extra chars like

::

or

...

cc @happy-kitchen-89482

Every Python tool we use interprets

act on this dir

as

act recursively on this dir and all subdirs

as does

git

. but probably no

go

or

JVM

tooling does

There are definitely cases where acting recursively is the obvious thing to do (

tailor

for example)

…less sure about the JVM. but tools all end up invoked by the build tool, so we have our choice of semantics there probably.

Note that we have a way to signify non-recursion: A single

:

What would you say those cases are? I agree with

tailor

for sure. Others don't seem clear to me. I think it's really useful to say only format

src/python/pants

, without recursing into all ~200 files under it. I think it would be more useful to default to current dir, and have 2-3 chars to signify recursion

And having Pants behave differently in different contexts is, I dunno, weird

i agree.

Well and we can make it more convenient than

"path/to/**"

, either by "fixing"

::

to do the right thing with generated targets, or by adding

...

for example

so then consensus is on

dir

being equivalent to

dir:

, yea? and the other thread suggests that

dir:

should match generated targets

I agree with the consensus that

dir

does not recurse by default I'm trying to figure out the connection between

dir

and `dir:`/`dir::`... One crucial nuance. Even if you completely ignore generated targets,

dir::

only matches targets defined in

dir

and below. Whereas file arguments match targets that "own" files even if those targets live are defined in ancestor directories. I think taht directory specs probably should also match ancestor targets. Does that make sense?

Yeah, people are thinking about files, not targets

Great, I agree! So then, that would mean these semantics: • `f.ext`: all targets that own

f.ext

• `dir`: all targets that own files in

dir/*

, and (per the other thread) also all targets that are defined in

dir

(including generated targets that ~live in that dir) •

dir:

, all targets that are defined in

dir

(including generated targets that ~live in that dir) •

dir::

, all targets that are defined in

dir

or subdirs (including gneerated targets that ~live in those dirs) So,

dir:

is a subset of

dir

. The generated targets thing is a bug fix we have to make, which was impacting file targets too. Fixing it means that

./pants test src/go/dir:

will test targets even if they were generated from

src/go:mod

dir::

, all targets that are defined in dir or subdirs (including generated targets that ~live in those dirs)

there is a lot hanging on the “~live” bit… but yea. i think it’s clear what we want to happen there.

I think this gets to what @happy-kitchen-89482 wants! Almost always, you can indeed use directory and file specs! What's really crucial to that is

dir

including file-less targets, per the other thread. Otherwise

./pants package dir

wouldn't build

go_binary

targets etc

yea. but actually having files have file addresses continues to be really handy, and helps with this case.

Agreed Oh so one other open question with the semantics outlined in https://pantsbuild.slack.com/archives/C0D7TNJHL/p1633982930218400?thread_ts=1633981409.210200&cid=C0D7TNJHL Do we want a recursive directory syntax? As noted there,

dir:

is different than

dir

because it does not include targets defined in ancestor BUILD files who "own" files in

dir

.

dir::

would be different than

dir/...

in the same way And

"dir/**/*"

is also not the same thing as

dir/...

, given that file arguments don't match file-less targets

So

dir::

does not incude files whose targets are in a parent?

Correct, and it has always been that way since Pants 1.x

that’s looking more and more like a bug… but i thought that’s what you meant by

dir::

, all targets that are defined in dir or subdirs (including generated targets that ~live in those dirs)

…? it doesn’t feel like we need separate

dir/…

syntax if

dir::

does that…?

yeah file glob syntax ("dir/**/*") isn't very useful, agreed. We should probably at least stop documenting it once we have dir specs

it doesn’t feel like we need separate dir/… syntax if dir:: does that…?

Indeed, depends if we change

dir::

to include ancestor targets that "own" files in the dirs. Unclear to me what the value of the old semantics is, only that it's more precise than

dir/...

(a little bit offtopic. but i don’t think we should expand our syntax (

…

) exclusively for the sake of file-globs)

a little bit offtopic.

Doesn't seem off topic - I think it's very relevant. What do we want

:

and

::

to mean? Do we keep them and add

dir

and

dir/...

with those subtly different ownership semantics? If we keep them and "fix them", why have

dir:

given that

dir

is the same thing?

Something having been some way since v1 is neither here nor there

the purpose of

dir/...

? Imo it's much more useful than

dir::

. It operates on everything that "lives" in

dir

and below, whether that be that the target is defined there, it's generated into there, or it owns a file in there

dir::

is what's weird - why would you only want to operate on what is defined there or generated into there, but not operate on ancestor targets that own files there? Project introspection I think is the only thing I can come up

we should fix

::

rather than adding

…

So then what do we do with

:

? If we "fix"

::

, then we should apply the same fix to

:

, which means

dir

is identical to

dir:

yes. i thought that that was the conclusion of the other thread

Oh I didn't interpret it that way. I thought we agreed that we should "fix" generated targets, but != ancestor targets who include files in the dir (Owners code path)

why wouldn’t we do both?

if we think there's value in this:

you only want to operate on what is defined there or generated into there, but not operate on ancestor targets that own files there

It's unclear to me if there is? For example, we use DescendentAddresses in our plugin code a lot. Do we really want it to be using Owners code path?

Yes, this is what I am getting at. Having

...

just adds confusion.

dir:

should probably be the same as

dir

and

dir::

should be "the targets for all files under `dir`"

Which, at that point, we never document

:

because

dir

and

dir:

would be the same. I think

dir::

generally makes sense to have two colons as it mirrors

**

vs.

*