https://pantsbuild.org/ logo
h

hundreds-father-404

10/11/2021, 7:43 PM
Directory specs question #2: do we privilege recursive semantics by default? 🧵
I think we probably want to support both...they are both really useful! Which is the default when you say
dir
, vs. having some new syntax?
@fast-nail-55400 proposed copying Go for recursive semantics,
dir/...
Maybe we stick with
dir::
and fix the weirdness with address specs and generated targets so that does what the user expects
f

fast-nail-55400

10/11/2021, 7:44 PM
My vote is for a directory spec to not be recursive without some additional sign (
::
 or 
) that the user wants recursion. Otherwise, we will likely need a way to signify not recursing, for example, when the user just wants to run a test in a single Go package.  (Also, note that Go is not recursive by default.)
👍 2
proposed copying Go for recursive semantics, 
dir/...
To clarify, I had pointed it out as a type of syntax signifying recursion, but I don’t have a particular care whether that is the syntax or not.
1
h

hundreds-father-404

10/11/2021, 7:45 PM
I agree with Tom on not defaulting to recursion. I think it's sensible for Pants to default to doing the minimum work possible, such as only running tests for that directory If you do want to do more work, then add 2-3 extra chars like
::
or
...
Also Tom and I discussed how
./pants tailor
is recursive, which works great! And we should probably keep. But I think that
tailor
and
test
have very different use cases, and we can have the semantics diverge possibly. If we don't want to diverge, privilege `test`/`list`/`lint` etc over
tailor
w

witty-crayon-22786

10/11/2021, 7:50 PM
cc @happy-kitchen-89482
h

happy-kitchen-89482

10/11/2021, 7:50 PM
Every Python tool we use interprets
act on this dir
as
act recursively on this dir and all subdirs
w

witty-crayon-22786

10/11/2021, 7:51 PM
as does
git
. but probably no
go
or
JVM
tooling does
👍 1
h

happy-kitchen-89482

10/11/2021, 7:52 PM
There are definitely cases where acting recursively is the obvious thing to do (
tailor
for example)
Others are more ambiguous, and depend on user expectations
w

witty-crayon-22786

10/11/2021, 7:52 PM
…less sure about the JVM. but tools all end up invoked by the build tool, so we have our choice of semantics there probably.
h

happy-kitchen-89482

10/11/2021, 7:53 PM
Note that we have a way to signify non-recursion: A single
:
This is tricky
In some contexts users will expect one thing, in others another
h

hundreds-father-404

10/11/2021, 7:53 PM
What would you say those cases are? I agree with
tailor
for sure. Others don't seem clear to me. I think it's really useful to say only format
src/python/pants
, without recursing into all ~200 files under it. I think it would be more useful to default to current dir, and have 2-3 chars to signify recursion
h

happy-kitchen-89482

10/11/2021, 7:53 PM
And having Pants behave differently in different contexts is, I dunno, weird
1
What you're suggesting will be unintuitive to anyone who uses the standalone Python tools
Their expectation will absolutely be that recursing is the behavior
But then Go users will not, so...
Gah
On balance "how do I specify recursion" is a slightly less weird thing for a user to ponder than "how do I specify not-recursion"
💯 1
1
E.g.,
"path/to/**"
is understood by users, and would actually work as long as they quoted the glob so the shell didn't expand it
So probably recursion should not be the default
👍 1
Annoying as that will be in some cases
w

witty-crayon-22786

10/11/2021, 7:57 PM
i agree.
👍 1
h

hundreds-father-404

10/11/2021, 7:57 PM
Well and we can make it more convenient than
"path/to/**"
, either by "fixing"
::
to do the right thing with generated targets, or by adding
...
for example
w

witty-crayon-22786

10/11/2021, 7:59 PM
so then consensus is on
dir
being equivalent to
dir:
, yea? and the other thread suggests that
dir:
should match generated targets
h

hundreds-father-404

10/11/2021, 8:01 PM
I agree with the consensus that
dir
does not recurse by default I'm trying to figure out the connection between
dir
and `dir:`/`dir::`... One crucial nuance. Even if you completely ignore generated targets,
dir::
only matches targets defined in
dir
and below. Whereas file arguments match targets that "own" files even if those targets live are defined in ancestor directories. I think taht directory specs probably should also match ancestor targets. Does that make sense?
h

happy-kitchen-89482

10/11/2021, 8:05 PM
Yeah, people are thinking about files, not targets
The target is just "where this file happens to get its metadata"
👍 1
h

hundreds-father-404

10/11/2021, 8:08 PM
Great, I agree! So then, that would mean these semantics: • `f.ext`: all targets that own
f.ext
• `dir`: all targets that own files in
dir/*
, and (per the other thread) also all targets that are defined in
dir
(including generated targets that ~live in that dir) •
dir:
, all targets that are defined in
dir
(including generated targets that ~live in that dir) •
dir::
, all targets that are defined in
dir
or subdirs (including gneerated targets that ~live in those dirs) So,
dir:
is a subset of
dir
. The generated targets thing is a bug fix we have to make, which was impacting file targets too. Fixing it means that
./pants test src/go/dir:
will test targets even if they were generated from
src/go:mod
w

witty-crayon-22786

10/11/2021, 8:09 PM
dir::
, all targets that are defined in dir or subdirs (including generated targets that ~live in those dirs)
there is a lot hanging on the “~live” bit… but yea. i think it’s clear what we want to happen there.
h

hundreds-father-404

10/11/2021, 8:10 PM
I think this gets to what @happy-kitchen-89482 wants! Almost always, you can indeed use directory and file specs! What's really crucial to that is
dir
including file-less targets, per the other thread. Otherwise
./pants package dir
wouldn't build
go_binary
targets etc
there is a lot hanging on the ~live
Agreed. Does this make sense for implementation? Generated targets gain a new internal mechanism to say what directory/file they "live in". Even though
src/go#./dir
really is defined in
src/go
, it can indicate via a property that it ~represents/~lives/~adopts the dir
src/go/dir
Our Specs code will generate all candidate targets and inspect that property to determine it should match I think this is what you've been saying, that we need to change Specs code, not necessarily other places. I'm trying to avoid having to change the Address syntax. I really like that
generated_name
can be anything you want, such as a file-less name like
#Django
vs. a file name like
#foo.py
vs.
./go_dir
w

witty-crayon-22786

10/11/2021, 8:14 PM
yea. but actually having files have file addresses continues to be really handy, and helps with this case.
how likely is a world where only generated targets which don’t have associated files use the generated target syntax?
probably a topic for another thread.
👍 1
h

hundreds-father-404

10/11/2021, 8:20 PM
Agreed Oh so one other open question with the semantics outlined in https://pantsbuild.slack.com/archives/C0D7TNJHL/p1633982930218400?thread_ts=1633981409.210200&cid=C0D7TNJHL Do we want a recursive directory syntax? As noted there,
dir:
is different than
dir
because it does not include targets defined in ancestor BUILD files who "own" files in
dir
.
dir::
would be different than
dir/...
in the same way And
"dir/**/*"
is also not the same thing as
dir/...
, given that file arguments don't match file-less targets
Adding
dir/...
would make address specs less and less useful. The only time to use them is when you want to avoid
Owners
code, to get really precise about which targets to operate on At that point, does it even make sense to support
::
and
:
? (The literal target address probably always makes sense)
h

happy-kitchen-89482

10/11/2021, 9:40 PM
So
dir::
does not incude files whose targets are in a parent?
h

hundreds-father-404

10/11/2021, 9:41 PM
Correct, and it has always been that way since Pants 1.x
w

witty-crayon-22786

10/11/2021, 9:42 PM
that’s looking more and more like a bug… but i thought that’s what you meant by
dir::
, all targets that are defined in dir or subdirs (including generated targets that ~live in those dirs)
…? it doesn’t feel like we need separate
dir/…
syntax if
dir::
does that…?
honestly, i’m not sure that file-glob syntax (
./pants validate "**/*"
) is carrying its weight, given that it basically always needs to be escaped
and if you forget to escape it, for most simple cases the right thing happens anyway.
h

hundreds-father-404

10/11/2021, 9:46 PM
yeah file glob syntax ("dir/**/*") isn't very useful, agreed. We should probably at least stop documenting it once we have dir specs
it doesn’t feel like we need separate dir/… syntax if dir:: does that…?
Indeed, depends if we change
dir::
to include ancestor targets that "own" files in the dirs. Unclear to me what the value of the old semantics is, only that it's more precise than
dir/...
w

witty-crayon-22786

10/11/2021, 9:46 PM
(a little bit offtopic. but i don’t think we should expand our syntax (
) exclusively for the sake of file-globs)
h

hundreds-father-404

10/11/2021, 9:49 PM
a little bit offtopic.
Doesn't seem off topic - I think it's very relevant. What do we want
:
and
::
to mean? Do we keep them and add
dir
and
dir/...
with those subtly different ownership semantics? If we keep them and "fix them", why have
dir:
given that
dir
is the same thing?
h

happy-kitchen-89482

10/11/2021, 11:34 PM
Something having been some way since v1 is neither here nor there
We're discussing new semantics
What is the proposal for
all files in all dirs under this dir
?
...
?
h

hundreds-father-404

10/11/2021, 11:49 PM
the purpose of
dir/...
? Imo it's much more useful than
dir::
. It operates on everything that "lives" in
dir
and below, whether that be that the target is defined there, it's generated into there, or it owns a file in there
dir::
is what's weird - why would you only want to operate on what is defined there or generated into there, but not operate on ancestor targets that own files there? Project introspection I think is the only thing I can come up
w

witty-crayon-22786

10/11/2021, 11:52 PM
we should fix
::
rather than adding
h

hundreds-father-404

10/11/2021, 11:55 PM
So then what do we do with
:
? If we "fix"
::
, then we should apply the same fix to
:
, which means
dir
is identical to
dir:
w

witty-crayon-22786

10/11/2021, 11:56 PM
yes. i thought that that was the conclusion of the other thread
h

hundreds-father-404

10/11/2021, 11:56 PM
Oh I didn't interpret it that way. I thought we agreed that we should "fix" generated targets, but != ancestor targets who include files in the dir (Owners code path)
w

witty-crayon-22786

10/11/2021, 11:57 PM
why wouldn’t we do both?
…i guess that one answer to that is: “because there is a question of whether a file ‘~lives’ in its own directory, or ‘~lives’ in the declaring target’s directory”
h

hundreds-father-404

10/12/2021, 12:00 AM
if we think there's value in this:
you only want to operate on what is defined there or generated into there, but not operate on ancestor targets that own files there
It's unclear to me if there is? For example, we use DescendentAddresses in our plugin code a lot. Do we really want it to be using Owners code path?
i guess that one answer to that is: “because there is a question of whether a file ‘~lives’ in its own directory, or ‘~lives’ in the declaring target’s directory”
Reminder that address specs != "files". They operate solely in the domain of targets/addresses/BUILD files
I suspect there's some value to keeping
::
and
:
, but that the majority of folks wouldn't need it and it would mostly be used for plugin API
Thanks to directory specs including file-less targets, it means that Benjy can make the docs change he's been wanting of basically not documenting address specs beyond a tooltip. But to do that, we do need a recursive directory spec
h

happy-kitchen-89482

10/12/2021, 12:11 AM
Yes, this is what I am getting at. Having
...
just adds confusion.
dir:
should probably be the same as
dir
and
dir::
should be "the targets for all files under `dir`"
Unclear to me what use the old semantics of
dir::
are in a filesystem-centric CLI world
h

hundreds-father-404

10/12/2021, 12:20 AM
Which, at that point, we never document
:
because
dir
and
dir:
would be the same. I think
dir::
generally makes sense to have two colons as it mirrors
**
vs.
*