https://pantsbuild.org/ logo
Join Slack
Powered by
Directory specs question #2: do we privilege recur...
# development
h
Directory specs question #2: do we privilege recursive semantics by default? šŸ§µ
I think we probably want to support both...they are both really useful! Which is the default when you say 
dir
, vs. having some new syntax?
@fast-nail-55400 proposed copying Go for recursive semantics, 
dir/...
Maybe we stick with 
dir::
and fix the weirdness with address specs and generated targets so that does what the user expects
f
My vote is for a directory spec to not be recursive without some additional sign (
::
Ā orĀ 
ā€¦
) that the user wants recursion. Otherwise, we will likely need a way to signifyĀ notĀ recursing, for example, when the user just wants to run a test in a single Go package.Ā  (Also, note that Go is not recursive by default.)
šŸ‘ 2
proposed copying Go for recursive semantics,Ā 
dir/...
To clarify, I had pointed it out as a type of syntax signifying recursion, but I donā€™t have a particular care whether that is the syntax or not.
āž• 1
h
I agree with Tom on not defaulting to recursion. I think it's sensible for Pants to default to doing the minimum work possible, such as only running tests for that directory If you do want to do more work, then add 2-3 extra chars like 
::
or 
...
Also Tom and I discussed how 
./pants tailor
is recursive, which works great! And we should probably keep. But I think that 
tailor
and 
test
have very different use cases, and we can have the semantics diverge possibly. If we don't want to diverge, privilege `test`/`list`/`lint` etc over 
tailor
w
cc @happy-kitchen-89482
h
Every Python tool we use interprets 
act on this dir
as 
act recursively on this dir and all subdirs
w
as does 
git
. but probably no 
go
or 
JVM
tooling does
šŸ‘ 1
h
There are definitely cases where acting recursively is the obvious thing to do (
tailor
for example)
Others are more ambiguous, and depend on user expectations
w
ā€¦less sure about the JVM. but tools all end up invoked by the build tool, so we have our choice of semantics there probably.
h
Note that we have a way to signify non-recursion: A single 
:
This is tricky
In some contexts users will expect one thing, in others another
h
What would you say those cases are? I agree with 
tailor
for sure. Others don't seem clear to me. I think it's really useful to say only format 
src/python/pants
, without recursing into all ~200 files under it. I think it would be more useful to default to current dir, and have 2-3 chars to signify recursion
h
And having Pants behave differently in different contexts is, I dunno, weird
āž• 1
What you're suggesting will be unintuitive to anyone who uses the standalone Python tools
Their expectation will absolutely be that recursing is the behavior
But then Go users will not, so...
Gah
On balance "how do I specify recursion" is a slightly less weird thing for a user to ponder than "how do I specify not-recursion"
šŸ’Æ 1
āž• 1
E.g., 
"path/to/**"
is understood by users, and would actually work as long as they quoted the glob so the shell didn't expand it
So probably recursion should not be the default
šŸ‘ 1
Annoying as that will be in some cases
w
i agree.
šŸ‘ 1
h
Well and we can make it more convenient than 
"path/to/**"
, either by "fixing" 
::
to do the right thing with generated targets, or by adding 
...
for example
w
so then consensus is on 
dir
being equivalent to 
dir:
, yea? and the other thread suggests that 
dir:
should match generated targets
h
I agree with the consensus that 
dir
does not recurse by default I'm trying to figure out the connection between 
dir
and `dir:`/`dir::`... One crucial nuance. Even if you completely ignore generated targets, 
dir::
only matches targets defined in 
dir
and below. Whereas file arguments match targets that "own" files even if those targets live are defined in ancestor directories. I think taht directory specs probably should also match ancestor targets. Does that make sense?
h
Yeah, people are thinking about files, not targets
The target is just "where this file happens to get its metadata"
šŸ‘ 1
h
Great, I agree! So then, that would mean these semantics: ā€¢ `f.ext`: all targets that own 
f.ext
ā€¢ `dir`: all targets that own files in 
dir/*
, and (per the other thread) also all targets that are defined in 
dir
(including generated targets that ~live in that dir) ā€¢ 
dir:
, all targets that are defined in 
dir
(including generated targets that ~live in that dir) ā€¢ 
dir::
, all targets that are defined in 
dir
or subdirs (including gneerated targets that ~live in those dirs) So, 
dir:
is a subset of 
dir
. The generated targets thing is a bug fix we have to make, which was impacting file targets too. Fixing it means that 
./pants test src/go/dir:
will test targets even if they were generated from 
src/go:mod
w
dir::
, all targets that are defined in dir or subdirs (including generated targets that ~live in those dirs)
there is a lot hanging on the ā€œ~liveā€ bitā€¦ but yea. i think itā€™s clear what we want to happen there.
h
I think this gets to what @happy-kitchen-89482 wants! Almost always, you can indeed use directory and file specs! What's really crucial to that is 
dir
including file-less targets, per the other thread. Otherwise 
./pants package dir
wouldn't build 
go_binary
targets etc
there is a lot hanging on the ~live
Agreed. Does this make sense for implementation? Generated targets gain a new internal mechanism to say what directory/file they "live in". Even though 
src/go#./dir
really is defined in 
src/go
, it can indicate via a property that it ~represents/~lives/~adopts the dir 
src/go/dir
Our Specs code will generate all candidate targets and inspect that property to determine it should match I think this is what you've been saying, that we need to change Specs code, not necessarily other places. I'm trying to avoid having to change the Address syntax. I really like that 
generated_name
can be anything you want, such as a file-less name like 
#Django
vs. a file name like 
#foo.py
vs. 
./go_dir
w
yea. but actually having files have file addresses continues to be really handy, and helps with this case.
how likely is a world where only generated targets which donā€™t have associated files use the generated target syntax?
probably a topic for another thread.
šŸ‘ 1
h
Agreed Oh so one other open question with the semantics outlined in https://pantsbuild.slack.com/archives/C0D7TNJHL/p1633982930218400?thread_ts=1633981409.210200&cid=C0D7TNJHL Do we want a recursive directory syntax? As noted there, 
dir:
is different than 
dir
because it does not include targets defined in ancestor BUILD files who "own" files in 
dir
. 
dir::
would be different than 
dir/...
in the same way And 
"dir/**/*"
is also not the same thing as 
dir/...
, given that file arguments don't match file-less targets
Adding 
dir/...
would make address specs less and less useful. The only time to use them is when you want to avoid 
Owners
code, to get really precise about which targets to operate on At that point, does it even make sense to support 
::
and 
:
? (The literal target address probably always makes sense)
h
So 
dir::
does not incude files whose targets are in a parent?
h
Correct, and it has always been that way since Pants 1.x
w
thatā€™s looking more and more like a bugā€¦ but i thought thatā€™s what you meant by 
dir::
, all targets that are defined in dir or subdirs (including generated targets that ~live in those dirs)
ā€¦? it doesnā€™t feel like we need separate 
dir/ā€¦
syntax if 
dir::
does thatā€¦?
honestly, iā€™m not sure that file-glob syntax (
./pants validate "**/*"
) is carrying its weight, given that it basically always needs to be escaped
and if you forget to escape it, for most simple cases the right thing happens anyway.
h
yeah file glob syntax ("dir/**/*") isn't very useful, agreed. We should probably at least stop documenting it once we have dir specs
it doesnā€™t feel like we need separateĀ dir/ā€¦Ā syntax ifĀ dir::Ā does thatā€¦?
Indeed, depends if we change 
dir::
to include ancestor targets that "own" files in the dirs. Unclear to me what the value of the old semantics is, only that it's more precise than 
dir/...
w
(a little bit offtopic. but i donā€™t think we should expand our syntax (
ā€¦
) exclusively for the sake of file-globs)
h
a little bit offtopic.
Doesn't seem off topic - I think it's very relevant. What do we want 
:
and 
::
to mean? Do we keep them and add 
dir
and 
dir/...
with those subtly different ownership semantics? If we keep them and "fix them", why have 
dir:
given that 
dir
is the same thing?
h
Something having been some way since v1 is neither here nor there
We're discussing new semantics
What is the proposal for 
all files in all dirs under this dir
?
...
?
h
the purpose of 
dir/...
? Imo it's much more useful than 
dir::
. It operates on everything that "lives" in 
dir
and below, whether that be that the target is defined there, it's generated into there, or it owns a file in there 
dir::
is what's weird - why would you only want to operate on what is defined there or generated into there, but not operate on ancestor targets that own files there? Project introspection I think is the only thing I can come up
w
we should fix 
::
rather than adding 
ā€¦
h
So then what do we do with 
:
? If we "fix" 
::
, then we should apply the same fix to 
:
, which means 
dir
is identical to 
dir:
w
yes. i thought that that was the conclusion of the other thread
h
Oh I didn't interpret it that way. I thought we agreed that we should "fix" generated targets, but != ancestor targets who include files in the dir (Owners code path)
w
why wouldnā€™t we do both?
ā€¦i guess that one answer to that is: ā€œbecause there is a question of whether a file ā€˜~livesā€™ in its own directory, or ā€˜~livesā€™ in the declaring targetā€™s directoryā€
h
if we think there's value in this:
you only want to operate on what is defined there or generated into there, but not operate on ancestor targets that own files there
It's unclear to me if there is? For example, we use DescendentAddresses in our plugin code a lot. Do we really want it to be using Owners code path?
i guess that one answer to that is: ā€œbecause there is a question of whether a file ā€˜~livesā€™ in its own directory, or ā€˜~livesā€™ in the declaring targetā€™s directoryā€
Reminder that address specs != "files". They operate solely in the domain of targets/addresses/BUILD files
I suspect there's some value to keeping 
::
and 
:
, but that the majority of folks wouldn't need it and it would mostly be used for plugin API
Thanks to directory specs including file-less targets, it means that Benjy can make the docs change he's been wanting of basically not documenting address specs beyond a tooltip. But to do that, we do need a recursive directory spec
h
Yes, this is what I am getting at. Having 
...
just adds confusion. 
dir:
should probably be the same as 
dir
and 
dir::
should be "the targets for all files under `dir`"
Unclear to me what use the old semantics of 
dir::
are in a filesystem-centric CLI world
h
Which, at that point, we never document 
:
because 
dir
and 
dir:
would be the same. I think 
dir::
generally makes sense to have two colons as it mirrors 
**
vs. 
*
Open in Slack
PreviousNext
Powered by