https://pantsbuild.org/ logo
#development
Title
# development
w

wooden-thailand-8386

11/06/2020, 6:25 PM
Hey there! I was asking @hundreds-father-404 earlier about this - https://github.com/pantsbuild/pants/issues/11050 and it’s a better idea to bring the discussion here. For my use case I’d love to be able to get a file/folder directly when running a command such as
./pants --changed-since ... filter --target-type=files --granularity=target
and that would bring me only:
Copy code
services-terraform/docs
instead of (the current results when using filter)
Copy code
services/terraform/docs
services/terraform/docs/backend.tf
services/terraform/docs/main.tf
services/terraform/docs/outputs.tf
services/terraform/docs/variables.tf
services/terraform/docs/versions.tf
and with that I can use it with
terraform
or
docker
(when I need to know which Dockerfile) on my CI. Eric then pointed out that what @witty-crayon-22786 is proposing is a slightly different proposal than what I may want:
Stu’s proposal still will be printing Pants addresses every time:
* services/terraform/docs is a “base” address, and shorthand for service/terraform/docs:docs
* services/terraform/docs:tests is a “base” address
* services/terraform/docs/backend.tf is a “file” address, and shorthand for services/terraform/docs/backend.tf:docs, meaning, that the original metadata comes from the services/terraform/docs:docs target
* services/terraform/docs/backend.tf:tests is a “file” address, meaning the original metadata comes from services/terraform/docs:tests
So, with Stu’s proposal, you’d often get things that look like just file names or just directories, but you could still get services/terraform/docs:tests and services/terraform/docs/backend.tf:tests in the mix
We ended up discussing that maybe this wouldnt be an “issue” if we already had plugins availables for tools such as
docker
,
helm
and
terraform
and that
plugins
is a way forward but it would be nice if we could also support a better “piping” support so it could be used together with those other tools. I had a very simplistic idea of maybe having another target such as
output
or
pipe
that would bring me what I need the way I want. Eric said:
I think this could be as simple as adding a flag like --raw to list, filter, dependencies, and dependees. Rather than outputting addresses, it will always output file names and/or directories
That’s a very rough summary of the conversation and would love to have everyone’s thoughts 🙂
a

average-vr-56795

11/06/2020, 6:48 PM
I don’t think I fully understand the use-case, but is there a reason to do this in pants, instead of outside (assuming there was a flat to return target addresses instead of files, which there at least used to be…)?
Copy code
./pants --changed-since ... filter --target-type=files --granularity=target | cut -d: -f1 | sort | uniq
w

wooden-thailand-8386

11/06/2020, 7:07 PM
yeah so my current solution is similar to what you posted but using groovy (Jenkinsfile) it’s doable and fine but can feel a little bit “hacky” when you’re trying to get something as specific as a Dockerfile and there’s like 20 other files inside the same folder for each one of the projects.
a

average-vr-56795

11/06/2020, 7:10 PM
The correct-file filtering and narrowing-to-a-directory feel like orthogonal problems? How are you doing the just-find-my-Dockerfile filtering at the moment?
w

wooden-thailand-8386

11/06/2020, 7:57 PM
just filtering it using a groovy function that removes everything but strings with
Dockerfile
in it
e

enough-analyst-54434

11/06/2020, 8:00 PM
Is using
pants
as the middleman here required at all? Can you just use
git
?
w

wooden-thailand-8386

11/06/2020, 8:06 PM
I think I need
pants
for this one bc those Dockerfiles show up only when I use
--changed-dependees=transitive
👍 1
w

witty-crayon-22786

11/06/2020, 8:27 PM
So, with Stu’s proposal, you’d often get things that look like just file names or just directories, but you could still get services/terraform/docs:tests and services/terraform/docs/backend.tf:tests in the mix
i’d say that the point of the proposal on #11050 is to be able to effectively say “show me only files” or “show me only targets”
h

hundreds-father-404

11/06/2020, 8:28 PM
I don’t think I fully understand the use-case, but is there a reason to do this in pants, instead of outside?
Indeed, it’s possible to post-process results. But the idea here is to add better first-class support for piping from Pants to external tooling. Map from the world of targets to files/directories
w

witty-crayon-22786

11/06/2020, 8:28 PM
the terminology is still up in the air, so the word “base” might be confusing
h

hundreds-father-404

11/06/2020, 8:29 PM
“show me only files”
This isn’t quite it, iiuc. They are “file” addresses, not files. Often they look like files, but not always, e.g.
services/terraform/docs/backend.tf:tests
I think there are two proposals here: 1. Your proposal to have
filter --granularity={target,file}
2. A
--raw
flag, which rather than loses the target info and gives back directories/files We’d need a combo of both for Thale’s use case.
e

enough-analyst-54434

11/06/2020, 8:32 PM
@wooden-thailand-8386 your use case seems under-specified.
./pants --changed-since ... filter --target-type=files
is ===
git diff --name-only ...
h

hundreds-father-404

11/06/2020, 8:32 PM
Except that he needs
--changed-dependees=transitive
, which git doesn’t have on its own. Plus filtering by target type
e

enough-analyst-54434

11/06/2020, 8:33 PM
OK - that but is not in the original post or maybe I missed it.
w

witty-crayon-22786

11/06/2020, 8:33 PM
e

enough-analyst-54434

11/06/2020, 8:35 PM
What is a dependee of a files target?
That makes ~no sense.
Where the dependee is a Dockerfile.
It would be lovely if the problem was spelled out in more detail in one-non-tranisent sport likean issue. The current issue seems more like a conclusing than a detailed problem description.
w

witty-crayon-22786

11/06/2020, 8:39 PM
@wooden-thailand-8386: i think that this problem will still be an issue even if:
We ended up discussing that maybe this wouldnt be an “issue” if we already had plugins availables for tools such as 
docker
helm
 and 
terraform
 and
that 
plugins
 is a way forward but it would be nice if we could also support a better “piping” support so it could be used together with those other tools.
as @enough-analyst-54434 said, it’s odd that the final thing you’re trying to match on is a
files
target rather than a
docker
target, for example… but it’s still the case that if what you want out of your query is “the files (not targets) that were affected transitively by a changed file”, then we need to answer the question of how to filter to just files, regardless of the target type that owns them.
👍 1
a

average-vr-56795

11/06/2020, 8:47 PM
@witty-crayon-22786 I’m not sure why that query language would need to live in pants, though? I get that in other cases, where target selection is being done, it’s useful to keep it all in one graph execution to maximise parallelism, but:
./pants --changed-since ... filter --target-type=files --granularity=target | ./pants list-files
seems like it’d be a pretty reasonable solution to this problem?
h

hundreds-father-404

11/06/2020, 8:49 PM
./pants list-files
(I think that’s basically
./pants filedeps
, without the BUILD files included)
w

witty-crayon-22786

11/06/2020, 8:49 PM
--granularity=target
doesn’t exist currently, but yea, potentially. i also don’t think it’s strictly necessary for the command you showed though… the second command can consume either type
@average-vr-56795: maybe not obvious here is that all of these are valid addresses/CLI specs (when we added file addresses, we gave them a unified syntax)… so the granularity flag you included isn’t necessary in that command
it would only be necessary if you didn’t want to pipe and wanted the first command to filter to just files
https://pantsbuild.slack.com/archives/C0D7TNJHL/p1604695789290300?thread_ts=1604687104.284700&cid=C0D7TNJHL not quite… filedeps expands a level… so if you passed it a file it would find the deps of the file
w

wooden-thailand-8386

11/06/2020, 8:56 PM
Sure, @enough-analyst-54434 I’m sorry if it wasn’t clear. Like I said I had a more detailed conversation with Eric before putting the rough summary here. I have a project that when something changes I’ll have my CI build it and create an
archive
that contains the .pex file and some loose files there… the next step I’ll move that
.tar.gz
to a folder with a
Dockerfile
and build that image. The thing is, I’m not even using
docker build
directly inside my pipeline, it’s a company’s rule thing so I have to pass where that Dockerfile lives inside the repo and they build/do whatever they do (including pushing it to our internal registry). I solved that issue by asking here and someone else also had a similar issue and he suggested that I’d simple declare a `files`that contains that
Dockerfile
and add the
archive
/
pex_binary
as a dependency of that
files
. Semantically it doesn’t make sense, it definitely should be
docker
target or something like that but when I do it this way, if something changes on my project I can then just use ./pants with the tag
--build
and that’ll return the files that I need.. I do the same thing for everything else: my terraform modules and helm charts.
It might not make sense, like you said, but that’s the workaround I had to do in order to accomplish things here at the company using Jenkins and company’s weird constraints.
e

enough-analyst-54434

11/06/2020, 9:00 PM
OK, thanks. There is still not enough detail exposed here to make intelligible suggestions, but understood that you have difficult constraints. I'd just encourage writing down exactly what you need to accomplish in detail whenever possible because often a good answer is to question your framing of the question altogether. That's hard to do without full details.
w

witty-crayon-22786

11/06/2020, 10:24 PM
@wooden-thailand-8386: I'll follow up on that ticket to flesh out the alternatives a bit more
I believe that I understand the problem.
👍 1
Is this blocking you, or do you have a workaround for now?