Hey there! I was asking @hundreds-father-404 earlier about this - https://github.com/pantsbuild/pants/issues/11050 and it’s a better idea to bring the discussion here. For my use case I’d love to be able to get a file/folder directly when running a command such as

./pants --changed-since ... filter --target-type=files --granularity=target

and that would bring me only:

services-terraform/docs

instead of (the current results when using filter)

services/terraform/docs
services/terraform/docs/backend.tf
services/terraform/docs/main.tf
services/terraform/docs/outputs.tf
services/terraform/docs/variables.tf
services/terraform/docs/versions.tf

and with that I can use it with

terraform

or

docker

(when I need to know which Dockerfile) on my CI. Eric then pointed out that what @witty-crayon-22786 is proposing is a slightly different proposal than what I may want:

Stu’s proposal still will be printing Pants addresses every time:

* services/terraform/docs is a “base” address, and shorthand for service/terraform/docs:docs

* services/terraform/docs:tests is a “base” address

* services/terraform/docs/backend.tf is a “file” address, and shorthand for services/terraform/docs/backend.tf:docs, meaning, that the original metadata comes from the services/terraform/docs:docs target

* services/terraform/docs/backend.tf:tests is a “file” address, meaning the original metadata comes from services/terraform/docs:tests

So, with Stu’s proposal, you’d often get things that look like just file names or just directories, but you could still get services/terraform/docs:tests and services/terraform/docs/backend.tf:tests in the mix

We ended up discussing that maybe this wouldnt be an “issue” if we already had plugins availables for tools such as

docker

,

helm

and

terraform

and that

plugins

is a way forward but it would be nice if we could also support a better “piping” support so it could be used together with those other tools. I had a very simplistic idea of maybe having another target such as

output

or

pipe

that would bring me what I need the way I want. Eric said:

I think this could be as simple as adding a flag like --raw to list, filter, dependencies, and dependees. Rather than outputting addresses, it will always output file names and/or directories

That’s a very rough summary of the conversation and would love to have everyone’s thoughts 🙂

I don’t think I fully understand the use-case, but is there a reason to do this in pants, instead of outside (assuming there was a flat to return target addresses instead of files, which there at least used to be…)?

./pants --changed-since ... filter --target-type=files --granularity=target | cut -d: -f1 | sort | uniq

yeah so my current solution is similar to what you posted but using groovy (Jenkinsfile) it’s doable and fine but can feel a little bit “hacky” when you’re trying to get something as specific as a Dockerfile and there’s like 20 other files inside the same folder for each one of the projects.

The correct-file filtering and narrowing-to-a-directory feel like orthogonal problems? How are you doing the just-find-my-Dockerfile filtering at the moment?

just filtering it using a groovy function that removes everything but strings with

Dockerfile

in it

Is using

pants

as the middleman here required at all? Can you just use

git

?

I think I need

pants

for this one bc those Dockerfiles show up only when I use

--changed-dependees=transitive

So, with Stu’s proposal, you’d often get things that look like just file names or just directories, but you could still get services/terraform/docs:tests and services/terraform/docs/backend.tf:tests in the mix

i’d say that the point of the proposal on #11050 is to be able to effectively say “show me only files” or “show me only targets”

I don’t think I fully understand the use-case, but is there a reason to do this in pants, instead of outside?

Indeed, it’s possible to post-process results. But the idea here is to add better first-class support for piping from Pants to external tooling. Map from the world of targets to files/directories

the terminology is still up in the air, so the word “base” might be confusing

“show me only files”

This isn’t quite it, iiuc. They are “file” addresses, not files. Often they look like files, but not always, e.g.

services/terraform/docs/backend.tf:tests

I think there are two proposals here: 1. Your proposal to have

filter --granularity={target,file}

2. A

--raw

flag, which rather than loses the target info and gives back directories/files We’d need a combo of both for Thale’s use case.

@wooden-thailand-8386 your use case seems under-specified.

./pants --changed-since ... filter --target-type=files

is ===

git diff --name-only ...

Except that he needs

--changed-dependees=transitive

, which git doesn’t have on its own. Plus filtering by target type

OK - that but is not in the original post or maybe I missed it.

yea, true. sorry. he mentioned it afterwards: https://pantsbuild.slack.com/archives/C0D7TNJHL/p1604693163286100?thread_ts=1604687104.284700&cid=C0D7TNJHL

What is a dependee of a files target?

@wooden-thailand-8386: i think that this problem will still be an issue even if:

We ended up discussing that maybe this wouldnt be an “issue” if we already had plugins availables for tools such as 

docker

, 

helm

 and 

terraform

 and

that 

plugins

 is a way forward but it would be nice if we could also support a better “piping” support so it could be used together with those other tools.

as @enough-analyst-54434 said, it’s odd that the final thing you’re trying to match on is a

files

target rather than a

docker

target, for example… but it’s still the case that if what you want out of your query is “the files (not targets) that were affected transitively by a changed file”, then we need to answer the question of how to filter to just files, regardless of the target type that owns them.

@witty-crayon-22786 I’m not sure why that query language would need to live in pants, though? I get that in other cases, where target selection is being done, it’s useful to keep it all in one graph execution to maximise parallelism, but:

./pants --changed-since ... filter --target-type=files --granularity=target | ./pants list-files

seems like it’d be a pretty reasonable solution to this problem?

./pants list-files

(I think that’s basically

./pants filedeps

, without the BUILD files included)

--granularity=target

doesn’t exist currently, but yea, potentially. i also don’t think it’s strictly necessary for the command you showed though… the second command can consume either type

Sure, @enough-analyst-54434 I’m sorry if it wasn’t clear. Like I said I had a more detailed conversation with Eric before putting the rough summary here. I have a project that when something changes I’ll have my CI build it and create an

archive

that contains the .pex file and some loose files there… the next step I’ll move that

.tar.gz

to a folder with a

Dockerfile

and build that image. The thing is, I’m not even using

docker build

directly inside my pipeline, it’s a company’s rule thing so I have to pass where that Dockerfile lives inside the repo and they build/do whatever they do (including pushing it to our internal registry). I solved that issue by asking here and someone else also had a similar issue and he suggested that I’d simple declare a `files`that contains that

Dockerfile

and add the

archive

/

pex_binary

as a dependency of that

files

. Semantically it doesn’t make sense, it definitely should be

docker

target or something like that but when I do it this way, if something changes on my project I can then just use ./pants with the tag

--build

and that’ll return the files that I need.. I do the same thing for everything else: my terraform modules and helm charts.

OK, thanks. There is still not enough detail exposed here to make intelligible suggestions, but understood that you have difficult constraints. I'd just encourage writing down exactly what you need to accomplish in detail whenever possible because often a good answer is to question your framing of the question altogether. That's hard to do without full details.

@wooden-thailand-8386: I'll follow up on that ticket to flesh out the alternatives a bit more