https://pantsbuild.org/ logo
Join Slack
Powered by
Target generators: I'm thinking about when to use ...
# plugins
p
Target generators: I'm thinking about when to use a target generator vs only supporting the singular target. Target generators seem to expand along 1-dimension (1 field): • 
python_sources
expands 
sources
=> 
source
on 
python_source
• 
shell_sources
expands 
sources
=> 
source
on 
shell_source
• 
files
expands 
sources
=> 
source
on 
file
• 
resources
expands 
sources
=> 
source
on 
resource
• 
pex_binaries
expands 
entry_points
=> 
entry_point
on 
pex_binary
A target generator that expands along multiple fields by pulling info from a file and using other fields to refine the details: • 
python_requirements
expands 
source
(a file) and 
*module_mapping
=> 
requirements
and 
modules
on 
python_requirement
But, that pattern is only useful if you are pulling details from another file. What if I wanted to create a target generator that expands 2 fields (without pulling anything from another file)? Looking through the targets, the closest I see to what I'm trying to do is 
relocated_files
, but that's not actually a target generator. It has 3 key fields: 
files_targets
, 
src
, 
dest
. Basically, I'm creating several targets where people define 
src
and 
dest
, where 
dest
is the path that a system package (rpm/deb/...) would install the file. But there are several other fields that will probably be the same across many of the targets: 
file_mode
, 
file_owner
, 
file_group
, 
file_mtime
, so a target generator seems like a great way to have, say, one target generator for all of the executable files, and another for many non-executable files, and another for the config files that should be owned by root, etc. But, I'm not sure about the UX of having several src/dest field pairs defined on the generator. Any ideas?
b
Would a macro be a better choice?
c
First, I think the 
python_requirements
example is just another example like the previous ones: 
python_requirements
expands 
source
=> 
requirements
on 
python_requirement
one target per requirement line from the source file. The module mapping is just configuration on top of that, it doesn’t affect what targets are generated.
But, I’m not sure about the UX of having several src/dest field pairs defined on the generator.
Do you have a brief example of what this could look like? To me I don’t directly see an issue with having a target generator use multiple fields.
p
Would a macro be a better choice?
This is for a plugin that will go in pants, so an actual target is preferable I think. Though I suppose it could be implemented as a macro instead. 🤔
b
I couldn't say for you, just a question that should be asked and answered 🙂
p
This is for the nfpm plugin: https://github.com/pantsbuild/pants/pull/18914 target: 
nfpm_content_symlink
target generator: 
nfpm_content_symlinks
This one is does not have any source files in the workspace - the list of symlinks gets embedded in the rpm, deb, etc so that the packager creates the symlink when the nfpm-generated-package is installed.
Copy code
nfpm_content_symlinks(
    symlinks=[
        ("/actual/path/src1", "/symlink/path/dest1"),
        ("/actual/path/src2", "/symlink/path/dest2"),
        ("/actual/path/src3", "/symlink/path/dest3"),
    ],
    file_owner="root",
    file_group="root",
    file_mtime="...",  # a timestamp string
)
c
seems perfectly fine setup with generator here, imo. There’s a concept on generators of “moved” fields, that are fields that don’t have a real use on the generator itself, but merely acts as a place holder to be “moved” onto the generated targets. So the various 
file_
fields seems like they could be in that category.
p
cool. Is the 
symlinks
field as a list of tuples confusing?
c
not to me.. but I’m perhaps not the best judge on what is confusing in this regard 😛
just toying with an alternative representation for symlinks:
Copy code
nfpm_content_symlinks(
    symlinks={
        "/actual/path/src1": "/symlink/path/dest1",
        "/actual/path/src2": "/symlink/path/dest2",
        "/actual/path/src3": "/symlink/path/dest3",
    },
    file_owner="root",
    file_group="root",
    file_mtime="...",  # a timestamp string
)
both could be easily supported (if desired) as the latter is just 
dict(symlinks)
of the former
p
oh. I like that. It implicitly prevents duplicates... oh. But we would need to prevent duplicate dest (symlink path), the same src could easily have multiple symlinks pointing to it.
c
that could be a runtime check (or inverse the key/value, but that might be confusing 😬)..
p
Copy code
nfpm_content_symlinks(
    symlink_srcs={
        "/symlink/path/src1": "/actual/path/dest1",
        "/symlink/path/src2": "/actual/path/dest2",
        "/symlink/path/src3": "/actual/path/dest3",
    },
    file_owner="root",
    file_group="root",
    file_mtime="...",  # a timestamp string
)
c
about avoiding duplicates, that’s really not the best place to have in the generator, as you may provide the individual targets as well directly in the build file, and then that “check” is side-stepped
at least, it shouldn’t be the only place that is checked
p
yeah. List of tuples seems clearest to me.
👍 1
I've got an appointment in a few minutes, and then I'll put together another example, this time with actual source files (ie not symlinks).
target: 
nfpm_content_config
target generator: 
nfpm_content_configs
Copy code
nfpm_content_configs(
    configs=[
        ("src1.conf", "/etc/dest1.conf"),
        ("sysconfig/src2.conf", "/etc/sysconfig/dest2.conf"),
        (":src3", "/etc/foo/dest3.conf"),
    ],
    noreplace=True,  # bool only used for rpm
    file_owner="root",
    file_group="root",
    file_mode=0700,
    file_mtime="...",  # a timestamp string
)
Copy code
nfpm_content_config(
     src="src1.conf",  # an address or path
     dest="/etc/dest1.conf",
     noreplace=True,  # bool only used for rpm
     file_owner="root",
     file_group="root",
     file_mode=0700,
     file_mtime="...",  # a timestamp string
 )
Here 
src
is working as a combined 
source
and 
dependency
field. Either it owns the source files or it depends on the target that provides it. How confusing is that? (Is that going to be difficult to implement?)
c
eh.. slightly confusing, yes. so the 
src
value could be either a source glob or a target address?
what if the generator made the distinction at generation time, so you end up with:
Copy code
nfpm_content_config(
     source="src1.conf",  # a path
     dest="/etc/dest1.conf",
...
)
nfpm_content_config(
     dependencies=[":src3"],  # an address
     dest="/etc/foo/dest3.conf",
...
)
p
Though, I suppose 
nfpm_content_configs
could generate 2 targets for each file, a 
file
target and the 
nfpm_content_config
with a dependency on it. Or if a target address is provided, skip the 
file
target. That would mean something like this:
Copy code
nfpm_content_config(
    src=":src_file",  # or call this "dependency"instead of "src"
    dest="/etc/foo.conf,
    ...
)
Then, the target never directly owns any files, it acts like one of those wrapper targets.
c
that could work too, I would suggest using 
dependencies
though..
for consistency
also if there’s ever a need to infer more dependencies in the future, it would be weird to have many single valued fields for it
p
dependencies
with an assertion that there can only be one and it must be a 
file
target?
c
or accept any, but use only the first 
file
dependency as the source for the config
or limit that there is only a single 
file
dependency
p
also if there’s ever a need to infer more dependencies in the future, it would be weird to have many single valued fields for it
Hmm. Would I need to do something like the 
relocated_files
target then with 
dependencies
and a 
src
field? Because dest would be a single file, so if dependencies pulls in additional files, how do I know which one is the config file?
Oh. Just use the first file dep. That could work.
c
not sure what impact it has, as target generators have a classvar indicating what type of target they generate, there’s no stopping it from generating whatever, not up to speed what it is used for.
p
The JavaScript generator creates multiple target types.
c
yea, I believe I’ve done that too.. just throwing it out there to potentially investigate the purpose for that classvar… (cc @witty-crayon-22786 the purpose of https://github.com/pantsbuild/pants/blob/df749c145fc76f1d8d4d7de943c98a3d9c7d92c8/src/python/pants/engine/target.py#L1076 )
could be a facility for some generator helpers that rely on that only
i.e. if you don’t use the helpers, it’s a no-op.
Open in Slack
PreviousNext
Powered by