https://pantsbuild.org/ logo
Join Slack
Powered by
in my plugin, i'd like to be able to generate some...
# plugins
e
in my plugin, i'd like to be able to generate some sources and have them be pulled into the docker build context for a 
docker_image
target. is there an existing pattern i can follow to achieve this?
i've been looking at this, which is how the file sources get included into the context, and this generator code seems to imply what i want to do is possible, but i can't seem to figure out how to make it work.
c
do you have codegen setup and working for your sources?
e
i'm putting a gist together to show you what i have
👍 1
c
if so, this ought to be all you have to do, but for your own source field
e
maybe my codegen is incorrect then because i do have that
c
I'll have to double check my claim as well, it's been a while.. 😬
could be, that if you need to rely on codegen, that it needs to codegen into a 
FileSourceField
type..
instead of registering, as the comment seems to imply that it works for sources directly w/o using codegen then.. 🤔
e
c
yea, if that doesn't work, try changing https://gist.github.com/vito-laurenza-zocdoc/fc9e0b8434d8632e9120dbbd530066ea#file-targets-py-L37 to
output = FileSourceField
and then comment out https://gist.github.com/vito-laurenza-zocdoc/fc9e0b8434d8632e9120dbbd530066ea#file-targets-py-L24-L27 this ought to indicate how to generate a file source from your custom target, which will then be included in the docker build context.
e
okay i'll give that a shot too and report back
👍 1
c
oh, wait.. I see another issue, I think...
or.. I'm not sure, I wonder why you'd need those "internal type marker for codegen" classes...
I think those are cludges in the docker backend to work around some codegen pecularities that you shouldn't need...
I think this is a healthier example for drawing codegen inspiration from, from the docker backend https://github.com/pantsbuild/pants/blob/77446e9f6d247b327ea4721daf2255de330a4b64/src/python/pants/backend/docker/util_rules/dockerfile.py#L15-L19
(except that the input and output types are the same... that's another cludge) 😛
e
thanks for the additional example. one thing i'm puzzled by is that, in my use case, i don't have a "real" target field like 
DockerImageSourceField
that provides any value. i don't want my users to need to do anything other than use the 
my_docker_image
target in place of 
docker_image
.
that's why i added those "internal type marker" classes in the first place
c
mm.. right. let me try this out, I need to figure this out from scratch (been away from pants development a while..)
e
fwiw i tried your first suggestion of 
output = FileSourceField
still don't see my file in the sandbox
c
huh, yea, I suspect there's a few `UnionRule()`'s missing.. but I've so far failed to come up with a working example. I have to break here.. see if I'll come back to it tomorrow, it shouldn't be this difficult. 😕
e
no worries. i appreciate the help! i'll keep tinkering on my end.
🙏 1
reporting back, still no luck on this
g
Can you share a full example? Or is the gist up to date at least?
e
the gist is about as full as i can provide
what else did you have in mind?
my use case is: inject a file into the docker build context. that file should be dynamically generated, i.e. it doesn't exist in the source repo.
g
A runnable example is easier to look at and debug than gists 🙂 For example, that code doesn't do any rule registration so if that is wrong we can't tell.
e
ah sorry
i do register a UnionRule with the target gen request
didn't realize i didn't put it in the gist
g
A thing to keep in mind is that the consumer of the source also has to opt in to codegen. Very common source of issues
e
i.e. 
UnionRule(GenerateTargetsRequest, GenerateMyDockerImageTargetsRequest)
can you explain what you mean there? the consumer of the source in this case should be the core pants code
i thought that was the purpose of the register call
DockerContextFilesAcceptableInputsField.register(...)
g
That might be it, just speaking in general terms! I am not familiar with the built-in docker backend code. First of, do you also have a rule like 
UnionRule(GenerateSourcesRequest, GenerateKubernetesFromKustomizeRequest)
?
e
yes
oh
hm
maybe not!
let me double check
ah yes i do
g
e
right
so the docker context builder code in the core does that
g
I have to explicitly say that when gathering the sources it's OK to consume those from codegen'd sources and what types it can accept from codegen
If it does, perfect
e
and 
DockerContextFilesSourcesField
is one of the source types it requests
and there's that 
ABC.register
hook that should supposedly allow us to register other sources that would work with it
i realize i'm probably missing one tiny thing 🙁
but i can't for the life of me figure out what it is
g
Yeah, a lot of Pants rules machinery is... finicky, and most faults are not errors... it just doesn't call the code. Ruling out the stupid stuff, does the docker target depend on your target, and does 
pants peek
on the various targets involved look right?
e
yeah i verified that the docker_image target depends on my target with peek
and with debug logging
g
Ok. One thing of note that might be irrelevant, but your target generator should probably specify core fields
I've never seen a target generator that doesn't specify all fields there, and then the fields are either specified as copied or moved.
Oh; have you ran 
pants export-codegen
against your generated file target and seen what happens? If that is even picked up
Something is funky, I've retraced your code and ^ doesn't respect that it's a codegen'd target.
Copy code
✦ ❯ pants export-codegen //:foo#my_file && cat dist/codegen/my-file.txt
21:48:10.81 [INFO] Writing generated files to dist/codegen
foo
Your MyDockerImageMyFileTarget has the wrong source field
e
thanks for looking at this some more. i was moving things around when testing, but i did have the export-codegen working at some point. even with that working, i don't see it get generated in the context for the docker_image packaging.
i found a sort of work around. instead of relying on the magic docker context code gen source file request, i'm generating the file in a package rule for my target generator.
2 Views
Open in Slack
PreviousNext
Powered by