https://pantsbuild.org/ logo
Powered by
w

witty-family-13337

02/10/2022, 8:25 AM
Hi, quick question regarding the inner workings of 
GenerateSourcesRequest
-based rules. Assuming that I’ve got a target like 
mytarget(sources=['source1.ext', 'source2.ext'])
and then I write a custom 
GenerateMySourcesRequest
like:
Copy code
class GenerateMySourcesRequest(GenerateSourcesRequest):
  input = MySourcesField
  output = MyGeneratedSourcesField
How or where does Pants attach that 
MyGeneratedSourcesField
referenced in the output? Does Pants create a target on the fly with that field or does it simply attach the field to the original 
mytarget
?
c

curved-television-6568

02/10/2022, 9:28 AM
Hi Alonso, This is what in Pantsbuild is referred to as a 
codegen
rule. What it does, it takes your input sources (the sources from the 
MySourcesField
) which are associated with your 
mytarget
and using your codegen rule, translates those into 
MyGeneratedSourcesField
sources. The generated sources are not directly addressable besides being requested via codegen, but are instead consumed along with any other sources you may have. In other words, they’re simply generated ad-hoc when requested. Here’s a trail for such a codegen request: https://github.com/pantsbuild/pants/blob/main/src/python/pants/backend/python/util_rules/python_sources.py#L88-L95 https://github.com/pantsbuild/pants/blob/main/src/python/pants/core/util_rules/source_files.py#L54-L58 https://github.com/pantsbuild/pants/blob/main/src/python/pants/engine/internals/graph.py#L836-L840 And in order for your codegen rule to work, you need to register your union member as 
UnionRule(GenerateSourcesRequest, GenerateMySourcesRequest)
. Hope this makes it more clear, but plz do send any follow up questions 🙂
w

witty-family-13337

02/10/2022, 9:37 AM
many thanks for the answer Andreas, the first links seem to be helpful
my scenario is that I have that custom 
mytarget
and a codegen rule that generates some additional files from the 
sources
field of 
mytarget
, and wanted to be able to collect the files referenced in the 
sources
field plus the generated ones when using 
SourceFilesRequest
f

fast-nail-55400

02/10/2022, 12:44 PM
you’ll need to set 
enable_codegen=True
for 
SourceFilesRequest
👍 1
(and probably 
for_sources_types
)
then the codegen’ed files will be produced
if you need the originals and the generated files, then your rule should probably issue two 
SourceFilesRequest
one with 
enable_codegen=False
and the other with 
enable_codegen=True
then merge the resulting 
Digest
’s using 
MergeDigests
c

curved-television-6568

02/10/2022, 12:59 PM
Or, if this is always the case, your codegen rule may return both sets at once.. By looking at this example, which selectively returrns the original source OR some generated source, it shows that you could merge the two right there. https://github.com/pantsbuild/pants/blob/0ce449f9181c690dbf60b2bc51b0e97f285eb9ae/src/python/pants/backend/docker/util_rules/dockerfile.py#L46-L55
w

witty-family-13337

02/10/2022, 1:42 PM
thanks a lot for the details, for some reason 
./pants export-codegen
is indeed generating the files but 
SourceFilesRequest
is not returning them even when I add 
enable_codegen=True
and the field type that is used in the 
GenerateSourcesRequest.output
field. The original sources are returned properly though
in my case, the 
GenerateSourcesRequest.input
and 
GenerateSourcesRequest.output
fields are of different type but in the same target, though that could be an issue but seeing Andreas example now I just discarded that could be reason it doesn’t seem to work
I’ll re-check the code thoroughly, most likely I have something missing in some point
f

fast-nail-55400

02/10/2022, 1:47 PM
can you share some of the plugin code?
DM is fine if there is stuff in there that shouldn’t be public.
h

hundreds-father-404

02/10/2022, 3:06 PM
Yeah you'll want to make sure 
GenerateSourcesRequest.output
is what you are generating into, and that 
for_sources_types
includes that Also double checking you have the 
UnionRule
registered for 
GenerateSourcesRequest
, although it sounds like you do with 
export-codegen
working
w

witty-family-13337

02/10/2022, 3:59 PM
could make a tarball with the plugin code, don’t mind sharing it, in fact we are thinking of open source it
❤️ 1
just a bit of a busy day in here so may not be able to pass it over until tomorrow
sorry for the delay on coming back to this, just wanted to let you know that thanks to your pointers I managed to debug this and now it works 🙂
❤️ 1
👍 1
c

curved-television-6568

02/14/2022, 11:33 AM
Great! Thanks for letting us know it works now, Alonso 🙂
w

witty-family-13337

02/14/2022, 1:20 PM
btw, if you guys are interested on a Helm plugin for Pants, let me know 😅
c

curved-television-6568

02/14/2022, 1:30 PM
+1 🙂 As the topic of a 
deploy
goal has floated around the last couple of days, I think 
helm
support would be a sweet addition along side ansible.. 🙂 (as we happen to use both of those technologies at work)
w

witty-family-13337

02/14/2022, 2:13 PM
yeah, we use helm a lot (and terraform), since we saw that you guys had already a draft plugin for terraform, we wrote one for Helm
it is quite experimental, it adds three new targets: 
helm_chart
(first party custom charts),
helm_artifact
(third party helm charts) and 
helm_deployment
(helm configuration for a given deployment) and it’s capable of inferring dependencies among them and with other Docker images built in the same repo, although it is a bit rough around the edges
happy to provide with a PR with this code as we think that other people using it and providing their ideas would be much more beneficial than we having it in our private repo
but obviously, this may be something the main team is not interested on or does not have the bandwidth for, so as you guys suggest
h

hundreds-father-404

02/14/2022, 3:33 PM
cc @polite-garden-50641, who uses Helm a lot. Even if it's highly experimental and not polished, I'd definitely be curious to see what you have as PR. It might not make sense for the project to land (yet), but we could at a minimum give you a code review to help your plugin be more idiomatic 🙂 (we love helping people, and it is also useful for us to see how people are using the API)
w

witty-family-13337

02/14/2022, 4:04 PM
I’ll prepare a draft PR so you can give any feedback (which I would really appreciate), hopefully I can have it tomorrow before you guys start the day in the US. No rush on giving comments, we’ll keep a copy of the code in our monorepo as at the moment it serves us well 🙂
📣 1
🙌 1
p

polite-garden-50641

02/14/2022, 6:52 PM
from my personal POV, helm is a deployment orchestration tool, I don't think deploying code to production is something pants should be handling (just my opinion) as by definition deployments to various environment have various side effects beyond the scope of the machine running the tool so there is no advantage for using pants and levering its features like caching (and maybe remote execution) and fine grained dependencies, etc...
👍 1
c

curved-television-6568

02/14/2022, 7:11 PM
Caching and remote execution aside, I still think Pants as a platform/framework have strong features for executing tasks that may have intricate dependencies and complex logic/rules. If that (deployment) is a fit as a core Pants feature, maybe not, and in that case I’d still consider using the engine for implementing a deployment tool. Perhaps as a sibling to Pants 😉
p

polite-garden-50641

02/14/2022, 7:17 PM
the deployment space/problem is a very different problem from the build tools base. it is best not to mix them. my 2c is that pants should focus in the latter and shouldn't try to tackle deployments (or even publishing for that matter, since it is a type of deployment), there are other tools in that space. Pants should focus on becoming a better build tool and include support from more languages rather than try to tackle other domain/spaces.
👍 2
again, just my perspective.
c

curved-television-6568

02/14/2022, 7:24 PM
Agree with not adding too much to the mix. Publishing however I see as a natural step, as you almost always want to publish it somewhere. Deployment is the next step after that, but at least after publishing, the built artifact has a natural resting place for other tools to take it further.
w

witty-family-13337

02/15/2022, 8:06 AM
the deployment side is defo a grey area and I assume we could several different opinions there. The motivation for us for the plugin comes from the fact that in our scenario, a Helm Chart is also a deliverable, not just a deployment description. What I mean is that we have a product that we offer in two different deployment models to our customers: a managed service (in which we deploy it in a cloud environment we control) or on-premise, in which we deliver the binaries (docker images) and Helm Charts to our customers and they are the ones that need to tailor the deployment to their needs.
because of that, the engineering team has to code, test, package and publish those Helm Charts that then later are used by either our managed services team (to run their deployments) or by our customers to do the same thing into their environments. So in this scenario Helm Charts are code as well, code that follows the same lifecycle as the rest of the code in the same repo. Besides, Charts are versioned artifacts that can have (and they have in our case) dependencies among themselves or with 3rd parties, also they may have dependencies with docker images built in the same repo, and on top of that, helm has some sort of ecosystem of tooling (i.e. we use the 
unittest
plugin to run tests, instead of 
helm test
, we generate 
README.md
doc files using 
helm-docs
, etc. etc.) so the version hell problem is quite present in Helm projects (either in relation to the build artifacts or to the versions of the tools being used), specially those that involve medium to large systems.
💯 1
the code we wrote is mostly targeted at the Helm is code kind of scenario, although it’s true that we also have a goal for deploying them (but we recognize that is more of a fringe kind of thing). And to be honest, Helm has help us a lot removing problems like developers having different tooling versions (or environment) in their machines, or in orchestrating the different tools and having a consistent build order thanks to dependency inference.
So I would suggest to look at the Helm situation not just from a deployment perspective, as that is just one of Helm aspects.
Anyway, we were already thinking of opening the code as we see a benefit on others also handling the Helm is code scenario using a built tool, whether this belongs in Pants main distribution or as part of a pants-contrib one (or even as it’s own individual repo) is obviously something debatable and I can see valid arguments for either one. From our perspective, we would like to have some feedback on the implementation so where the code finally lands is less important.
❤️ 1
c

curved-television-6568

02/15/2022, 8:47 AM
Indeed, Helm charts, Ansible collections (with roles etc) are versionable things that you need to manage in terms of test, dependencies and distribution, which I feel warrants a home in the Pants ecosystem in some form.
h

hundreds-father-404

02/15/2022, 9:00 AM
We could also use the same philosophy we have with Shell: focus on where Pants brings unique value, don't focus on what is already well served. With Shell, this means that we integrate with Shellcheck, Shfmt, and Shunit2 test runner. But at least for now, we don't integrate with 
./pants run
because Pants has no value running the bash script for you With Helm, that would mean integrating with things like 
helm test
and 
helm-docs
, but leaving the actual side-effecty deploy for you to do directly
➕ 1
w

witty-family-13337

02/15/2022, 10:51 AM
glad to hear that there is some room for this in the Pants ecosystem, I’ll prepare the PR and submit it as a Draft one, so don’t feel like you need to rush its review or even to merge it at any point
🙌 1
as promised, here is the draft PR for the plugin: https://github.com/pantsbuild/pants/pull/14487
🙌 1
Powered by