https://pantsbuild.org/ logo
Join Slack
Powered by
Is there any 'best practice' for building/structur...
# plugins
a
Is there any 'best practice' for building/structuring codegen code? I'm doing a heavy amount of transforming of python code based on the results of the 
inspect.
python module (mostly inspect, to get attributes, then inspect.getsource() to stick into a jinja template), and want to tie it into Pants. should I build it as a standalone pex and then have pants call that? or should I just run it straight from within my pants plugin.
h
The most convenient approach is probably to use a codegen plugin. You can even wire into Pants's dependency inference so that Pants understands an import of 
my.codegen.foo
translates to 
src/codegen/custom_template.json
for example (whatever scheme you want) With that set up, your users simply need to depend on the file. Then Pants will pick up the dependency and automate doing codegen for you: making sure you always use the most up-to-date
a
ok, so there's no restrictions on importing non-plugin code in plugin code or anything?
h
Sort of: you have to set 
[GLOBAL].pythonpath
which causes Pantsd to invalidate anytime a file is changed. Bad for performance But I don't think I'm following that question: you can write a plugin that creates a new target type like 
transformed_python()
, and then add codegen for it. The rest of your repo can now use that just like any other target
a
yeah, my concern is how to build the transformed-python code. I certainly don't want to kill performance by invalidating the cache whenever a file changes. But I have a (stupid?) requirement that I have to share transformed code between 'normal' targets and transformed code (that's effectively just in-lining code) So if I can't directly import non-plugin code from within the plugin, I guess I'm going to come at it from another direction.
h
I don't think I'm following still. Your plugin only teaches Pants how to generate code + some new target type. I don't think your plugin code needs to directly depend on the transformed code, right? It's only defining a generic mechanism for the transformation Maybe an alternative explanation: all of 
pantsbuild/pants
has no idea that your repository 
my_repo/
exists when we write the code. We're writing generic targets/rules to operate on your code Does that make sense?
a
yes.
The generic targets/rules thing makes 100% sense. But, I have a bit of a mess I think. I've got a group of classes that are packaged up 'traditionally' into a docker container and shipped off. (pants covers this fine) But then we're also wanting to run our code-generator over these classes and spit out a singular bastardised python file that runs in a hyper-restricted environment (basically we're using annotations to strip out and restructure the bits that work in the environment) But we're using 
inspect
which I think this means that our codegen needs to know about the entire project.
h
But we're using inspect which I think this means that our codegen needs to know about the entire project.
I'm curious how the code is working? What's the 
inspect
doing? Note, for example, that Python dep inference from pantsbuild/pants knows about your whole repo to create it's module_mapping. But, we only need file names. We don't have to parse individual files. Which means it should not be invalidated infrequently -- To clarify, manual generation via a Pex is one valid approach. You run the risk of it becoming stale, like forgetting to rerun generation. But depending on how niche this is + your engineering resources, manual generation can definitely be a valid tradeoff
a
I'm just spitballing the code generation at the moment. But the end-goal is to create a singular, import-free, python file that runs in an environment with some global scope. But then also have some of this code reusable in our docker-hosted services (which is at odds with the global scope) The best PoC I've done is inherit from an abstract class that we 
inspect
and then enumerate through the functions, and then use 
inspect.getsource()
to flatten the class (no 
super()
supported. And then spit out some functions that call the class w/ the global scope passed in.
Then the tests take the 'compiled' code, upload it to the suppliers simulator and run the tests against it.
I do feel I'm missing a trick somewhere
5 Views
Open in Slack
PreviousNext
Powered by