https://pantsbuild.org/ logo
Join Slack
Powered by
Hi! New to Pants. We have some autogenerated Pytho...
# general
n
Hi! New to Pants. We have some autogenerated Python code that we'd like to lint with ruff. What's the right way to do that? 1. 
python_sources()
globs only seem to read from the build root, not the the sandbox, and autogenerated files live in the sandbox. 2. 
GenerateSourcesRequest
outputs into the sandbox and integrates with the dependency graph, but its outputs don't seem to wire up to ruff and pyright automatically. Is a custom 
LintTargetsRequest
only way? Or is some built-in mechanism for wiring ruff up to the output of a codegen plugin?
a
Have you ran 
pants tailor ::
to setup your BUILD files? Running 
pants fix ::
or 
pants fmt ::
should be sufficient, could you share your pants.toml file? and the output of all the commands I've shared?
n
Sorry, for clarity: the autogenerated Python code isn't generated until build time, it doesn't exist in the build root. The difficulty I'm facing is how to check (but not fix) linting/formatting for files in the sandbox
a
Once autogenerated code is generated, you will need to run 
pants tailor ::
for pants to understand that it is a target and how it is related to other parts of your codebase. Then you can run 
pants lint <target-name>::
to check lint/formatting . You can also run 
pants list <target-name>::
to ensure you're targeting the right target
n
I'm confused. If the autogenerated code doesn't exist in the codebase - it only exists in the sandbox - what will 
pants tailor
be looking at? The generated code doesn't exist on my "normal" filesystem, only in 
/tmp/pants-sandbox/
h
I believe @acoustic-spring-13969ā€™s suggestion is based on the code being in the repo, but it sounds like you're trying to do something a bit unusual - run ruff on an intermediate build output that only exists in the LMDB store (and can get materialized to a sandbox)
n
Yeah, exactly
I'm trying to avoid committing autogenerated code to the repo; as far as I'm concerned it's just part of the build process
h
would need to think about that one, we didn't really design for that case as the assumption was that linting/style checking is for source code, not intermediate code
Out of curiosity, why run 
ruff
on it?
n
The use case we have is a Protobuf-based API that we're autogenerating code for. We're using buf.build to call custom Protobuf plugins that generate language-specific SDKs.
Obviously this is very polyglottal, so I was hoping pants would work here: ā€¢ pants calls 
buf build
, which checks the Protobuf files for correctness ā€¢ pants calls 
buf generate
, which calls our custom Protobuf plugins that generate code ā€¢ pants lints and formats both the generator and generated code
The generated code is what we ship to other teams at the company/external clients
I currently have a pants plugin (it shells out to 
buf generate
) that takes in a custom 
ProtoSourcesField
and produces a 
PythonSourceField
. Unfortunately, I couldn't figure out how to wire that up to ruff check rules
I think if ruff's 
rules()
method returned a union with something exensible (and hydrated with 
enable_codegen=True
), I could just subclass it? It looks like that's what Pytest does. But honestly I'm so new to pants I have no idea if that's true
h
Interesting use case! So I think this may call for a little custom plugin, rather than trying to work directly as a regular linter. I wonder if an 
ad_hoc
rule is enough?
n
By ad_hoc you mean this guy, or something else?
h
that guy indeed
I would need to think about this one a little bit
n
I have a little custom plugin to do the actual generation, but I was hoping to use the built-in ruff integration to check the linting and formating of the output
h
And the code being generated is python?
Or it's all sorts of things?
n
For now just Python, but we're going to add TypeScript (and probably Go) later
h
Got it
n
appreciate the help, i am sadly very unfamiliar with pants
h
I think your custom plugin might have to run 
ruff
itself, but I imagine it can still use the existing machinery for downloading and invoking it
possibly Claude might be able to help here. I've found it to be reasonably decent at monkey-see-monkey-do type stuff in the pants repo
w
Any reason this couldn't be just a script that runs after build? An adhoc script might work too, if it takes the generated output as a dep
https://gist.github.com/sureshjoshi/98fb09f2a340f7c1dad270c4887865a0 If you wanted it to live inside of Pants entirely, something like this - where each step takes the output of the previous might work
n
We have some non-autogenerated tests that test the autogenerated code, which means it can't run after build šŸ˜ž. e.g.:
Copy code
python_tests(
    name="tests",
    sources=["test/**/*.py"],
    dependencies=[
        "//protobuf:autogenerated-python",
    ],
)
w
I think the adhoc approach might have a chance, but what about running the tests, and then hitting it with a linter/formatter at the end of the pipe? Unless the concern is that the act of formatting might affect the output? Or is it a fail early thing because tests take too long?
n
That would force us to call the linter/formatter/pyrite manually though, right? It wouldn't be able to use pant's integrations, and it wouldn't run with 
pants lint
/`pants format` right?
w
I don't think you'd be able to use a vanilla 
lint
or 
format
right now, though, which is the problem, correct? We chain commands like 
pants fix fmt lint check test
(or whatever), so, a custom plugin could work that just calls the generation and lint commands internally (like, call the python functions, or a command line). With 
adhoc
you'd be calling something like 
pants run :special_target
and it would hopefully work. That means no custom plugin.
n
I have a slightly convoluted plugin which lets me run 
pants lint
/ 
pants format
, but it just calls ruff directly
I think at this point I'm mostly wondering about this? I'm happy to PR something if that's true, but I honestly am just too new to pants to know if it is
w
Maybe, it's a bit of a weird one for a few reasons. Generated code, which isn't committed to the repo, which means that it would default to be hidden by pants_ignore (probably?). I'd need to think about that for a min
We also have workspace-centric functionality, so maybe this would be a "run in workspace" usecase, so it would end up hitting the generated code too maybe (edit: As opposed to the sandbox)
n
not 100% sure I follow but sure šŸ˜…
Keep me posted on your thinking, happy to try and work on this some more
w
Can you share what your code-generation workflow build file looks like?
n
I think so, let me start with just the build files and if that's not sufficient I'll see about getting the sources too
one second
w
Is it public?
n
not yet šŸ˜ž
w
ah
THat at least explains why I've never used it yet šŸ˜†
In looking at your example, and reading through the Pants codebase, it feels like we're missing a target (or maybe a synthetic target, or some other mechanism) to access intermediate state, or to chain steps... I don't quite know what it would look like, and I'm reviewing the docs and code to see if we already do something similar elsewhere, but yeah, it feels like we're missing the ability to operate more actions on generated code
ā¤ļø 1
šŸ‘€ 1
We use the internal files in imports without exporting, so I would think that there should be some way to run tooling over them using transitive checks or something.
šŸ‘€ 1
Open in Slack
PreviousNext
Powered by