https://pantsbuild.org/ logo
#general
Title
# general
h

high-yak-85899

03/14/2022, 10:10 PM
I'm not there yet, but I'm curious about this statement I found in the docs
In-repo plugin code should not depend on other in-repo code outside of the
pants-plugins
folder. The
pants-plugins
folder helps isolate plugins from regular code, which is necessary due to how Pants's startup sequence works
This seems really prohibitive to developing new plugins. I have some custom code generation that uses other internal libraries to verify the generated code along the way among other things. I'm curious what's recommended for cases like this.
h

hundreds-father-404

03/14/2022, 10:14 PM
I think it's technically fine to do? Only you want to set
--pantsd-invalidation-globs
properly? pantsbuild/pants is basically one big plugin running on itself 😛
w

witty-crayon-22786

03/14/2022, 10:16 PM
https://github.com/pantsbuild/pants/issues/9902 describes the downside in a bit more detail
basically: it’s fine to put something larger than an isolated directory on the pythonpath of Pants, but
pantsd
will restart when anything on the pythonpath changes (unless, as Eric said, you exclude items).
h

high-yak-85899

03/14/2022, 10:42 PM
Yeah, I'm trying to figure out how this would work in the future. Basically, we have some tooling that tells us how to take data that comes down in a protobuf format and transform it into something that goes to a database (among a few other things). So we're basically going from yaml to a python module. Our internal libraries are used because some of the python module stuff has been abstracted out and, perhaps the biggest one, we can use our generated protobuf sources to verify if that yaml was set up properly. So what I'm gathering is I'd either have to completely strip that validation at generation time and likely end up with a system where other in-repo code depends on code from
pants-plugin
? Does that sound right?
h

hundreds-father-404

03/14/2022, 10:44 PM
It's probably fine to keep the current factoring you have. Iiuc, the worst that happens is Pantsd doesn't get invalidated, so when you are iterating on your non-plugin-code that is used by your plugin code, you might be confused why changes aren't happening. Which can be fixed by
rm -rf .pids
, or updating
pantsd_invalidation_globs
or updating
pantsd_invalidation_globs
There, the only downside is that restarting pantsd means you have to redo some work you already had done. It's not a correctness issue, only suboptimal for perf, but it's not the end of the world
w

witty-crayon-22786

03/14/2022, 10:45 PM
so when you are iterating on your non-plugin-code that is used by your plugin code, you might be confused why changes aren’t happening.
that can’t happen: if the code is loadable, it’s because it’s on the pythonpath. and the entire pythonpath is invalidated
👍 1
…unless (sorry)! you have excluded something on the pythonpath from
pantsd_invalidation_globs
. that can cause an edit to not restart the server, but that’s opt in.
👍 1
so, assuming that you put a large directory like
src/python
on the pythonpath, which contains a lot of stuff unrelated to your plugin,
pantsd
will restart more than you would like: things will be correct, but slower for your users. you can reduce that by excluding things from invalidation (that you know are on the pythonpath, but not relevant to your plugin) via
pantsd_invalidation_globs
… e.g.
!/src/python/this/is/not/relevant
… but that needs to be done carefully/accurately, or
pants
won’t restart for the plugin change.
👍 1
h

high-yak-85899

03/14/2022, 10:54 PM
It seems like how we've gotten around this type of problem when using Bazel was to use a
genrule
in a way that lets us throw the generated code right back into the repo.
I don't think I see anything similar in Pants, but, at this point, I'm not too sure which would be the more appropriate way to do things. Putting generated code in the repo seems to be a slight anti-pattern of Pants. Having a heavily segregated plugin directory that requires precise control over the build system to optimize seems to be an anti-pattern of a monorepo.
w

witty-crayon-22786

03/14/2022, 10:57 PM
but afaik,
genrule
doesn’t actually allow for mutating the repo, does it? … except maybe by violating the sandbox?
with
experimental_shell_command
:
The command is limited to operating on the specific set of input files provided as dependencies, and only produces output files for other targets to consume. It is not possible to mutate any file in the workspace.
h

high-yak-85899

03/14/2022, 11:01 PM
We play some tricks with
location
which I assume gives us an absolute path which does violate the sandbox to put things back in the repo structure.
Okay, well certainly something to think about. It seems like we kind of hacked our way around this class of problem in Bazel and avoided it altogether before using Pants because nothing about our Python usage was sandboxed. I'll revisit this likely when we're closer to making this happen. We've got a couple use cases of tools like this beyond just the one I mentioned above.