I m not there yet but I m curious about this statement I fou Pants #general

I'm not there yet, but I'm curious about this stat...

high-yak-85899

03/14/2022, 10:10 PM

I'm not there yet, but I'm curious about this statement I found in the docs

In-repo plugin code should not depend on other in-repo code outside of the
pants-plugins
folder. The
pants-plugins
folder helps isolate plugins from regular code, which is necessary due to how Pants's startup sequence works

This seems really prohibitive to developing new plugins. I have some custom code generation that uses other internal libraries to verify the generated code along the way among other things. I'm curious what's recommended for cases like this.

hundreds-father-404

03/14/2022, 10:14 PM

I think it's technically fine to do? Only you want to set

--pantsd-invalidation-globs

properly? pantsbuild/pants is basically one big plugin running on itself 😛

witty-crayon-22786

03/14/2022, 10:16 PM

https://github.com/pantsbuild/pants/issues/9902 describes the downside in a bit more detail

witty-crayon-22786

03/14/2022, 10:19 PM

basically: it’s fine to put something larger than an isolated directory on the pythonpath of Pants, but

pantsd

will restart when anything on the pythonpath changes (unless, as Eric said, you exclude items).

high-yak-85899

03/14/2022, 10:42 PM

Yeah, I'm trying to figure out how this would work in the future. Basically, we have some tooling that tells us how to take data that comes down in a protobuf format and transform it into something that goes to a database (among a few other things). So we're basically going from yaml to a python module. Our internal libraries are used because some of the python module stuff has been abstracted out and, perhaps the biggest one, we can use our generated protobuf sources to verify if that yaml was set up properly. So what I'm gathering is I'd either have to completely strip that validation at generation time and likely end up with a system where other in-repo code depends on code from

pants-plugin

? Does that sound right?

hundreds-father-404

03/14/2022, 10:44 PM

It's probably fine to keep the current factoring you have. Iiuc, the worst that happens is Pantsd doesn't get invalidated, so when you are iterating on your non-plugin-code that is used by your plugin code, you might be confused why changes aren't happening. Which can be fixed by

rm -rf .pids

, or updating

pantsd_invalidation_globs

or updating
pantsd_invalidation_globs

There, the only downside is that restarting pantsd means you have to redo some work you already had done. It's not a correctness issue, only suboptimal for perf, but it's not the end of the world

witty-crayon-22786

03/14/2022, 10:45 PM

so when you are iterating on your non-plugin-code that is used by your plugin code, you might be confused why changes aren’t happening.

that can’t happen: if the code is loadable, it’s because it’s on the pythonpath. and the entire pythonpath is invalidated

👍 1

witty-crayon-22786

03/14/2022, 10:46 PM

…unless (sorry)! you have excluded something on the pythonpath from

pantsd_invalidation_globs

. that can cause an edit to not restart the server, but that’s opt in.

👍 1

witty-crayon-22786

03/14/2022, 10:51 PM

so, assuming that you put a large directory like

src/python

on the pythonpath, which contains a lot of stuff unrelated to your plugin,

pantsd

will restart more than you would like: things will be correct, but slower for your users. you can reduce that by excluding things from invalidation (that you know are on the pythonpath, but not relevant to your plugin) via

pantsd_invalidation_globs

… e.g.

!/src/python/this/is/not/relevant

… but that needs to be done carefully/accurately, or

pants

won’t restart for the plugin change.

👍 1

high-yak-85899

03/14/2022, 10:54 PM

It seems like how we've gotten around this type of problem when using Bazel was to use a

genrule

in a way that lets us throw the generated code right back into the repo.

high-yak-85899

03/14/2022, 10:57 PM

I don't think I see anything similar in Pants, but, at this point, I'm not too sure which would be the more appropriate way to do things. Putting generated code in the repo seems to be a slight anti-pattern of Pants. Having a heavily segregated plugin directory that requires precise control over the build system to optimize seems to be an anti-pattern of a monorepo.

witty-crayon-22786

03/14/2022, 10:57 PM

the moral equivalent of

genrule

in Pants is https://www.pantsbuild.org/v2.11/docs/reference-experimental_shell_command, but it’s fairly new. more docs here: https://www.pantsbuild.org/v2.11/docs/run-shell-commands

witty-crayon-22786

03/14/2022, 10:58 PM

but afaik,

genrule

doesn’t actually allow for mutating the repo, does it? … except maybe by violating the sandbox?

witty-crayon-22786

03/14/2022, 10:59 PM

with

experimental_shell_command

The command is limited to operating on the specific set of input files provided as dependencies, and only produces output files for other targets to consume. It is not possible to mutate any file in the workspace.

high-yak-85899

03/14/2022, 11:01 PM

We play some tricks with

location

which I assume gives us an absolute path which does violate the sandbox to put things back in the repo structure.

high-yak-85899

03/14/2022, 11:11 PM

Okay, well certainly something to think about. It seems like we kind of hacked our way around this class of problem in Bazel and avoided it altogether before using Pants because nothing about our Python usage was sandboxed. I'll revisit this likely when we're closer to making this happen. We've got a couple use cases of tools like this beyond just the one I mentioned above.

Open in Slack

Previous Next