Good morning/evening, I'm starting to make some attempts at the

incremental adoption

method to get pants into one of our internal repositories. @hundreds-father-404 was kind enough to get me started with an example plugin. I'm moving my questions here though to not bog them down too much. The example plugin exists in its own repo, and I have a seperate target repo I'd like to start integrating pants into, I'm thinking I start a new branch in the target repo and add the pants.toml, but I've got a couple questions 1. When I set up my pants.toml and spell out the new plugin, is pants "system aware" of my plugin? Do I need to add the plugin to a "pants environment" or something to that effect? 2. When I run pants the first time in a repo it initializes some things in the project right? Is there anything that gets generated I need to commit. I'm thinking I should just try this but I'm so new to pants I don't want to break anything about the "pants install". I assume that's not a problem really but better safe to ask first 🙂

If the plugin is in another repo then presumably you're publishing it somewhere to be consumed by the target repo?

If the plugin is in another repo then presumably you're publishing it somewhere to be consumed by the target repo?

Correct, the separate repo was for some basic testing to make sure things were suitable. And we'd like to reuse the function on multiple target repos. So my question is as you stated: How does the target repo consume the plugins/become aware of them?

It would probably be easiest to not mix problems. Can you start the incremental migration without custom plugins? I assume at least 1 goal will work without them if even just a metadata goal like

pants list ::

.

To bring the plugin code to the attention of Pants you just need to add it to the

[GLOBAL].plugins

option in pants.toml:

[GLOBAL]
plugins = [
  "my.plugin==1.2.3",
]

Yup, makes sense to do it incrementally

Pants simply doesn't like absolute symlinks. You'll need to ignore the path: https://www.pantsbuild.org/docs/troubleshooting#pants-cannot-find-a-file-in-your-project

It does appear that my plugin works when using an absolute path for the pythonpath to point to the plugin. I just put soem big #FIXMEs around it for now 🙃 we do have an internal PyPI we can use though if that's a supported use case as Benjy mentioned.

Internal PyPI is the primary use case, along with in-repo plugins. Sibling repo plugins obviously works but is decidedly not the norm.

What's the reason for not supporting symlinks? I think that makes sense from a build-engineer perspective "The tool should know where sources are". But people use those a lot. And my target repo has a TON that I need to fix anyway.

Internal PyPI is the primary use case, along with in-repo plugins. Sibling repo plugins obviously works but is decidedly not the norm.

OH, WAIT DUH! I could just use a git submodule right?

You could.

Symlinks are fine, symlinks to absolute paths are not

Pants has a file watcher and limits its scope to the build root tree is why.

Ah ok that makes sense. One of few things in the plug in so far is validation for certain absolute paths. Some of our source files are required to be stored an an NFS at an absolute path so our plugin has a env-check that looks for source files listed by the pants.toml for backwards compatibility. So we've already got a work around actually.

No, Pants uses git read only.

Our shop is slowly moving into the 21st century. I started using our internal PyPI just last month for example. We've got a HUGE deficiency in coordinating the builds of our designs because the tools are ancient, as I mentioned. So that's why Pants appears to be a great solution still. From what I've seen so far I think Pants is still a great option that should help us out.

I assume folks who are in your situation generally don't try to shoehorn that management into pants. They use some other tool (say:

ensure-sane-setup

).

The submodule thing is mostly because with making so many changes to the plugin so frequently it makes sense to use that rather than push to artifactory, then pull a new version right now.

Why not host the plugin in-repo?

I assume folks who are in your situation generally don't try to shoehorn that management into pants. They use some other tool (say:

ensure-sane-setup

).

The first goal is

check-env

😅

I question the wisdom of that being a Pants goal is what I'm saying insofar as it deals with things outside the repo.

Why not host the plugin in-repo?

There's a division of permissions in the organization

Ok - good luck to you. I've worked for crazy town before in the financial industry.

Hey sorry I'm just catching up. Thank you John and Benjy for weighing in.

I'm moving my questions here though to not bog them down too much.

No worries about that, feel free to ask me questions directly 🙂

Why not host the plugin in-repo?

One reason I helped write the plugin but don't have auth for Austin's hardware repository, and it would be a big production to get it. The idea was to help write the generic code in a repo, then copy and paste that into the dedicated single repository. If multiple repos ended up needing it, then consider publishing

Thanks Eric! Yes I was just about to explain the whole "state of the eda industry" problem. I'm a happy member of crazy town that won't move out. 🙂 Just trying to fix it

I question the wisdom of that being a Pants goal is what I'm saying insofar as it deals with things outside the repo.

The idea is that the underlying tools like compilers will access the absolute paths. Have a goal to make sure it's set up. And that goal's rule will run automatically when doing things like

check

and

lint

Indeed, Pants's file watcher and LMDB caching would not be able to cover changes to those absolute file paths.

But this sounds like a ~1-time thing. How often are devs at big-corp installing new things? I'd think much less often then cowboy-companies.

Iirc, every few days you have to re-set up your license. So Austin was saying a common source of problems is that peoples' environments have become stale

Jesus.

Yea it's bad

Ok - crazy-town. Good luck to you both!

There's no scripted environment set up right now

Indeed 😅

Just trying to fix it

Josh Cannon's Pycon talk helped inspire Austin to want to make life better. It won't ever get as nice as we're spoiled with, but a 20% improvement is substantial given how low the bar is etc

The original stuff is a bunch of TCL scripts from 2006 that's been carried on for years at the point. It was great for the time... 😅

So it's weirdly big-corp cowboy. I'm slightly shocked no scripted machine / env setup.

The field is specifically EDA so making chips/asics. Google is the biggest player right now trying to get the openroad project going. But that's not compatible with a lot of the close source hush-hush stuff, so EVERYTHING is either tcl or bash with hieroglyphic SED commands. Very fun times 🙂 there's been good progress with OSS tools for FPGAs but that's just the languages, I haven't seen great build coordinators yet

So it sounds like you can use the Pants engine as the basis of your own custom build, basically

So it sounds like you can use the Pants engine as the basis of your own custom build, basically

Precisely. This is the plan. Rather than trying to design a custom build system or using Bazel Also, Austin and I think dep inference will be super valuable. The dep rules are hard for humans to understand, it sounds like, but fine for machines

So it sounds like you can use the Pants engine as the basis of your own custom build, basically

💯💯💯 Since Pants in it's purest form is a build system / coordinator, it's perfect for handling these esoteric use cases.

Also, Austin and I think dep inference will be super valuable. The dep rules are hard for humans to understand, it sounds like, but fine for machines

Oh absolutely, this is the part that I would even like to open source. I spent a lot of time thinking about how to handle this and seeing what's out there, unfortunately there's not a lot, this is one of the better toolkits I've found over the years. It makes calls to dependencies that would be good tools for handling dep inference I think https://terostechnology.github.io/terosHDLdoc/