Hi all! I wondered if toolchain management has bee...
# general
n
Hi all! I wondered if toolchain management has been discussed already somewhere. I'm coming at Pants from an existing monorepo which is built using Please, but interested in evaluating Pants as a potential replacement based on its healthier-seeming community and my hopes of a more mature / better documented tool (not that I don't like Please, just that it can be a bit of a trawl to figure out what's going on when it breaks...) One thing we have in our existing Please repo is a good set of rules for managing both Python and Go toolchains, which results in a high degree of hermeticity since everything right down to the language runtime is reproducibly managed by the build tool. I'm aware that Pants uses lockfiles for tools in Python, but not aware of what the (planned) hermeticity mechanisms are for Go, and both are important to us since we're a mixed-language repo. Really my questions here I think are -- what are people's experience with Pants' support for hermetic builds in the absence of toolchain management; are the lockfiles good enough? What does Pants plan for Go support? Is there a philosophical decision not to manage toolchains using the build tool, and if so what are the arguments for / against? Side-note: Please suffers similar Googlability problems to Pants 🙂
👖 1
🙏 2
👋 4
😂 1
b
@fast-nail-55400 has contributed most of the Go support, so is the best qualified person to tell you about that. :⁠-⁠)
In Resource Hub, there is also a lightning talk by @ancient-vegetable-10556 about Pants' hermeticity model. Hermeticity is provided by core, by the way, not individual backends.
(As an aside, I did a double take on your opening sentence because Pants' corporate sponsor is Toolchain, where I'm in management. Lol.)
😂 1
n
Thanks for the quick responses! And yes I'd found Toolchain's site by googling for this same problem!
c
Hi! and welcome @nice-park-16693 👋 you’ll also be happy to learn that the upcoming 2.16.0a0 release will include support for providing a managed Python also for user code https://github.com/pantsbuild/pants/pull/18352 ! 😄 provided by @bitter-ability-32190 🙌
🙌 1
n
Super cool, thanks for the context @curved-television-6568!
b
If you want to test it out, let me know! Id love for it to see some air time before the "big release"
So I can also speak to the experience of non-hermetic Python builds. It usually is OK. I manage an org of maybe 100 engineers/data scientists and only get hit by non-hermetic issues maybe once a month? And that's considering these folks don't know the nuances of the Python ecosystem and love running
pip install
(I blame Python, not people). That being said, it's really frustrating to use non-hermetic Python. That's why I'm divorcing Pants from system Python 🙂 The linked PR makes it so you can opt-into having Pants use
pyenv
to build and install Python in a read-only cache directory. So it should become way more hermetic. It'll still rely on system libraries, but nothing is truly hermetic and this is a great improvement.
f
@nice-park-16693: What did you want to know specifically about the Go support?
The Go backend currently only supports searching for Go on the system because of performance issues in earlier versions of the Go support due to having to unpack the Go SDK too much. Having Pants manage a Go SDK installation is something we would like to do again, but I'm not aware of engineering resources available at the moment to bring that support back.
a
@fast-nail-55400 was that before or after we fixed the same problem in Java-land?
f
well before the "immutable input" work on
Process
and the newer work superseding immutable inputs to symlink inputs instead
as in, the SDK download support existed only before the Go backend was ever released. it released with just searching the local system.
a
With that in mind: there’s a fairly clear path forward on how to manage the go SDK in a way that is Pants-efficient, which we can probably help someone towards implementing. It’s just not a priority for us to do the heavy lifting ourselves
b
Yeah we just got this for Python. Opt-in "Pants-provides-it" with the default of "Pants goes looking"
f
With that in mind: there’s a fairly clear path forward on how to manage the go SDK in a way that is Pants-efficient, which we can probably help someone towards implementing. It’s just not a priority for us to do the heavy lifting ourselves
I have done some initial thinking here. I wanted the Go backend to just have an equivalent to what
pyenv
,
coursier
, and
asdf
do and download and install the Go SDK into a Pants-managed directory.
rather than manage it as a tool that is symlinked into the execution sandbox
a
Do you mean doing what
coursier
does inside pants, when it’s used elsewhere?
f
I mean doing something similar to what
coursier
does in downloading JDKs but managed from within the Pants rules and not via an external utility as
coursier
is
a
ah sure
n
What did you want to know specifically about the Go support?
What interests me here really is what the future might hold for Go toolchain management by Pants. It seems like this might be the answer, which seems like it gives parity with what Please currently does:
I wanted the Go backend to just have an equivalent to what
pyenv
,
coursier
, and
asdf
do and download and install the Go SDK into a Pants-managed directory.
Correct me if I'm misinterpreting what you've said, but otherwise this seems promising and thanks for the responses!