https://pantsbuild.org/ logo
#general
Title
# general
n

narrow-judge-15725

08/12/2022, 2:41 PM
sorry, me again. i have a general conceptual question about hermeticity and pants. i just started so bear with me if this is stupid but i’ve seen that pants relies, for example, in having the python interpreter in the machine. this means that technically we could still have the “works on my machine”. i played with bazel and they have the concept of toolchains so one doesn’t need to install anything and it makes sure that the environment (including the tools) is the same always. did i miss anything or is there an explaination for this? thanks!
b

bitter-ability-32190

08/12/2022, 2:43 PM
bear with me if this is stupid
No such thing 🙂 We believe if it ain't obvious, or you can't find it easily that's on us 😛
You got it right, however as you'd imagine the actual answer is more nuanced 🙂 Pants uses system Python for two purposes: 1. Running Pants itself (although there's an ongoing project to use an embedded Python so we don't need it installed) 2. Running Python-related subsystems/goals/etc... For 2, we do use system Python, however we invoke it through PEX, which goes through great lengths to isolate itself from system Python. Of course, it isn't perfect, but I've seen only a handful of instances of issues at my org (and we have some, er, um, creative, engineers that like to mess with their system) In general, Pants (really us the community) prefers if Pants can provide the tools, so that we can protect from supply-chain attacks and "work on my machine" like you say. So of course, a long-standing desire would be to provide Python (akin to Bazel's, although likely very different in execution).
e

enough-analyst-54434

08/12/2022, 2:49 PM
I just did a quick read of toolchains and fwict it does not guaranty same tools always, just the same paths to them etc. Does it verify their content hashes or build the tools from source?
b

bitter-ability-32190

08/12/2022, 2:49 PM
It turns out building/providing a complete Python environment isn't easily done. But we might be able to leverage projects like
pyenv
🤷‍♂️
e

enough-analyst-54434

08/12/2022, 2:50 PM
But back to bazel toolchains - Josh you know Bazel - do toolchains actually guaranty identical toolchains? It looks like they just guaranty paths to binaries are fixed but not that they are actually the same as some fixed benchmark.
b

bitter-ability-32190

08/12/2022, 2:51 PM
No clue. My Bazel depth never needed to expand to understanding Toolchains. Only enough to "keep the lights on" w.r.t. the Python toolchain
e

enough-analyst-54434

08/12/2022, 2:51 PM
I'm just reading here and seeing nothing that addresses sameness: https://bazel.build/docs/toolchains
Ok. Gotcha.
h

happy-kitchen-89482

08/12/2022, 2:53 PM
Pants does install most of the tools it uses, where that is practical, but installing python interpreters can get tricky. But Pants does verify the specific identity of every interpreter it uses. It doesn’t just assume that something called
python
or even
python3.9
is what it says, but verifies that it is a specific point version of a specific implementation (e.g., cpython or pypy) before using it.
n

narrow-judge-15725

08/12/2022, 2:53 PM
Not an expert in bazel but what happens is that the toolchain is sort of a name which abstracts the underlying hardware. Under the hood it will download the tool (ie macOs binary) and be save with the sha. That tool gets cached like any other dependency.
Docker is a different beast but most tools do come in binaries I guess
h

happy-kitchen-89482

08/12/2022, 2:54 PM
Yeah Docker is a great example of an external dependency that Pants (and, I’m guessing, Bazel) expects to have on your system
b

bitter-ability-32190

08/12/2022, 2:55 PM
Bazel doesn't use Docker to build. It has its own in-house logic to produce images that are identical to what Docker would produce 😵
h

happy-kitchen-89482

08/12/2022, 2:55 PM
Pants does download and verify against a lockfile almost every tool it uses, for hermeticity, but not the python interpreter itself. But it does have very robust interpreter detection and selection mechanism that, for example, knows how to treat pyenv shims
b

bitter-ability-32190

08/12/2022, 2:55 PM
https://github.com/bazelbuild/rules_docker
These rules do not require / use Docker for pulling, building, or pushing images.
...
Also, unlike traditional container builds (e.g. Dockerfile), the Docker images produced by
container_image
are deterministic / reproducible.
h

happy-kitchen-89482

08/12/2022, 2:56 PM
Does bazel download an entire c/c++ toolchain for you? or does it assume the presence of xcode tools on macos for example?
n

narrow-judge-15725

08/12/2022, 2:56 PM
Yeah magic
e

enough-analyst-54434

08/12/2022, 2:59 PM
In practice, FWIW, I think the only issues we've seen with non-hermeticity of tools involve finding a viable Python Interpreter. Most of those issues have to do with Mac system interpreters. I have not heard of issues like this for go, shell or jvm.
👍 1
w

wide-midnight-78598

08/12/2022, 3:02 PM
@happy-kitchen-89482 https://bazel.build/tutorials/cc-toolchain-config Seems to, at least, allow using system gcc/clang - they may be able to download custom ones too (think I saw this in a StackOverflow some time ago)
h

happy-kitchen-89482

08/12/2022, 3:05 PM
Mac system interpreters are just broken
2
e

enough-analyst-54434

08/12/2022, 5:23 PM
So it sounds like Bazel toolchains are just a uniform way to specify download or use local. Pants allows you to do said same, but not as uniformly. Each backend does it a bit different.
2 Views