sorry, me again. i have a general conceptual quest...
# generaln
narrow-judge-15725
08/12/2022, 2:41 PMsorry, me again. i have a general conceptual question about hermeticity and pants. i just started so bear with me if this is stupid but i’ve seen that pants relies, for example, in having the python interpreter in the machine. this means that technically we could still have the “works on my machine”. i played with bazel and they have the concept of toolchains so one doesn’t need to install anything and it makes sure that the environment (including the tools) is the same always. did i miss anything or is there an explaination for this? thanks!
b
bitter-ability-32190
08/12/2022, 2:43 PMbear with me if this is stupidNo such thing 🙂 We believe if it ain't obvious, or you can't find it easily that's on us 😛
bitter-ability-32190
08/12/2022, 2:48 PMYou got it right, however as you'd imagine the actual answer is more nuanced 🙂
Pants uses system Python for two purposes:
1. Running Pants itself (although there's an ongoing project to use an embedded Python so we don't need it installed)
2. Running Python-related subsystems/goals/etc...
For 2, we do use system Python, however we invoke it through PEX, which goes through great lengths to isolate itself from system Python. Of course, it isn't perfect, but I've seen only a handful of instances of issues at my org (and we have some, er, um, creative, engineers that like to mess with their system)
In general, Pants (really us the community) prefers if Pants can provide the tools, so that we can protect from supply-chain attacks and "work on my machine" like you say. So of course, a long-standing desire would be to provide Python (akin to Bazel's, although likely very different in execution).
e
enough-analyst-54434
08/12/2022, 2:49 PMI just did a quick read of toolchains and fwict it does not guaranty same tools always, just the same paths to them etc. Does it verify their content hashes or build the tools from source?
b
bitter-ability-32190
08/12/2022, 2:49 PMIt turns out building/providing a complete Python environment isn't easily done. But we might be able to leverage projects like
pyenve
enough-analyst-54434
08/12/2022, 2:50 PMBut back to bazel toolchains - Josh you know Bazel - do toolchains actually guaranty identical toolchains? It looks like they just guaranty paths to binaries are fixed but not that they are actually the same as some fixed benchmark.
b
bitter-ability-32190
08/12/2022, 2:51 PMNo clue. My Bazel depth never needed to expand to understanding Toolchains. Only enough to "keep the lights on" w.r.t. the Python toolchain
e
enough-analyst-54434
08/12/2022, 2:51 PMI'm just reading here and seeing nothing that addresses sameness: https://bazel.build/docs/toolchains
enough-analyst-54434
08/12/2022, 2:51 PMOk. Gotcha.
h
happy-kitchen-89482
08/12/2022, 2:53 PMPants does install most of the tools it uses, where that is practical, but installing python interpreters can get tricky. But Pants does verify the specific identity of every interpreter it uses. It doesn’t just assume that something called
pythonpython3.9n
narrow-judge-15725
08/12/2022, 2:53 PMNot an expert in bazel but what happens is that the toolchain is sort of a name which abstracts the underlying hardware. Under the hood it will download the tool (ie macOs binary) and be save with the sha. That tool gets cached like any other dependency.
narrow-judge-15725
08/12/2022, 2:53 PMDocker is a different beast but most tools do come in binaries I guess
h
happy-kitchen-89482
08/12/2022, 2:54 PMYeah Docker is a great example of an external dependency that Pants (and, I’m guessing, Bazel) expects to have on your system
b
bitter-ability-32190
08/12/2022, 2:55 PMBazel doesn't use Docker to build. It has its own in-house logic to produce images that are identical to what Docker would produce 😵
h
happy-kitchen-89482
08/12/2022, 2:55 PMPants does download and verify against a lockfile almost every tool it uses, for hermeticity, but not the python interpreter itself. But it does have very robust interpreter detection and selection mechanism that, for example, knows how to treat pyenv shims
b
bitter-ability-32190
08/12/2022, 2:55 PMhttps://github.com/bazelbuild/rules_docker
These rules do not require / use Docker for pulling, building, or pushing images.
...
Also, unlike traditional container builds (e.g. Dockerfile), the Docker images produced byare deterministic / reproducible.container_image
h
happy-kitchen-89482
08/12/2022, 2:56 PMDoes bazel download an entire c/c++ toolchain for you? or does it assume the presence of xcode tools on macos for example?
n
narrow-judge-15725
08/12/2022, 2:56 PMYeah magic
e
enough-analyst-54434
08/12/2022, 2:59 PMIn practice, FWIW, I think the only issues we've seen with non-hermeticity of tools involve finding a viable Python Interpreter. Most of those issues have to do with Mac system interpreters. I have not heard of issues like this for go, shell or jvm.
👍 1
w
wide-midnight-78598
08/12/2022, 3:02 PM@happy-kitchen-89482 https://bazel.build/tutorials/cc-toolchain-config
Seems to, at least, allow using system gcc/clang - they may be able to download custom ones too (think I saw this in a StackOverflow some time ago)
wide-midnight-78598
08/12/2022, 3:03 PMh
happy-kitchen-89482
08/12/2022, 3:05 PMMac system interpreters are just broken
➕ 2
e
enough-analyst-54434
08/12/2022, 5:23 PMSo it sounds like Bazel toolchains are just a uniform way to specify download or use local. Pants allows you to do said same, but not as uniformly. Each backend does it a bit different.
2 Views