https://pantsbuild.org/ logo
#general
Title
# general
p

purple-plastic-57801

03/04/2023, 7:16 PM
Can / how does pants handle python lock files for different architectures? TL;DR: we have a mono repo with four source roots Shared Api Cloud Observatory Observatory runs on aarch64. Shared is a library that is shared between cloud an observatory. The current make solution that is being leveraged has become quite painful to maintain.
e

enough-analyst-54434

03/04/2023, 7:20 PM
It currently locks for Linux + Mac and all architectures for both.
p

purple-plastic-57801

03/04/2023, 7:20 PM
Do I have to configure it to do so or does it always generate lock files for all architectures?
e

enough-analyst-54434

03/04/2023, 7:20 PM
That's the default. No config.
p

purple-plastic-57801

03/04/2023, 7:21 PM
This seems too good to be true. 😜
How does distribution of packages work in this world? I’m not very familiar with PEX packaging.
e

enough-analyst-54434

03/04/2023, 7:22 PM
The tool it uses is Pex for this:
Copy code
pex3 lock create --style universal --resolver-version pip-2020-resolver --target-system linux --target-system mac ...
What do you mean by distribution of packages? PEX is a binary.
PEX is Like Google PAR, it just does not include a Python interpreter. That has to be on the targtet host already.
p

purple-plastic-57801

03/04/2023, 7:23 PM
I’m not exactly sure. At the end of the day I want to push out the observatory and have it work on the architecture that it is running on.
e

enough-analyst-54434

03/04/2023, 7:24 PM
Yeah, so you package a PEX binary. That's 1 executable file. Then scp and run remote.
Do you develop on production or something else like Mac for example?
p

purple-plastic-57801

03/04/2023, 7:27 PM
For development, the teams split across Mac and Linux, locally and via VS code, remote. Production systems are containers that either run in kubernetes in the cloud, or at the edge on raspberry pis via balena.io
e

enough-analyst-54434

03/04/2023, 7:28 PM
Ok, so 2 options: 1: Use https://www.pantsbuild.org/docs/reference-pex_binary#codecomplete_platformscode 2: Use environments - aka build the Pex in a local docker container matching prod: https://www.pantsbuild.org/docs/environments
So #1 allows you to generate a json file on the target host 1 time, then refer to that locally when building PEXes.
That JSON describes the remote Python interpreter fully.
This technique has the limitation of every platform-specific dependency needing to have a pre-built wheel resolvable though.
Option 2 has no such limitation of course.
The one further twist that may or may not be useful to you is 1 PEX can contain wheels for many platforms - it'll pick the right subset when it runs.
p

purple-plastic-57801

03/04/2023, 7:30 PM
Interesting. Fortunately, the container builds for arm take place on arm hosts.
I see. Unfortunately, a current pain point is the size of the containers that are pushed to the edge due to terrible networking available at those locations the smaller the better.
e

enough-analyst-54434

03/04/2023, 7:31 PM
Ok, if you use production to build production you don't need 1 or 2 and you are officially sane.
Many, many people are insane; thus we support #1 and #2.
p

purple-plastic-57801

03/04/2023, 7:32 PM
When it comes to things like OCI container layers, does the pex package only contain a single artifact? Essentially the requirements to change too often. Can I put the dependencies on one layer and application code on another?
e

enough-analyst-54434

03/04/2023, 7:32 PM
But yes.
p

purple-plastic-57801

03/04/2023, 7:34 PM
Thanks. I will.
e

enough-analyst-54434

03/04/2023, 7:35 PM
A dogfood lock example for Pants' own internal requirements: https://github.com/pantsbuild/pants/blob/main/3rdparty/python/user_reqs.lock That shows the multi-platformness if you look at various locked dependency artifacts. We ship to {Linux, Mac} x {aarch64, x84_64} from that 1 lock.
p

purple-plastic-57801

03/04/2023, 7:37 PM
:)
Thank you. I’ll dive in.
e

enough-analyst-54434

03/04/2023, 7:38 PM
Bear in mind this is ~obviously a bad idea. You rely on package authors being sane and foo-1.2.3-mac being the same as foo-1.2.3-linux ...
They generally are sane though.
p

purple-plastic-57801

03/04/2023, 7:42 PM
Yeah, we’ve run into some issues when I was in the case. That said the build takes place on the appropriate architecture.., that might be less of an issue?
e

enough-analyst-54434

03/04/2023, 7:50 PM
Well, lock and build are two different things. My point is mainly that it would be more sane to both generate a lock file and test it works in the same commit. Generally you can't do that, except perhaps with robust CI, for a multiplatform lock since you necessarily create the lock on just 1 platform machine.
The
pex3 lock create
supports the style of only locking for the current platform (
--style strict
- and that's the default), but letting the full lockfile containing many such locks, 1 per platform. Pants doesn't use this, it uses the
--style universal
lock where its 1 lockfile, 1 contained lock, but that lock contains artifacts for all architectures.
3 Views