Can / how does pants handle python lock files for ...
# generalp
purple-plastic-57801
03/04/2023, 7:16 PMCan / how does pants handle python lock files for different architectures?
TL;DR: we have a mono repo with four source roots
Shared
Api
Cloud
Observatory
Observatory runs on aarch64.
Shared is a library that is shared between cloud an observatory.
The current make solution that is being leveraged has become quite painful to maintain.
e
enough-analyst-54434
03/04/2023, 7:20 PMIt currently locks for Linux + Mac and all architectures for both.
p
purple-plastic-57801
03/04/2023, 7:20 PMDo I have to configure it to do so or does it always generate lock files for all architectures?
e
enough-analyst-54434
03/04/2023, 7:20 PMThat's the default. No config.
p
purple-plastic-57801
03/04/2023, 7:21 PMThis seems too good to be true. 😜
purple-plastic-57801
03/04/2023, 7:22 PMHow does distribution of packages work in this world? I’m not very familiar with PEX packaging.
e
enough-analyst-54434
03/04/2023, 7:22 PMThe tool it uses is Pex for this:
Copy code
pex3 lock create --style universal --resolver-version pip-2020-resolver --target-system linux --target-system mac ...enough-analyst-54434
03/04/2023, 7:22 PMWhat do you mean by distribution of packages? PEX is a binary.
enough-analyst-54434
03/04/2023, 7:23 PMPEX is Like Google PAR, it just does not include a Python interpreter. That has to be on the targtet host already.
p
purple-plastic-57801
03/04/2023, 7:23 PMI’m not exactly sure. At the end of the day I want to push out the observatory and have it work on the architecture that it is running on.
e
enough-analyst-54434
03/04/2023, 7:24 PMYeah, so you package a PEX binary. That's 1 executable file. Then scp and run remote.
enough-analyst-54434
03/04/2023, 7:24 PMDo you develop on production or something else like Mac for example?
p
purple-plastic-57801
03/04/2023, 7:27 PMFor development, the teams split across Mac and Linux, locally and via VS code, remote.
Production systems are containers that either run in kubernetes in the cloud, or at the edge on raspberry pis via balena.io
e
enough-analyst-54434
03/04/2023, 7:28 PMOk, so 2 options:
1: Use https://www.pantsbuild.org/docs/reference-pex_binary#codecomplete_platformscode
2: Use environments - aka build the Pex in a local docker container matching prod: https://www.pantsbuild.org/docs/environments
enough-analyst-54434
03/04/2023, 7:28 PMSo #1 allows you to generate a json file on the target host 1 time, then refer to that locally when building PEXes.
enough-analyst-54434
03/04/2023, 7:28 PMThat JSON describes the remote Python interpreter fully.
enough-analyst-54434
03/04/2023, 7:29 PMThis technique has the limitation of every platform-specific dependency needing to have a pre-built wheel resolvable though.
enough-analyst-54434
03/04/2023, 7:29 PMOption 2 has no such limitation of course.
enough-analyst-54434
03/04/2023, 7:30 PMThe one further twist that may or may not be useful to you is 1 PEX can contain wheels for many platforms - it'll pick the right subset when it runs.
p
purple-plastic-57801
03/04/2023, 7:30 PMInteresting. Fortunately, the container builds for arm take place on arm hosts.
purple-plastic-57801
03/04/2023, 7:31 PMI see. Unfortunately, a current pain point is the size of the containers that are pushed to the edge due to terrible networking available at those locations the smaller the better.
e
enough-analyst-54434
03/04/2023, 7:31 PMOk, if you use production to build production you don't need 1 or 2 and you are officially sane.
enough-analyst-54434
03/04/2023, 7:32 PMMany, many people are insane; thus we support #1 and #2.
p
purple-plastic-57801
03/04/2023, 7:32 PMWhen it comes to things like OCI container layers, does the pex package only contain a single artifact?
Essentially the requirements to change too often. Can I put the dependencies on one layer and application code on another?
e
enough-analyst-54434
03/04/2023, 7:32 PMenough-analyst-54434
03/04/2023, 7:33 PMBut yes.
p
purple-plastic-57801
03/04/2023, 7:34 PMThanks. I will.
e
enough-analyst-54434
03/04/2023, 7:35 PMA dogfood lock example for Pants' own internal requirements: https://github.com/pantsbuild/pants/blob/main/3rdparty/python/user_reqs.lock
That shows the multi-platformness if you look at various locked dependency artifacts. We ship to {Linux, Mac} x {aarch64, x84_64} from that 1 lock.
p
purple-plastic-57801
03/04/2023, 7:37 PM:)
purple-plastic-57801
03/04/2023, 7:37 PMThank you. I’ll dive in.
e
enough-analyst-54434
03/04/2023, 7:38 PMBear in mind this is ~obviously a bad idea. You rely on package authors being sane and foo-1.2.3-mac being the same as foo-1.2.3-linux ...
enough-analyst-54434
03/04/2023, 7:38 PMThey generally are sane though.
p
purple-plastic-57801
03/04/2023, 7:42 PMYeah, we’ve run into some issues when I was in the case. That said the build takes place on the appropriate architecture.., that might be less of an issue?
e
enough-analyst-54434
03/04/2023, 7:50 PMWell, lock and build are two different things. My point is mainly that it would be more sane to both generate a lock file and test it works in the same commit. Generally you can't do that, except perhaps with robust CI, for a multiplatform lock since you necessarily create the lock on just 1 platform machine.
enough-analyst-54434
03/04/2023, 7:51 PMThe
pex3 lock create--style strict--style universal3 Views