Okay, sigh... need some assistance with platforms ...
# general
Okay, sigh... need some assistance with platforms and pex I'm afraid. I'm really trying to shift us to using pex-based docker images; the reduction in code and the reduction in built image size is just compelling... ...but we have several developers that use MacOS. Sigh. Which screws things up considerably. I see a number of solutions, from trying to work out platform strings to creating a Dockerized build platform and building docker within docker using pants (ugh), but I'd love some sort of solution because I'm full sure this is a solved problem. If I'm going to start with the seemingly-obvious, what platform string should I use? I tried looking at pex within my container and under "platform" it said
Copy code
For the current interpreter at                       /usr/local/bin/python3.9 the full platform string is                        manylinux_2_28_x86_64-cp-3.9.9-cp39
... which is what you get from `pex3 interpreter inspect --all --verbose --indent 4:
Copy code
  "path": "/usr/local/bin/python3.9",
  "version": "3.9.9",
  "requirement": "CPython==3.9.9",
  "platform": "manylinux_2_28_x86_64-cp-3.9.9-cp39",
  "venv": false
I tried "pex3 interpreter inspect --markers --tags" and got... well, showers of things, which didn't help. Building my pex setting
to the above string errored out with
(https://pypi.org/project/PyYAML/5.4.1/#files) which has cp39 manylinux1 and manylinux2014 wheels, so I can't even get past something that seems to have compatible wheels much less the problems I'm going to have when I get to source-only distributions (build a wheel in pants somehow?). I know pants is relatively new to the Docker scene, but "build a pex that can run on docker" seems a useful thing, even if we need to spin up a container to do it. But I can't sell this to my team unless they can build locally (the old version was built entirely within Docker so worked just fine). If that's the way forward, is there a "known best way" to build a pants repository within docker that maintains some of the pants goodness like caching etc?
It turns out the tag you should use for Pex's
or Pant's `pex_binary.platforms`is `linux_x86_64-cp-3.9.9-cp39`; i.e.: Drop the manylinux bit. Both Pex via
and Pants via
[python-setup] resolver_manylinux
default to assuming foreign Linux platforms are
compliant / compatible. I.E.: this works
Copy code
$ pex --platform=linux_x86_64-cp-3.9.9-cp39 PyYAML==5.4.1 --python-shebang "/usr/bin/env python3.9" -opyyaml.pex
$ ./pyyaml.pex -c 'import yaml; print(yaml.__file__)'
So, there's clearly some docs that need to be fixed up at the very least. But, first, to explain some of your journey and missed paths, read on:
I'm not sure what led you to
pex3 interpreter inspect --markers --tags
but that shower of things would also work! You need to save that in a file and then use via
in Pex and via
in Pants (https://www.pantsbuild.org/v2.10/docs/reference-pex_binary#codecomplete_platformscode).
Copy code
$ pex3 interpreter inspect --python python3.9 --markers --tags > my_laptop.py39
$ pex --complete-platform my_laptop.py39 PyYAML==5.4.1 --python-shebang "/usr/bin/env python3.9" -opyyaml.pex
$ ./pyyaml.pex -c 'import yaml; print(yaml.__file__)'
@echoing-farmer-15630 the
argument to Pex is very old and was always a hack. There is simply not enough information in those abbreviated platform strings to derive all data needed to solve a resolve (environment markers and all compatible tags). So using
is really the way to go. It allows you to resolve exactly as if you were on the machine using the python you grabbed that complete platform data with, except for not being able to build sdists.
It might even make sense for Pants to deprecate platforms and force complete_platforms. Its a bit more up fron work to go gather your fleet complete platform data and check in those files in the repo, but its fully correct and unambiguous.
👍 1
But you end up having to go through more work as you have here figuring everything out anyway. Sussing out platforms to get MacOS building Linux PEXes is a very very common headache.
🙂 What led me down the
pex3 interpreter inspect --markers --tags
was an info string when running the pex executable within a container in a desperate effort to figure something out... Right now I'm running down a rabbit hole with remote execution, which I think is EVENTUALLY going to be the solution (some can build a local buildfarm on their system, others can use a cloud-deployed version, and either way we get what we want: a versioned system for build tools that does something more repeatable). But I'll try the complete-platforms thing in a sec. ...sigh. Ran the
pex3 interpreter inspect...
in the docker container image, did complete_platforms and this time it failed with SQLAlchemy-Paginator, which is a source dist. And we have a few more (including our own git repos). Thanks for the help but it feels like this is a bust for us; perhaps the remote execution path will be more fruitful.
So, this has worked for others (Twittter, 4SQ, ...) , but they have all doubled down and invested in a wheel building infrastructure to pre-build wheels for all platforms they target and then serve those wheels. They then hookup that internal wheel repository with either
[python-repos] repos
[python-repos] indexes
depending on how pre-built wheels are hosted. It's really the only way you can use
for any real world projects, you must invest in pre-building wheels.
I can see that, although if I have to double down on building more infrastructure, the buildfarm/REAPI cache seems like better ROI across projects. Ah well. The wheels repo isn't a bad idea either, mind you.
Yeah, pick your poison, but the reapi is more general too. It will work for not-Python issues of same style.
The 3rd option is to eliminate all macs from your fleet! That is an excellent option iff you don't do macOS / iOS application development.
O how I wish I could. I've had to do plenty of "but why doesn't this work on my mac?" support questions. But then I'm choosing to run arch on a homebuilt PC as my daily drive, so I'm just asking for trouble anyway. Seriously though, thanks--that was great support and I learned a lot, and still very impressed with pants. We'll get something going even if I have to back out that change from git in the short term.