https://pantsbuild.org/ logo
Join Slack
Powered by
Does the lack of aarch64 linux support mean that I...
# general
g
Does the lack of aarch64 linux support mean that I can’t use the docker target from a M1 Mac?
e
No, these are independent things. Here you built a pex_binary on Mac and you want to run it in Linux - x86_64 / aarch64 aside - that will never fly. You need to tell Pants currently that the pex_binary is intended for a foreign platform with the platforms or complete_platforms fields.
This is tricky and fiddly and may not work. It only works if all dependencies your pex_binary requires are available as pre-build aarch64 compatible wheels.
g
Is there a way for me to tell it the docker requires the linux pex, but I want to still use the MacOS llocally?
e
Well, the simple way is 2 pex_binary targets side-by-side.
We used to support 
platforms=["current", "...other..."]
- that may work for doing this with one target instead of 2.
g
Perfect, I just found that myself, thank you.
e
Like I said - very titchy - you must have pre-built wheels available for Linux aarch64 out in PyPI or via additional --find-links or --indexes for your full transitive dependency set,
Just 1 sdist-only dep will stop you in your tracks.
(We don't support cross-building)
h
I think soon Pants should be able to support cross-building in a container via Environments? I'm not sure of the scope of that feature
e
But that requires Pants works on Linux aarch64
Which it doesn't quite yet
I think you've done work to make that closer to so though, right @happy-kitchen-89482?
g
Given that I’ve now waited for 10 minutes for it to build a pex with 2 deps, I don’t think this is working…
image.png
Anything look off here?
e
Yeah - didn't you claim the target image was aarch64?
Just a sec for a command that will unambiguously give you the right platform or complete_platform json
g
I’m on a M1 Mac, so it defaults to aarch64, but this will go into production as an x86 image.
e
Ok.
g
But right now I’m still building the pex with the right deps for my image, haven’t even gotten to the docker_image target yet.
e
Copy code
$ docker run --rm -it python:3.11 sh -c 'python -mvenv venv && venv/bin/pip -q install pex && venv/bin/pex3 interpreter inspect -v -i2'
Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg.
{
  "path": "/venv/bin/python",
  "version": "3.11.1",
  "requirement": "CPython==3.11.1",
  "platform": "manylinux_2_31_x86_64-cp-3.11.1-cp311",
  "venv": true,
  "base_interpreter": "/usr/local/bin/python3.11"
}
You want that 
"platform"
for that image, etc.
Better is to use 
complete_platforms
. You get that via:
Copy code
$ docker run --rm -it python:3.11 sh -c 'python -mvenv venv && venv/bin/pip -q install pex && venv/bin/pex3 interpreter inspect --markers --tags -i2'
Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg.
{
  "path": "/venv/bin/python",
  "compatible_tags": [
    "cp311-cp311-manylinux_2_31_x86_64",
...
    "py30-none-any"
  ],
  "marker_environment": {
    "implementation_name": "cpython",
    "implementation_version": "3.11.1",
    "os_name": "posix",
    "platform_machine": "x86_64",
    "platform_python_implementation": "CPython",
    "platform_release": "5.15.79.1-microsoft-standard-WSL2",
    "platform_system": "Linux",
    "platform_version": "#1 SMP Wed Nov 23 01:01:46 UTC 2022",
    "python_full_version": "3.11.1",
    "python_version": "3.11",
    "sys_platform": "linux"
  }
}
Put that JSON in a file, write a BUILD 
file
target for it, and reference that target address in your `complete_platforms`list.
Obviously replace the image I used with the image you care about and the python inside it you care about.
(You can nuke the 
"path"
in the complete platform JSON - just the `"compatile_tags"`and `"marker_environment"`are needed to form a complete platform specification.
And, if you did not realize resolves in the whacky world of Python could depend on fields like 
"platform_version": "#1 SMP Wed Nov 23 01:01:46 UTC 2022"
- well, there you go. They can! Crazy. I've not seen one that does yet, but it's in the spec in PEP-508
g
Well, that is certainly complicated…
e
It is. 1 time setup per image though.
And its the only actual correct thing. `platforms`is a drastic abbreviation of all the info needed for a resolve and is lossy.
g
So right now I have the platforms for my local mac, the aarch64 image I want to use, and the x64 image I want to use:
Or do I have to move to the complete_platform with all those giant blobs of JSON?
e
You need 
complete_platforms
when you need it. It is correct. The 
platforms
(abbreviated platforms) are a lie that often can work. Try the lie and see how it goes, If you get weird errors, the 1st thing to do is switch to telling the whole truth.
Basically all that data in the complete platforms is needed to resolve. Given an abbreviated platform, Pex fills in the details of a complete platform that it can, but the full list of supported tags will be incomplete - so you'll miss resolving some matching wheels you otherwise could. Also, the environment markers will be incomplete; so if there is a requirement encountered in the resolve graph walk that uses one of those missing markers, the resolve will blow up and say something about a missing marker.
g
After 30 minutes of building, the PEX never finished.
I think I’m going to give up on this for now and maybe revisit it in the future.
a
https://pantsbuild.slack.com/archives/C046T6T9U/p1671206760462129?thread_ts=1671206302.655719&cid=C046T6T9U Right: Once there’s a stable Pants version with Target Environments, you should be able to get an x86 binary, by building inside docker using a 
docker_environment
, but this feature isn’t on a stable version yet.
If you’re interested in seeing if an alpha version works, we may be able to help a bit
h
After 30 minutes of building, the PEX never finished.
Eeep... Which version of Pants is this? Using a newer underlying version of Pip may help with resolve-related perf issues. This is in the 
pip_version
option in 
[python]
.
oh, but this will only be available in 2.16
g
@happy-kitchen-89482, this was using 2.14, fresh repo created just to show you this screenshot. It never actually finished, so it was likely stuck in an infinite loop somewhere. So I don’t think making it faster would help. And I wouldn’t mind trying out an alpha build for this whole thing. I also do a lot of development inside of VSCode devcontainers, which are obviously linux containers, but on my M1 Mac that also makes them aarch64 Linux containers, which means I can’t currently use pants in my development environment. We are slowly removing all “local development” off of physical hardware and into things like devcontainers for more consistent and reproducible devX. It makes onboarding a lot easier when I don’t have to give you a list of toolchains to install and I instead just say “attach your IDE of choice to this devcontainer and everything is pre-configured for you to start running the project and developing on it”.
e
@gorgeous-eve-12553 older versions of Pip can actually take 30 minutes to hours in some perverse backtracking cases.
So Benjy's rec. may actually be relevant here.
@gorgeous-eve-12553 do you have interpreter constraints configured? Our defaults of 
>=3.7,<4
don't help at all. Thats a massive range to find compatible artifacts for. If you don't need such a range, its a good idea to set an IC range that makes sense for your project(s).
Even when a 
pex_binary
lists platforms, these are resolved from the lock (assuming you have enabled resolves / use a lock). The lock will still be universal and attempt to solve for the whole IC range.
g
Gross, ok. I can test some things you if you would like me to.
e
Ok, then you'd just try setting 
2.16.0.dev3
as your 
pants_version
and add a:
Copy code
[python]
pip_version = "22.3"
... and set up ICs if not done already! We should never default these for you like we do.
Something like the default 
<4
clause is very bad, that includes the current Python 3.12 alphas which you almost certainly do not mean to support, etc.
h
Yeah, our default ICs are not smart. We should force users to pick specific interpreter versions.
Open in Slack
PreviousNext
Powered by