https://pantsbuild.org/ logo
Join Slack
Powered by
Are there plans to support Pants on ARM? And, some...
# general
g
Are there plans to support Pants on ARM? And, somewhat related to that, is there support for cross compilation (building for ARM on x86)?
h
Do you mean, for example, the upcoming Mac machines? We haven’t put much thought into it, but generally we want to support the most popular workstations, which has meant for us Linux and macOS. One way or another, I think we’ll need to continue working with the new macs. I suspect it’ll be dependent a bit on how our dependencies handle it, like Rust and the Python libraries we use
I’ve personally alway wanted us to support windows, but that’s a big project
g
Yes exactly, the first question was aiming at e.g. the upcoming ARM Macs, but also Linux on ARM (for example, Raspberry Pi, Nvidia Jetson). The cross-compilation part of the second question means can I compile a pex for ARM on my x86 workstation?
To offer a bit more background, we use Raspberry Pi and Nvidia Jetson for quite a few things, but the 
.pex
built with Pants on our x86 desktop/laptop doesn't run on those devices (because of their ARM64 architecture). In an ideal world we could build ARM-compatible binaries on our workstation/CI, but an intermediate solution could be to run Pants directly on an ARM device.
Regarding Windows, I've been using Pants and other Linux tools very successfully on WSL2. WSL2 is a huge step up from WSL1 and I had only minor issues so far, it's basically a well-integrated Linux VM running under Windows. Perhaps a low effort / high reward first step could be to document and CI test Pants on Windows via WSL2 (e.g. the issue I encountered because the repo was not mounted under 
~
, a subtle WSL gotcha).
h
Have you seen the platforms flag with Pex and python_binary()? It allows you to do exactly what you’re describing. You can say to build for a different platform than you’re currently using It requires though that all your 3rd party requirements already have built wheels because you wouldn’t be able to build those wheels for the other platform if you were given an sdist (source distribution)
g
Ah, no I hadn't seen it. Thanks, that sounds indeed like the thing we need 🙂 I'll give it a try
h
It’s a little tricky to figure out how to get working. If you haven’t already done so, it can be helpful to download Pex and get things working directly via Pex, and then translate it into your 
python_binary
I use 
pipx
to install Pex, personally. Then you can run something like 
pex --platform=manylinux2014_x86_64-cp-38-cp38 cryptography -o foo.pex
Once you find a platform string that works, add 
platforms=["$platform"]
to your 
python_binary
pex --help
has some good info too
g
thanks for the tip, the platform str is exactly what I'm struggling with atm 😄
h
Pex should also eagerly validate the platform if its malformed. PEP 425 may be helpful too, which is where it all comes from. https://www.python.org/dev/peps/pep-0425/
And we’re happy to help too 🙂 I’ve tried several times to document this all at https://www.pantsbuild.org/v1.29/docs/python-platforms but always get stuck because it’s particularly hard to write understandable, yet still comprehensive documentation for
g
Thanks, yeah I checked the PEPs, this one seems related as well: https://www.python.org/dev/peps/pep-0599/ (it mentions the 
linux-aarch64
/ 
linux-armv7l
that I'm after)
👍 1
Hm, I'm trying to get a generic platform tag for x86 to work first (since I know that works): 
"linux-x86_64-cp-37-cp37"
h
use 
cp37m
at the end
with py38, it’s 
cp38
, but every earlier Python version ends in 
m
for the abi tag
linux-x86_64-cp-37-cp37m
g
Ah thanks, but running the pex still fails with
Copy code
Failed to execute PEX file. Needed manylinux2014_x86_64-cp-27-cp27mu compatible dependencies for:
 1: protobuf
    But this pex only contains:
      protobuf-3.12.2-cp37-cp37m-manylinux1_x86_64.whl
h
What’s the full command? It looks like it needs cp27 but you’re telling it to use cp37
g
even though the interpreter constraint in 
pants.toml
is "==3.7.8"
the full command is
Copy code
./pants binary src/py/<org>/<pkg>:<pkg>-bin
18:09:04 [WARN] <string>:1: DeprecationWarning: invalid escape sequence \/

18:09:04.16 [INFO] Wrote <pkg>-bin.pex
h
which btw, you can specify multiple platforms. With Pex, I think you repeat 
--platform
twice. With Pants, you put multiple values in the list.
even though the interpreter constraint in pants.toml is “==3.7.8”
Yeah, this is the incomplete warning on those Pants docs. 
platform
overrides Python interpreter constraints because the platform already specified a more fine grained interpreter version
Ah, what is the 
python_binary
target definition?
g
Copy code
python_binary(
    name = "<pkg>-service",
    dependencies = [
        ":<pkg>",
    ],
    platforms = [
        "linux-x86_64-cp-37-cp37m",
    ],
    entry_point = "org.pkg.main:main",
)
h
Huh. And that error is after you run 
dist/<pkg>-bin.pex
? 
./pants dependencies --type=3rdparty --transitive src/py/<org>/<pkg>:<pkg>-bin
will also give you the list of every 3rd party req that is being installed
g
Right, the error is after ./dist/<pkg>-bin.pex
Copy code
./pants dependencies --type=3rdparty --transitive src/py/...
18:18:52 [INFO] initializing pantsd...
18:18:52 [WARN] File handle limit is capped to: 4096. To avoid 'too many open file handle' errors, we recommend a limit of at least 10000: please see <https://pants.readme.io/docs/troubleshooting#too-many-open-files-error> for more information.
18:18:55 [INFO] pantsd initialized.
18:18:55 [WARN] <string>:1: DeprecationWarning: invalid escape sequence \/

protobuf==3.12.2
👍 1
h
can you please run 
unzip -p dist/<pkg>-bin PEX-INFO
? That file is what stores all the metadata about your built Pex. It’s used to determine how to run your built Pex
g
w/o the platforms tag it works as expected, I'm a bit puzzled by the cp27
Copy code
{
  "always_write_cache": false,
  "build_properties": {
    "class": "CPython",
    "pex_version": "2.1.14",
    "platform": "manylinux2014_x86_64",
    "version": [
      2,
      7,
      17
    ]
  },
  "code_hash": "8134d691270357b1cd3fae9f3c2e629fdc8c161a",
  "distributions": {
    "protobuf-3.12.2-cp37-cp37m-manylinux1_x86_64.whl": "adba80a20818548435da928289efd151b710e821",
    "setuptools-49.2.1-py3-none-any.whl": "0eb682df0273c6573c4056eed158b3f9498fce4a",
    "six-1.15.0-py2.py3-none-any.whl": "0cb007434fc9c1d2ee600dbd964088b69f5d670a"
  },
  "emit_warnings": false,
  "entry_point": "<org>.<pkg>.main:main",
  "ignore_errors": false,
  "inherit_path": "false",
  "interpreter_constraints": [],
  "pex_path": null,
  "requirements": [
    "protobuf==3.12.2",
    "setuptools==49.2.1; python_version >= \"3.5\"",
    "six==1.15.0; python_version != \"3.0.*\" and python_version != \"3.1.*\" and python_version != \"3.2.*\" and python_version >= \"2.7\""
  ],
  "strip_pex_env": true,
  "unzip": false,
  "zip_safe": true
}
h
I am a little confused by that too. Let me reach out to John - he has a better understanding of what is intended. In the meantime, I recommend directly installing Pex via 
pipx install pex
, or 
pip install pex
, or downloading the binary from https://github.com/pantsbuild/pex/releases Once you get things working how you’d like with Pex, it’s easier to figure out how to translate to Pants. It’s possible that this is a bug in how Pants calls Pex.
g
Thanks @hundreds-father-404!
❤️ 1
h
One not very satisfying workaround is to set 
shebang='#!/usr/bin/env python3.7
on the 
python_binary
. That will force the runtime to use that shebang, rather than the normal mechanism
e
@gifted-afternoon-62260 can you confirm that you can run the generated pex with 
python3.7 dist/<pkg>-bin
? IE run the PEX file using python3.7 instead of running the PEX file directly.
👍 1
I'm pretty sure this will work and I'll be writing up the root problem here we need to solve and linking an issue shortly.
g
Thanks, AFK atm, will try both approaches when I'm back 👍
@hundreds-father-404 @enough-analyst-54434 both approaches work
One thing I noticed is that when I build with 
platforms = ["linux-aarch64-cp-37-cp37m"],
it still says 
"platform": "manylinux2014_x86_64",
in the resulting PEX-INFO
h
Yes, that is intended behavior. See the option manylinux in python-setup to change https://www.pantsbuild.org/v2.0/docs/reference-python-setup
g
Thanks @hundreds-father-404, but I'm afraid I don't quite understand. When I run 
pex ... --platform=manylinux2014_aarch64-cp-37-cp37m
the resulting PEX_INFO says 
"platform": "manylinux2014_x86_64"
, shouldn't it be 
"platform": "manylinux2014_aarch64"
(same when I run it through Pants with either 
manylinux2014_aarch64-cp-37-cp37m
or 
linux-aarch64-cp-37-cp37m
)? I'm sorry but I didn't find anything related to that in the 
python-setup
reference, which bit were you referring to?
e
@gifted-afternoon-62260 the platform in PEX-INFO is the platform of the interpeter used to build the PEX file (it's in "build_properties" which is info about the building of the PEX file). The 
--platform=...
argument(s) indicate the platform of the wheels to resolve and install in the PEX file. For example, on my linux machine:
Copy code
$ pex --platform=manylinux2014_aarch64-cp-37-cp37m pex -o example.pex
$ unzip -qc example.pex PEX-INFO | jq .
{
  "always_write_cache": false,
  "build_properties": {
    "class": "CPython",
    "pex_version": "2.1.13",
    "platform": "manylinux2014_x86_64",
    "version": [
      3,
      8,
      5
    ]
  },
  "code_hash": "da39a3ee5e6b4b0d3255bfef95601890afd80709",
  "distributions": {
    "pex-2.1.15-py2.py3-none-any.whl": "bc4c28769a8f357bba9579aa5e24534b622d4ada"
  },
  "emit_warnings": true,
  "ignore_errors": false,
  "inherit_path": "false",
  "interpreter_constraints": [],
  "pex_path": null,
  "requirements": [
    "pex==2.1.15; python_version != \"3.0.*\" and python_version != \"3.1.*\" and python_version != \"3.2.*\" and python_version != \"3.3.*\" and python_version != \"3.4.*\" and python_version < \"3.9\" and python_version >= \"2.7\""
  ],
  "strip_pex_env": true,
  "unzip": false,
  "zip_safe": true
}
g
Ah I see, thanks!
Open in Slack
PreviousNext
Powered by