https://pantsbuild.org/ logo
#general
Title
# general
g

gifted-afternoon-62260

08/05/2020, 10:07 AM
Are there plans to support Pants on ARM? And, somewhat related to that, is there support for cross compilation (building for ARM on x86)?
h

hundreds-father-404

08/06/2020, 4:29 PM
Do you mean, for example, the upcoming Mac machines? We haven’t put much thought into it, but generally we want to support the most popular workstations, which has meant for us Linux and macOS. One way or another, I think we’ll need to continue working with the new macs. I suspect it’ll be dependent a bit on how our dependencies handle it, like Rust and the Python libraries we use
I’ve personally alway wanted us to support windows, but that’s a big project
g

gifted-afternoon-62260

08/07/2020, 9:30 AM
Yes exactly, the first question was aiming at e.g. the upcoming ARM Macs, but also Linux on ARM (for example, Raspberry Pi, Nvidia Jetson). The cross-compilation part of the second question means can I compile a pex for ARM on my x86 workstation?
To offer a bit more background, we use Raspberry Pi and Nvidia Jetson for quite a few things, but the
.pex
built with Pants on our x86 desktop/laptop doesn't run on those devices (because of their ARM64 architecture). In an ideal world we could build ARM-compatible binaries on our workstation/CI, but an intermediate solution could be to run Pants directly on an ARM device.
Regarding Windows, I've been using Pants and other Linux tools very successfully on WSL2. WSL2 is a huge step up from WSL1 and I had only minor issues so far, it's basically a well-integrated Linux VM running under Windows. Perhaps a low effort / high reward first step could be to document and CI test Pants on Windows via WSL2 (e.g. the issue I encountered because the repo was not mounted under
~
, a subtle WSL gotcha).
h

hundreds-father-404

08/07/2020, 1:49 PM
Have you seen the platforms flag with Pex and python_binary()? It allows you to do exactly what you’re describing. You can say to build for a different platform than you’re currently using It requires though that all your 3rd party requirements already have built wheels because you wouldn’t be able to build those wheels for the other platform if you were given an sdist (source distribution)
g

gifted-afternoon-62260

08/07/2020, 3:25 PM
Ah, no I hadn't seen it. Thanks, that sounds indeed like the thing we need 🙂 I'll give it a try
h

hundreds-father-404

08/07/2020, 3:28 PM
It’s a little tricky to figure out how to get working. If you haven’t already done so, it can be helpful to download Pex and get things working directly via Pex, and then translate it into your
python_binary
I use
pipx
to install Pex, personally. Then you can run something like
pex --platform=manylinux2014_x86_64-cp-38-cp38 cryptography -o foo.pex
Once you find a platform string that works, add
platforms=["$platform"]
to your
python_binary
pex --help
has some good info too
g

gifted-afternoon-62260

08/07/2020, 3:37 PM
thanks for the tip, the platform str is exactly what I'm struggling with atm 😄
h

hundreds-father-404

08/07/2020, 3:39 PM
Pex should also eagerly validate the platform if its malformed. PEP 425 may be helpful too, which is where it all comes from. https://www.python.org/dev/peps/pep-0425/
And we’re happy to help too 🙂 I’ve tried several times to document this all at https://www.pantsbuild.org/v1.29/docs/python-platforms but always get stuck because it’s particularly hard to write understandable, yet still comprehensive documentation for
g

gifted-afternoon-62260

08/07/2020, 3:51 PM
Thanks, yeah I checked the PEPs, this one seems related as well: https://www.python.org/dev/peps/pep-0599/ (it mentions the
linux-aarch64
/
linux-armv7l
that I'm after)
👍 1
Hm, I'm trying to get a generic platform tag for x86 to work first (since I know that works):
"linux-x86_64-cp-37-cp37"
h

hundreds-father-404

08/07/2020, 4:08 PM
use
cp37m
at the end
with py38, it’s
cp38
, but every earlier Python version ends in
m
for the abi tag
linux-x86_64-cp-37-cp37m
g

gifted-afternoon-62260

08/07/2020, 4:10 PM
Ah thanks, but running the pex still fails with
Copy code
Failed to execute PEX file. Needed manylinux2014_x86_64-cp-27-cp27mu compatible dependencies for:
 1: protobuf
    But this pex only contains:
      protobuf-3.12.2-cp37-cp37m-manylinux1_x86_64.whl
h

hundreds-father-404

08/07/2020, 4:10 PM
What’s the full command? It looks like it needs cp27 but you’re telling it to use cp37
g

gifted-afternoon-62260

08/07/2020, 4:11 PM
even though the interpreter constraint in
pants.toml
is "==3.7.8"
the full command is
Copy code
./pants binary src/py/<org>/<pkg>:<pkg>-bin
18:09:04 [WARN] <string>:1: DeprecationWarning: invalid escape sequence \/

18:09:04.16 [INFO] Wrote <pkg>-bin.pex
h

hundreds-father-404

08/07/2020, 4:12 PM
which btw, you can specify multiple platforms. With Pex, I think you repeat
--platform
twice. With Pants, you put multiple values in the list.
even though the interpreter constraint in pants.toml is “==3.7.8”
Yeah, this is the incomplete warning on those Pants docs.
platform
overrides Python interpreter constraints because the platform already specified a more fine grained interpreter version
Ah, what is the
python_binary
target definition?
g

gifted-afternoon-62260

08/07/2020, 4:13 PM
Copy code
python_binary(
    name = "<pkg>-service",
    dependencies = [
        ":<pkg>",
    ],
    platforms = [
        "linux-x86_64-cp-37-cp37m",
    ],
    entry_point = "org.pkg.main:main",
)
h

hundreds-father-404

08/07/2020, 4:15 PM
Huh. And that error is after you run
dist/<pkg>-bin.pex
?
./pants dependencies --type=3rdparty --transitive src/py/<org>/<pkg>:<pkg>-bin
will also give you the list of every 3rd party req that is being installed
g

gifted-afternoon-62260

08/07/2020, 4:18 PM
Right, the error is after ./dist/<pkg>-bin.pex
Copy code
./pants dependencies --type=3rdparty --transitive src/py/...
18:18:52 [INFO] initializing pantsd...
18:18:52 [WARN] File handle limit is capped to: 4096. To avoid 'too many open file handle' errors, we recommend a limit of at least 10000: please see <https://pants.readme.io/docs/troubleshooting#too-many-open-files-error> for more information.
18:18:55 [INFO] pantsd initialized.
18:18:55 [WARN] <string>:1: DeprecationWarning: invalid escape sequence \/

protobuf==3.12.2
👍 1
h

hundreds-father-404

08/07/2020, 4:20 PM
can you please run
unzip -p dist/<pkg>-bin PEX-INFO
? That file is what stores all the metadata about your built Pex. It’s used to determine how to run your built Pex
g

gifted-afternoon-62260

08/07/2020, 4:22 PM
w/o the platforms tag it works as expected, I'm a bit puzzled by the cp27
Copy code
{
  "always_write_cache": false,
  "build_properties": {
    "class": "CPython",
    "pex_version": "2.1.14",
    "platform": "manylinux2014_x86_64",
    "version": [
      2,
      7,
      17
    ]
  },
  "code_hash": "8134d691270357b1cd3fae9f3c2e629fdc8c161a",
  "distributions": {
    "protobuf-3.12.2-cp37-cp37m-manylinux1_x86_64.whl": "adba80a20818548435da928289efd151b710e821",
    "setuptools-49.2.1-py3-none-any.whl": "0eb682df0273c6573c4056eed158b3f9498fce4a",
    "six-1.15.0-py2.py3-none-any.whl": "0cb007434fc9c1d2ee600dbd964088b69f5d670a"
  },
  "emit_warnings": false,
  "entry_point": "<org>.<pkg>.main:main",
  "ignore_errors": false,
  "inherit_path": "false",
  "interpreter_constraints": [],
  "pex_path": null,
  "requirements": [
    "protobuf==3.12.2",
    "setuptools==49.2.1; python_version >= \"3.5\"",
    "six==1.15.0; python_version != \"3.0.*\" and python_version != \"3.1.*\" and python_version != \"3.2.*\" and python_version >= \"2.7\""
  ],
  "strip_pex_env": true,
  "unzip": false,
  "zip_safe": true
}
h

hundreds-father-404

08/07/2020, 4:24 PM
I am a little confused by that too. Let me reach out to John - he has a better understanding of what is intended. In the meantime, I recommend directly installing Pex via
pipx install pex
, or
pip install pex
, or downloading the binary from https://github.com/pantsbuild/pex/releases Once you get things working how you’d like with Pex, it’s easier to figure out how to translate to Pants. It’s possible that this is a bug in how Pants calls Pex.
g

gifted-afternoon-62260

08/07/2020, 4:25 PM
Thanks @hundreds-father-404!
❤️ 1
h

hundreds-father-404

08/07/2020, 4:42 PM
One not very satisfying workaround is to set
shebang='#!/usr/bin/env python3.7
on the
python_binary
. That will force the runtime to use that shebang, rather than the normal mechanism
e

enough-analyst-54434

08/07/2020, 5:00 PM
@gifted-afternoon-62260 can you confirm that you can run the generated pex with
python3.7 dist/<pkg>-bin
? IE run the PEX file using python3.7 instead of running the PEX file directly.
👍 1
I'm pretty sure this will work and I'll be writing up the root problem here we need to solve and linking an issue shortly.
g

gifted-afternoon-62260

08/07/2020, 5:10 PM
Thanks, AFK atm, will try both approaches when I'm back 👍
@hundreds-father-404 @enough-analyst-54434 both approaches work
One thing I noticed is that when I build with
platforms = ["linux-aarch64-cp-37-cp37m"],
it still says
"platform": "manylinux2014_x86_64",
in the resulting PEX-INFO
h

hundreds-father-404

08/08/2020, 4:02 PM
Yes, that is intended behavior. See the option manylinux in python-setup to change https://www.pantsbuild.org/v2.0/docs/reference-python-setup
g

gifted-afternoon-62260

08/11/2020, 8:37 AM
Thanks @hundreds-father-404, but I'm afraid I don't quite understand. When I run
pex ... --platform=manylinux2014_aarch64-cp-37-cp37m
the resulting PEX_INFO says
"platform": "manylinux2014_x86_64"
, shouldn't it be
"platform": "manylinux2014_aarch64"
(same when I run it through Pants with either
manylinux2014_aarch64-cp-37-cp37m
or
linux-aarch64-cp-37-cp37m
)? I'm sorry but I didn't find anything related to that in the
python-setup
reference, which bit were you referring to?
e

enough-analyst-54434

08/11/2020, 3:10 PM
@gifted-afternoon-62260 the platform in PEX-INFO is the platform of the interpeter used to build the PEX file (it's in "build_properties" which is info about the building of the PEX file). The
--platform=...
argument(s) indicate the platform of the wheels to resolve and install in the PEX file. For example, on my linux machine:
Copy code
$ pex --platform=manylinux2014_aarch64-cp-37-cp37m pex -o example.pex
$ unzip -qc example.pex PEX-INFO | jq .
{
  "always_write_cache": false,
  "build_properties": {
    "class": "CPython",
    "pex_version": "2.1.13",
    "platform": "manylinux2014_x86_64",
    "version": [
      3,
      8,
      5
    ]
  },
  "code_hash": "da39a3ee5e6b4b0d3255bfef95601890afd80709",
  "distributions": {
    "pex-2.1.15-py2.py3-none-any.whl": "bc4c28769a8f357bba9579aa5e24534b622d4ada"
  },
  "emit_warnings": true,
  "ignore_errors": false,
  "inherit_path": "false",
  "interpreter_constraints": [],
  "pex_path": null,
  "requirements": [
    "pex==2.1.15; python_version != \"3.0.*\" and python_version != \"3.1.*\" and python_version != \"3.2.*\" and python_version != \"3.3.*\" and python_version != \"3.4.*\" and python_version < \"3.9\" and python_version >= \"2.7\""
  ],
  "strip_pex_env": true,
  "unzip": false,
  "zip_safe": true
}
g

gifted-afternoon-62260

08/11/2020, 3:48 PM
Ah I see, thanks!