https://pantsbuild.org/ logo
Join Slack
Powered by
Hi. I am trying to pipeline `docker_image` with `p...
# general
m
Hi. I am trying to pipeline 
docker_image
with 
pex_binary
on M1 . docker image created but when i try to run it i get
Copy code
Failed to find compatible interpreter on path /usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin.

Examined the following interpreters:
1.) /usr/local/bin/python3.9 CPython==3.9.13

No interpreter compatible with the requested constraints was found:

  A distribution for pandas could not be resolved for /usr/local/bin/python3.9.
  Found 1 distribution for pandas that do not apply:
  1.) The wheel tags for pandas 1.4.3 are cp39-cp39-macosx_11_0_arm64 which do not match the supported tags of /usr/local/bin/python3.9:
  cp39-cp39-manylinux_2_31_aarch64
  ... 367 more ...
As I understand pex and docker image was built for different architectures. How can I solve it?
e
You'll want to specify the target platforms needed for your `pex_binary`: https://www.pantsbuild.org/docs/reference-pex_binary#codeplatformscode
N.B.: When building a PEX for a foreign platform all its dependencies must be satisfied by pre-built wheels. If a dependency is only available as an sdist, the PEX build will fail. Neither Pex nor Pants supports cross-building currently. If you hit that scenario we can provide more pointers.
m
You mean 
complete_platforms
or 
platforms
? I have tried to set 
pex_bynary.platforms
Copy code
pex_binary(
    name="bin",
    entry_point="...",
    output_path="...",
    platforms=[
        "manylinux_2_31_aarch64-cp-39-cp39",
    ]
)
and got
Copy code
./pants package ::                                                                                                                                 
17:26:45.55 [INFO] Completed: Building etl_cloudtrail_events_to_snowflake/bin.pex with 4 requirements: marshmallow-dataclass<9.0.0,>=8.5.8, nest-asyncio<2.0.0,>=1.5.5, orca-infra==1.12.0.dev2624633449, pandas<2.0.0,>=1.4.3
17:26:45.55 [ERROR] 1 Exception encountered:

  ProcessExecutionFailure: Process 'Building etl_cloudtrail_events_to_snowflake/bin.pex with 4 requirements: marshmallow-dataclass<9.0.0,>=8.5.8, nest-asyncio<2.0.0,>=1.5.5, orca-infra==1.12.0.dev2624633449, pandas<2.0.0,>=1.4.3' failed with exit code 1.
stdout:

stderr:
ERROR: Could not find a version that satisfies the requirement pandas<2.0.0,>=1.4.3
ERROR: No matching distribution found for pandas<2.0.0,>=1.4.3
not exactly understand what should I set and where
e
No, you've got it right with platforms. There is no need to resort to complete_platforms yet. See here: https://pypi.org/project/pandas/1.4.3/#files the 3.9 Linux wheel is for an older version of glibc. Since glibc is generally backwards compatible you can just relax your platform. If you 
s/manylinux/linux/
, then Pants defaults will do the right thing here and default you to manylinux2014 compatibility, for which there is a wheel.
Sorry, 
s/manylinux_2_31/linux/
If you do alot of this and if your images are controlled and few, then investing in creating a complete_platform file for each image / interpreter pair is probably the way to go. Then all devs targeting a container can just point complete_platforms at the appropriate file, hopefully named / addressed in a way that makes it obvious which container / interpreter pair it works for.
m
do you mean to put 
linux_aarch64-cp-39-cp39
as platform? got long error with
Copy code
Failed to resolve for platform linux_aarch64-cp-39-cp39. Resolve requires evaluation of unknown environment marker: 'python_full_version' does not exist in evaluation environment.
e
You need to expand the 
cp-39-cp39
portion to 
cp-3.9.x-cp39
where 
x
you need to find out from the interpreter in your container. See the 
PYVER
note in https://www.pantsbuild.org/docs/reference-pex_binary#codeplatformscode
m
Thanks for the help. Unfortunately when I try to build pex on M1 for another platform it fails. After some investigation I found that some 3rd party that I use depends on another 3rd party that have only 
dist
package. And as I understand pants can’t build it locally for another architecture. Tried to run 
pants package
in docker container so that both pex and docker image will be created for same arch, but there is no 
pantsbuild.pants
whl package for linux on arm64.
Copy code
Bootstrapping Pants using /usr/local/bin/python3.9
Creating the virtualenv PEX.
Downloading the Pex PEX.
SHA256 fingerprint of <https://github.com/pantsbuild/pex/releases/download/v2.1.62/pex> verified.
Installing pantsbuild.pants==2.12.0 into a virtual environment at /root/.cache/pants/setup/bootstrap-Linux-aarch64/2.12.0_py39
ERROR: Could not find a version that satisfies the requirement pantsbuild.pants==2.12.0 (from versions: 0.0.17, 0.0.18, 0.0.20, 0.0.21, 0.0.22, 0.0.23, 0.0.24, 0.0.25, 0.0.26, 0.0.27, 0.0.28, 0.0.29, 0.0.30, 0.0.31, 0.0.32, 0.0.33, 0.0.34, 0.0.35, 0.0.36, 0.0.37, 0.0.38, 0.0.39, 0.0.40, 0.0.41, 0.0.42, 0.0.43, 0.0.44, 0.0.45, 0.0.46, 0.0.47, 0.0.48, 0.0.49, 0.0.50, 0.0.51, 0.0.52, 0.0.53, 0.0.54, 0.0.55, 0.0.56, 0.0.57, 0.0.58, 0.0.59, 0.0.60, 0.0.61, 0.0.62, 0.0.63, 0.0.64, 0.0.65, 0.0.66, 0.0.67, 0.0.68, 0.0.69, 0.0.70, 0.0.71, 0.0.72, 0.0.73, 0.0.74, 0.0.75, 0.0.76, 0.0.77, 0.0.79, 0.0.80, 0.0.81, 0.0.82, 1.0.0, 1.0.1, 1.1.0, 1.2.0, 1.2.1, 1.3.0, 1.4.0, 1.5.0, 1.6.0, 1.7.0)
ERROR: No matching distribution found for pantsbuild.pants==2.12.0
Is it possible to distribute the package also for linux arm64?
e
We'd like to. You can follow along or pitch in here: https://github.com/pantsbuild/pants/issues/12183
It does seem @rhythmic-morning-87313 has got Pants working from a custom build on aarch64: https://pantsbuild.slack.com/archives/C046T6T9U/p1651247157359569 So you might also try that; i.e. check out Pants at the version you use (tags are 
release_X.Y.Z
) and 
./build-support/bin/release.sh build-wheels
. You'd then have to make those wheels available on some corporate server and likely have to alter the 
./pants
script slightly to see that server in addition to PyPI.
I think Joongi just runs pants from sources, which may also work for you, but if not, pre-building the wheels on an aarch64 machine and hosting them should work fine,
If this path is of interest, please speak up with more questions. There are a few details to sort out.
r
Here is how we fallback to build pants from sources
e
N.B.: The patch Joongi mentions is not needed for >= 2.13.0a0: https://github.com/pantsbuild/pants/commit/1368a18e94027b393b41c8a1a4e162c5ce2f6a6d
Open in Slack
PreviousNext
Powered by