https://pantsbuild.org/ logo
Join Slack
Powered by
While doing the `pants package (or publish)` in g...
# general
b
While doing the 
pants package (or publish)
in github actions ci, I get the following errors,
Copy code
| #9 1.810 No interpreter compatible with the requested constraints was found:
| #9 1.810
| #9 1.810   Failed to resolve requirements from PEX environment @ /root/.pex/unzipped_pexes/8acfe8754db97c0a34dd5f0d9e2b161523c48582.
| #9 1.810   Needed cp311-cp311-manylinux_2_36_aarch64 compatible dependencies for:
I get this error when I run this command 
PEX_TOOLS=1 /usr/local/bin/python /libs_binary.pex venv --scope=deps --compile /bin/app
(basically when compiling the pex binary??). My build file is simply, build pex binaries of dependencies and sources and then in dockerfile I copy those binaries and compile. For local development I am using macos, and encountered the similar problem. I was able to solve it by passing file generated by 
pex interpreter inspect --markers --tags
to 
complete_platforms
. If anyone has encountered similar problem, would love to hear their approach on how to solve for this, because I am not entirely sure, if the things I did are the recommended way?
b
Complete platforms is the way. Can you share your BUILD file?
b
I am using the following BUILD file
Copy code
python_sources()

python_requirements(
    name="reqs",
)

kwargs = dict(
    # entry_point is required, it is the file which gets executed when we
    # do `pants run ....`. Also only when entry point is defined it does
    # dependency inference and includes all the dependencies in the pex
    # binary including third party dependencies. If entry_point is not
    # defined, default is to start the python interpreter without any
    # dependencies.
    entry_point="hunch.projects.followed_users_post.src.forecast:main",
    layout="packed",
    environment="build_environment",
    execution_mode="venv",
    # it is required to specify linux platform configuration because
    # the python3.11 image is using linux OS and arm64 architecture
    complete_platforms=["hunch:linux-platform"],
)

pex_binary(name="pex-dependencies", include_sources=False, **kwargs)

pex_binary(name="pex-sources", include_requirements=False, **kwargs)

docker_image(
    name="followed_users_post_img",
    repository="followed_users_post",
    registries=[
        env("ECR_REGISTRY", "ECR Registry name e.g. `<http://47474747.dkr.ecr.amazonaws.com|47474747.dkr.ecr.amazonaws.com>`")
    ],
    image_tags=["{build_args.IMAGE_LATEST_TAG}", "{build_args.IMAGE_VERSION_TAG}"],
    extra_build_args=("IMAGE_LATEST_TAG", "IMAGE_VERSION_TAG"),
)
and for complete platforms, I have created a 
file
target with following code:
Copy code
file(name="linux-platform", source="linux-platform.json")
where the file 
linux-platform.json
looks like following
Copy code
{
  "path": "/usr/local/bin/python3.11",
  "compatible_tags": [
    "cp311-cp311-manylinux_2_36_aarch64",
    ...
    ...
    ...
    "py30-none-linux_aarch64",
    "cp311-none-any",
    "py311-none-any",
    "py3-none-any",
    "py310-none-any",
    "py39-none-any",
    "py38-none-any",
    "py37-none-any",
    "py36-none-any",
    "py35-none-any",
    "py34-none-any",
    "py33-none-any",
    "py32-none-any",
    "py31-none-any",
    "py30-none-any"
  ],
  "marker_environment": {
    "implementation_name": "cpython",
    "implementation_version": "3.11.9",
    "os_name": "posix",
    "platform_machine": "aarch64",
    "platform_python_implementation": "CPython",
    "platform_release": "6.6.22-linuxkit",
    "platform_system": "Linux",
    "platform_version": "#1 SMP Fri Mar 29 12:21:27 UTC 2024",
    "python_full_version": "3.11.9",
    "python_version": "3.11",
    "sys_platform": "linux"
  }
}
b
ah okay, looks reasonable. Can you share the full failing build log?
b
I think, I am able to solve it, the above file, I generated on my local (macos m1), but on github actions ci, it is running x86_64. I pulled the image with this specific platform and then generated the complete-platform.json file
I have one question though, in my build file, in 
complete_platforms
argument, I am passing a list of file targets for each platform like 
complete_platforms=["maocs", "linux_arm", "linux_amd"]
, where each element is a file target with it's respective file. When I pass the list, does it mean, it's generating pex binary for each platform?
b
Good question! It'll be a single PEX, but it will contain the platform-specific code appropriate for all of those platforms, so the single pex can be used on macOS, linux arm or linux x86-64/amd, and it'll choose the right versions of dependencies for the current platform.
b
Okay in that case, I can probably refactor it by conditionally choosing files depending on where it's running, either in my local or in ci. If it's in ci, then choose linux-amd64, if it's in local then choose macos. I don't think, if/else statements are allowed inside BUILD files. Or is there an alternative approach? I would want to do this mainly because I am thinking it will keep my image size small.
b
`if`/`else` works fine in a 
BUILD
file, as does an inline 
... if ... else ...
ternary. If you don't already know, you can use the 
env
function to read an environment variable from the environment of the 
pants
command, e.g. 
env("CI") == "1"
might identify running in CI
b
Thanks a lot @broad-processor-92400 for your time and help. I didn't know that BUILD files support 
if/else
statement. I am able to do conditional build. However, my use case has gotten a bit complicated since then. Now I am building an image specific to the platform 
linux/amd64
, by adding 
FROM --platform=linux/amd64 python:3.11-slim
in my dockerfile. I am on macos (
arm64
). When I run the 
pants package path/to/image:target
or 
pants run path/to/image:target -- /bin/app/pex
, it fails because of compatible issues, as pex binaries are built on macos, however in my Dockerfile, I have compile step, so bascially binaries are built on arm64 but i am trying to compile on amd64. So in a way, I understand the issue, but can't figure out how to solve it? Related to this issue, is it possible to build pex binaries inside a docker container?
b
Can you share your current BUILD file and the error you’re now seeing?
b
My build file is this
Copy code
python_sources()

python_requirements(
    name="reqs",
)

# select platform depending on if running in CI or not
# and if running on linux or macos
if env("GITHUB_ACTIONS"):
    platform = ["hunch:ci-platform"]
else:
    # platform = ["hunch:linux-platform"]
    # NOTE: in macos-platform.json, the `path` is hardcoded to my local
    # hunch:macos-platform is required to run the image from local
    platform = ["hunch:linux-platform", "hunch:macos-platform"]

kwargs = dict(
    # entry_point is required, it is the file hich gets executed when we
    # do `pants run ....`. Also only when entry point is defined it does
    # dependency inference and includes all the dependencies in the pex
    # binary including third party dependencies. If entry_point is not
    # defined, default is to start the python interpreter without any
    # dependencies.
    entry_point="hunch.projects.followed_users_post.src.forecast:main",
    layout="packed",
    # environment="build_environment",
    execution_mode="venv",
    # it is required to specify linux platform configuration because
    # the python3.11 image is using linux OS and arm64 architecture
    complete_platforms=[*platform],
)

pex_binary(name="pex-dependencies", include_sources=False, **kwargs)

pex_binary(name="pex-sources", include_requirements=False, **kwargs)

docker_image(
    name="followed_users_post_img",
    repository="followed_users_post",
    registries=[
        env("ECR_REGISTRY", "ECR Registry name e.g. `<http://47474747.dkr.ecr.amazonaws.com|47474747.dkr.ecr.amazonaws.com>`")
    ],
    image_tags=["{build_args.IMAGE_LATEST_TAG}", "{build_args.IMAGE_VERSION_TAG}"],
    extra_build_args=("IMAGE_LATEST_TAG", "IMAGE_VERSION_TAG"),
)
hunch:macos-platform
and 
hunch:linux-platform
are file targets generated by running 
pex3 interpreter inspect --marker --tags
and my Dockerfile is
Copy code
FROM --platform=linux/amd64 python:3.11.9-slim as deps

COPY hunch.projects.followed_users_post/pex-dependencies.pex /libs_binary.pex
RUN PEX_TOOLS=1 /usr/local/bin/python /libs_binary.pex venv --scope=deps --compile /bin/app

FROM --platform=linux/amd64 python:3.11.9-slim as srcs
COPY hunch.projects.followed_users_post/pex-sources.pex /fp_binary.pex
RUN PEX_TOOLS=1 /usr/local/bin/python /fp_binary.pex venv --scope=srcs --compile /bin/app

FROM --platform=linux/amd64 python:3.11.9-slim
COPY --from=deps /bin/app /bin/app
COPY --from=srcs /bin/app /bin/app
And the error I get is:
Copy code
Step 3/9 : RUN PEX_TOOLS=1 /usr/local/bin/python /libs_binary.pex venv --scope=deps --compile /bin/app
 ---> [Warning] The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested
 ---> Running in df07246626c4
Failed to find compatible interpreter on path /usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin.

Examined the following interpreters:
1.) /usr/local/bin/python3.11 CPython==3.11.9

No interpreter compatible with the requested constraints was found:

  Failed to resolve requirements from PEX environment @ /root/.pex/unzipped_pexes/198a489e6d983ed8962c2ed2d9e55fc96c8b7c98.
  Needed cp311-cp311-manylinux_2_36_x86_64 compatible dependencies for:
   1: protobuf!=3.20.0,!=3.20.1,!=4.21.1,!=4.21.2,!=4.21.3,!=4.21.4,!=4.21.5,<5.0.0.dev0,>=3.19.5
      Required by:
        googleapis-common-protos 1.63.0
      But this pex had no ProjectName(raw='protobuf', validated=False, normalized='protobuf') distributions.
   2: protobuf!=3.20.0,!=3.20.1,!=4.21.0,!=4.21.1,!=4.21.2,!=4.21.3,!=4.21.4,!=4.21.5,<5.0.0.dev0,>=3.19.5
      Required by:
        google-api-core 2.19.0
      But this pex had no ProjectName(raw='protobuf', validated=False, normalized='protobuf') distributions.
   3: protobuf<5.0.0dev,>=3.19.0
      Required by:
        proto-plus 1.23.0
      But this pex had no ProjectName(raw='protobuf', validated=False, normalized='protobuf') distributions.
   4: charset-normalizer<4,>=2
      Required by:
        requests 2.32.3
      But this pex had no ProjectName(raw='charset-normalizer', validated=False, normalized='charset-normalizer') distributions.
   5: grpcio<2.0dev,>=1.33.2; extra == "grpc"
      Required by:
        google-api-core 2.19.0
      But this pex had no ProjectName(raw='grpcio', validated=False, normalized='grpcio') distributions.
   6: protobuf>=4.21.6
      Required by:
        grpcio-status 1.62.2
      But this pex had no ProjectName(raw='protobuf', validated=False, normalized='protobuf') distributions.
   7: grpcio>=1.62.2
      Required by:
        grpcio-status 1.62.2
      But this pex had no ProjectName(raw='grpcio', validated=False, normalized='grpcio') distributions.
   8: grpcio<2.0dev,>=1.49.1; python_version >= "3.11" and extra == "grpc"
      Required by:
        google-api-core 2.19.0
      But this pex had no ProjectName(raw='grpcio', validated=False, normalized='grpcio') distributions.
   9: google-crc32c<2.0dev,>=1.0
      Required by:
        google-resumable-media 2.7.0
      But this pex had no ProjectName(raw='google-crc32c', validated=False, normalized='google-crc32c') distributions.
   10: pandas>=0.24.2
      Required by:
        db-dtypes 1.2.0
      But this pex had no ProjectName(raw='pandas', validated=False, normalized='pandas') distributions.
   11: pyarrow>=3.0.0
      Required by:
        db-dtypes 1.2.0
        pandas-gbq 0.23.0
      But this pex had no ProjectName(raw='pyarrow', validated=False, normalized='pyarrow') distributions.
   12: numpy>=1.16.6
      Required by:
        db-dtypes 1.2.0
        pandas-gbq 0.23.0
      But this pex had no ProjectName(raw='numpy', validated=False, normalized='numpy') distributions.
   13: pandas>=1.1.4
      Required by:
        pandas-gbq 0.23.0
      But this pex had no ProjectName(raw='pandas', validated=False, normalized='pandas') distributions.

stderr:
DEPRECATED: The legacy builder is deprecated and will be removed in a future release.
            Install the buildx component to build images with BuildKit:
            <https://docs.docker.com/go/buildx/>

The command '/bin/sh -c PEX_TOOLS=1 /usr/local/bin/python /libs_binary.pex venv --scope=deps --compile /bin/app' returned a non-zero code: 1
b
Does 
hunch:linux-platform
reference `aarch64`/`arm64` or 
x86_64
/`amd64`?
b
It reference 
aarch64
b
Since you now have the Linux/amd64 platform arg, I think that’s the source: even when running locally the docker image is amd64/x86-64 (like CI), not aarch64
b
understood. Is it possible to build binaries specific to amd64 locally e.g. building binaries inside a docker container?
b
yes, you can either build things within a docker container via environments, as you've discovered in the other thread. If these pexes are only used in the docker file with 
FROM --platform linux/amd64
then I'd expected it to use if you use the complete platform file for that platform (the same one as CI)
5 Views
Open in Slack
PreviousNext
Powered by