I'm very new to pantsbuild and pex files. I'm tryi...
# general
f
I'm very new to pantsbuild and pex files. I'm trying to run a very simple fast-api application from a pex file using a container/pex packaged using pants. I'm able to run the application using the pex file with
./pants run src/python/maergo:main
. When attempting to run the application via docker with
./pants run src/docker/maergo:python
I encounter the following:
Copy code
Failed to find compatible interpreter on path /usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin.

Examined the following interpreters:
1.) /usr/local/bin/python3.10 CPython==3.10.7
2.)        /usr/bin/python3.9 CPython==3.9.2

No interpreter compatible with the requested constraints was found:

  Failed to resolve requirements from PEX environment @ /root/.pex/unzipped_pexes/dbe958d0761ce1dbfdab65e69c2ae9a4658f75b5.
  Needed cp310-cp310-manylinux_2_31_aarch64 compatible dependencies for:
   1: pydantic!=1.7,!=1.7.1,!=1.7.2,!=1.7.3,!=1.8,!=1.8.1,<2.0.0,>=1.6.2
      Required by:
        fastapi 0.85.2
      But this pex had no ProjectName(raw='pydantic', normalized='pydantic') distributions.
   2: pydantic<2.0.0,>=1.10.2
      But this pex had no ProjectName(raw='pydantic', normalized='pydantic') distributions.
This is a link to my test repository. Is there some other dependency I need to add to my Dockerfile to enable the execution of pex files?
e
So
cp310-manylinux_2_31_aarch64
- appears to be your Python / arch, what is the arch of the
python:3.10-bullseye
image?
They would need to match. Pants currently does not build the PEX in the image, it builds it on your machine and tries to use it in the image. No match, no bueno. There is ongoing work to support running Pants build steps in containers though.
f
support running Pants build steps in containers though.
-> this would be amazing
e
Yeah, I'm not fully versed in where that stands but @happy-kitchen-89482 probably knows better.
SO, @faint-dress-64989 are you ~saying aha - that's it, the image is x86_64?
f
I think so
e
The other possibility is its glibc < 2.31
That's the manylinux_2_31 bit.
f
So I need to add a value for platform to the pex_binary ?
e
Yes.
That can work if all deps are available as wheels for the Python / arch inside the container.
If any are sdists, won't work.
You'd need to pre-build those in the container as wheels and then make them available via
[python-setup] find-links
or similar.
Best is to use
complete_platforms
, that provides an exact match to your container with more one time setup work to generate the complete platform JSON. The `platforms`is lossy and can pick bad glibc versions, etc since its underspecified.
f
I guess it's not clear to me what platform value I need. I'm getting the follow error when trying to package with a platform value:
Copy code
ResolveError: The file or directory 'linux-x86_64-pp-310-none' does not exist on disk in the workspace, so the address 'linux-x86_64-pp-310-none' from the `complete_platforms` from the target src/python/maergo:main cannot be resolved.
Seems to be the same error regardless of what values for complete_platforms. For some context I'm running pants for my machine (osx m1) to compile a pex binary for debians (linux-x86_64-pp-310-none ?)
b
We are building Linux images on macOS too. We do it via complete platforms: we have
3rdparty/platforms/
which contains a
BUILD
like the following, along with
docker_python_3_9_10_bullseye_amd64.json
and
docker_python_3_9_10_bullseye_arm64.json
(amd vs. arm) generated by the commands listed
Copy code
# docker run -it --platform linux/arm64 python:3.9.10-bullseye bash -c 'pip install pex; pex3 interpreter inspect --markers --tags'
# docker run -it --platform linux/amd64 python:3.9.10-bullseye bash -c 'pip install pex; pex3 interpreter inspect --markers --tags'
files(
    name="docker_python_3_9_10_bullseye",
    sources=["docker_python_3_9_10_bullseye_*.json"],
)
(you'll presumably need to change the image used) This is then referred to like
pex_binary(...,  complete_platforms=("3rdparty/platforms:docker_python_3_9_10_bullseye",))
Slightly annoyingly, we build an ARM image on macOS, and an x86-64/AMD image on CI, hence having both complete platforms.
🙌 1
f
@broad-processor-92400 Thanks for chiming in! The disconnect for me is how do I define a complete platform? Could you share your
docker_python_3_9_10_bullseye_amd64.json
file? Also at what point to execute the
docker run
commands specified above?
@enough-analyst-54434 Do understand @broad-processor-92400 set up. Conceptual it seems very useful for my eventual use case of migrating build for spark applications that utilize a particular container (aka complete_platform) over to pantsbuild
e
Huon gave you the command lines right there above!
That's what those docker runs do, they get you the complete platform of the container python they use.
Thanks for that @broad-processor-92400.
👍 1
f
So are those images (docker runs) built outside of pants? The other thing that isn't clear to me is what should be in docker_python_3_9_10_bullseye_amd64.json? Sorry I think something is clicking for me this morning
It looks like those docker commands print some json to stdout. Is this is json file?
oh okay I see (i think) the docker run is in interactive mode so it's installing pex then calling
pex3 interpreter inspect
to generate the complete platform json file
e
Yes, this is a 1 time step outside of Pants to create a complete platform json file you then configure Pants with. You use a container that contains the exact Python you're targeting.
If you don't change the base image/python often, it makes sense. If you do change them often, then less so.
Clearly this is all a bit titchy and hard to understand. Toolchain folks are working on / have added support for running Pants rule steps in a docker container and that is the clean way to handle all this. I'm not sure exactly where that is all at, but you have this approach in the meantime anyway.
f
Thanks again for the help @enough-analyst-54434 and @broad-processor-92400!
👍 1