Hi all. I'm new to Pants but making some headway o...
# general
w
Hi all. I'm new to Pants but making some headway over the last few days getting my Python monorepo set up to use Pants. I am loving the way Pants introspects into the whole project and infers a bidirectional dependency graph to do its magic. But I'm starting to get into the weeds a bit. My goal is to build a pex binary of a FastAPI app and a Docker image that uses the generated pex file and that gets deployed in a container on an AWS ECS instance. Currently I can do the following: • Build a pex file of app and its dependencies and execute it locally to verify it works • Build a docker image using Pants to work out the build context (pretty neat) • Run the Docker image in a container and watch it error out. šŸ˜‰ I'm hitting the newbie "Failed to find compatible interpreter on path" issue that a few others have mentioned here. This sent me back to the docs, GitHub discussions and Slack conversations, where I've learned that my simplistic pex build is not yet sufficient for running the generated artifact in a container: I need to build the pex with some platform metadata. It looks like there are two fields I can use to ensure the right wheels are available in the image:
platforms
and
complete_platforms
. The former accepts a format defined by Pex like "linux_x86_64-cp-3.11.7-cp311" and the latter accepts a
file
target that points to a JSON doc that I can generate using pex. This is where I'm starting to get a bit lost. First, what determines whether I should use
platforms
or
complete_platforms
? Second, I see that Pants offers a preview of an Environments concept where I can define a local_environment or docker_environment that each can be configured with a platform. Then the BUILD for the pex_binary points to the named environment instead of directly to a platform string or complete_platform JSON. If I understand correctly, in my use case of wanting my app to run in a distro like Alpine Linux in prod, different from my local Linux OS I either need to make use of Environments or I need to run Pants in an Alpine Linux docker image. Is that correct? (Note: reading people talking about Environments is the first I've encountered the idea that Pants be installed in a docker container.) If I use environments do I need to create an environment for every platform that my devs are developing on as well as the target deployment platform? When I read that Environments is still not a stable feature, is there a little more color that could be put on that? What's the level of confidence regarding breaking changes now, and the strength of recommendation for using this feature today? I can't figure out the right path. Can someone help with these specific questions and/or walk me through the decision-making process?
g
Hey David, that sounds like a good start. If you search for FastAPI I think you'll find others also using it and maybe pick up some hints from those conversations. I'm not familiar with the domain, but I've understood there may be some specific issues around structure with it. Starting from the top, I'd definitely suggest using
complete_platforms
. While it isn't yet,
platforms
is being deprecated because it is too inexact and so adds more friction that it removes. With that said, I'm not certain that
complete_platforms
is needed for that error message. You can disregard that if you've done due diligence, but we've also had that show up by having a mismatch between interpreter version on host and in container, or between interpreter constraints and what's in the container. Environments I have less to add. I've never used it unfortunately, and I haven't seen a lot of work going into changing it largely. Not sure who'd be best to answer. There's a tracking issue for stabilization issues here: https://github.com/pantsbuild/pants/issues/17355. With that said, my understanding is that you'd need one "local" environment at most, and a target/docker environment.
w
Thanks, Tom. I think everything is fine in terms of the FastAPI specifics since I can execute the pex file and the app fires up in uvicorn as I expect. I tried following the docs for using Environments to resolve the "failed to find compatible interpreter" issue and get a different error:
Copy code
docker: Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: exec: "/bin/aerial-api.pex": stat /bin/aerial-api.pex: no such file or directory: unknown.
My Dockerfile (patterned after this example) looks like this:
Copy code
FROM python:3.11-slim-bookworm

ENTRYPOINT ["/bin/aerial-api.pex"]
COPY src.python/aerial-api.pex /bin
When I create a container from the image and inspect the file system I see that the aerial-api.pex archive is indeed missing because it has been extracted into the bin directory. This explains the "no such file" error, but I'm not sure what to do about it and don't understand why this Dockerfile seems to work differently than the one in the pantsbuild example-docker repo. The src/python/BUILD file for the pex binary in question looks like this, if that helps:
Copy code
python_sources()

pex_binary(
    name="aerial-api",
    entry_point="api.main",
    environment="docker_linux",
    # Optimal settings for Docker builds
    layout = "packed",
    execution_mode = "venv",
)
And the root BUILD file looks like this:
Copy code
python_requirements(
    name="reqs",
    source="requirements.txt",
)

local_environment(
  name="local_linux",
  compatible_platforms=["linux_x86_64"],
)

docker_environment(
    name="docker_linux",
    platform="linux_x86_64",
    image="python:3.11-slim-bookworm",
    python_bootstrap_search_path=["<PATH>"]
)

# Default all environment-aware targets to the docker environment
__defaults__(all=dict(environment="docker_linux"))
@gorgeous-winter-99296 I realize you said you didn't have any experience with the environments feature. It looked a bit simpler to implement so I tried that first. Will give the other approach a go now using
complete_platforms
json.
I see. The pex extraction behavior was because of the
layout="packed"
configuration of the pex_binary. The default layout is "zipapp" which packages the pex as an actual pex archive. So allowing the default layout and execution_mode settings solves the "no such file" error I was getting. The server starts now. But it doesn't handle requests. Still... getting closer.
šŸ‘ 1