https://pantsbuild.org/ logo
Join Slack
Powered by
Hey everyone, we're having some issues with gettin...
# general
r
Hey everyone, we're having some issues with getting a pants-packaged flask app to run on *G*oogle *A*pp *E*ngine (GAE). Project Structure:
Copy code
cli/
├─ run.py
├─ evaluate.py
├─ BUILD
data/
main_library/
web/
├─ server.py
├─ BUILD
app.yaml
pants.toml
requirements.txt
BUILD
Currently: • Our flask app is run by gunicorn (invoked through the CLI) on Google App Engine (without docker) What We Want: • The flask app should be run by 
gunicorn
(invoked through the CLI) • It should be packaged as Docker image and uploaded to the Google Artefact Registry • Deployment then would merely mean to pull a new image version into GAE (no pants involved) What We Tried: We have tried various ways of achieving the above and are currently stuck with building a 
pex
target of the 
web/server.py.
We build it cross-platform so it works both on our local machines (OSX) and the Docker image (linux) We ran into some issues with loading resources included in the 
pex
file. Unzipping the 
pex
after copying it onto the Docker image solved the issue, but feels like it might be a hacky solution. The image runs fine locally but not on app engine. We thought running flask via gunicorn and Docker should be trivial, but for some reason we can’t seem to get it to work. Has anyone here experience with this specific deployment scenario?
b
If there’s specific errors, I imagine we’ll be able to help more effectively if we can see you BUILD targets. However, before digging into that, have you seen https://blog.pantsbuild.org/optimizing-python-docker-deploys-using-pants/ ? In particular, the optimised versions end up just creating a normal venv within the image via the pex tools. (We use something like this for running a FastAPI server with uvicorn, and it works well)
👀 2
w
I might be able to help out here shortly, I’m doing similar-ish in azure. Once I’m back at my computer, I’ll try to grab my build files
Alright, this is using FastAPI - but I think similar-ish concept. I had to also unpack the pex on the Azure appservice previously, but that was because I was originally packaging these into 
scie
but AppService couldn't just run a random executable (which is stupid). I'm using Azure's docker app services - but I think with what I'm doing here, you can narrow down some of the stuff on your side as well.
Copy code
# BUILD.pants

pex_binary(
    name="myapi",
    dependencies=[
        ":libmyapi",
        "//:reqs#uvicorn",
        "//:reqs#gunicorn",
    ],
    include_tools=True,
    platforms=[
        "linux-x86_64-cp-311-cp311",
        "macosx-13.3-arm64-cp-311-cp311",
        "macosx-13.3-x86_64-cp-311-cp311",
    ],
)

docker_image(
    name="dockerized_myapi",
    image_tags=["mytag1"],
    registries=["myregistry"],
    repository="myrepo",
    skip_push=True,
)
Copy code
# Dockerfile

# Following instructions here for reduced startup time and smaller footprint
# <https://pex.readthedocs.io/en/latest/recipes.html?ref=blog.pantsbuild.org#pex-app-in-a-container>
# There are still some more optimizations that could be done, see below:
# <https://blog.pantsbuild.org/optimizing-python-docker-deploys-using-pants/>

FROM python:3.11-slim as deps
COPY backend.myapi/myapi.pex /api.pex
RUN PEX_TOOLS=1 /usr/local/bin/python3.11 /api.pex venv --scope=deps --compile /bin/app

FROM python:3.11-slim as srcs
COPY backend.myapi/myapi.pex /api.pex
RUN PEX_TOOLS=1 /usr/local/bin/python3.11 /api.pex venv --scope=srcs --compile /bin/app

FROM python:3.11-slim
COPY --from=deps /bin/app /bin/app
COPY --from=srcs /bin/app /bin/app

EXPOSE 8000
ENTRYPOINT ["/bin/app/bin/gunicorn", "myapi.main:app", "--bind=0.0.0.0", "--timeout", "600", "--forwarded-allow-ips=*", "-k", "uvicorn.workers.UvicornWorker", "-w", "1"]
Then I use something like 
pants package backend/myapi::
to build the docker image, and then in my case, I just deploy using 
podman
- but you might be able to use Pants for that too, if you setup the registries and logins
r
Thanks for the help! I've now refactored my code to make use of the virtual environments and it works quite nicely and without the need to unzip the pex file. Now for the actual deployment I have not been able to find a solution to this:
Copy code
WARNING: Deployment of service [default] will ignore the skip_files field in the configuration file, because the image has already been built.
Updating service [default] (this may take several minutes)...failed.       
                                
ERROR: (gcloud.app.deploy) Error Response: [9] An internal error occurred while processing task /app-engine-flex/flex_await_healthy/flex_await_healthy>2023-12-20T15:31:53.445Z3500.lclhrb.1: exec /bin/app/bin/gunicorn: exec format error
Apparently the 
exec format error
usually indicates a mismatch in platforms. We cross-compile from 
osx
onto 
linux_arm64
and the docker image works perfectly when deployed locally. It just doesn't work once deployed to GAE. Any ideas where this could come from?
w
GAE uses arm64?
r
Yeah that seems a bit weird, right? I've been trying to set the 
docker_environment
platform to 
linux_x86_64
but I then get this following error:
Copy code
16:39:34.25 [ERROR] 1 Exception encountered:

Engine traceback:
  in `package` goal

IntrinsicError: Failed to create Docker container: DockerResponseServerError { status_code: 404, message: "image with reference sha256:09485c9.....2f49afbba62cd5bd387 was found but does not match the specified platform: wanted linux/amd64, actual: linux/arm64/v8" }
So maybe the actual issue lies here?
Here's the BUILD file for reference:
Copy code
docker_environment(
    name="python_linux",
    platform="linux_arm64",  # Seems like the correct platform?
    image="python:3.10.13-bookworm",
    python_bootstrap_search_path=["<PATH>"],
)

pex_binary(
    name="server",
    dependencies=[
        "main_libarary:main_libarary",
        "//:reqs#gunicorn",
        "web/server.py",
        "systems:systems",
    ],
    environment="linux", # This maps to the docker_environment python_linux above
    include_tools=True,
    layout="packed",
    execution_mode="venv"
)

docker_image(
    name="server_image",
)
w
Do you need to specify your pex environments any more precisely?
Copy code
platforms=[
        "linux-x86_64-cp-311-cp311",
        "macosx-13.3-arm64-cp-311-cp311",
        "macosx-13.3-x86_64-cp-311-cp311",
    ],
r
When specifying the 
platforms
parameter through the pex_binary I get some issues with pyyaml
Copy code
No interpreter compatible with the requested constraints was found:

  A distribution for pyyaml could not be resolved for /usr/local/bin/python3.10.
  Found 2 distributions for pyyaml that do not apply:
  1.) The wheel tags for PyYAML 6.0.1 are cp310-cp310-macosx_11_0_arm64 which do not match the supported tags of /usr/local/bin/python3.10:
  cp310-cp310-manylinux_2_36_aarch64
  ... 517 more ...
  2.) The wheel tags for PyYAML 6.0.1 are cp310-cp310-manylinux_2_17_x86_64, cp310-cp310-manylinux2014_x86_64 which do not match the supported tags of /usr/local/bin/python3.10:
  cp310-cp310-manylinux_2_36_aarch64
  ... 517 more ...
Using the docker_environment solved those issues since you're directly building on what seemed like the correct platform. Looking into complete platforms now, maybe that will do the trick
Still a bit confused by complete platforms, but I tried with the regular platform strings:
Copy code
platforms=[
        "linux_2_17_x86_64-3.10.10-cp310"
        "macosx_14_0_arm64-cp-3.10.10-cp310"
    ]
I get the following error:
Copy code
stderr:
No pre-built wheel was available for PyYAML 6.0.1.
Successfully built the wheel PyYAML-6.0.1-cp39-cp39-macosx_11_0_arm64.whl from the sdist PyYAML-6.0.1.tar.gz but it is not compatible with the requested foreign target abbreviated platform cp310-cp310-linux_2_17_x86_64_3_10_10_cp310macosx_14_0_arm64.
You'll need to build a wheel from PyYAML-6.0.1.tar.gz on the foreign target platform and make it available to Pex via a `--find-links` repo or a custom `--index`.
Seems like it pants builds the wheel for PyYAML automatically, but does it for MacOS but not for linux. Given that I am on a Mac machine that makes sense, but I'm confused how to get around this. I used the docker_environment to solve this issue, but is there any other way?
b
```2.) The wheel tags for PyYAML 6.0.1 are cp310-cp310-manylinux_2_17_x86_64, cp310-cp310-manylinux2014_x86_64 which do not match the supported tags of /usr/local/bin/python3.10:
cp310-cp310-manylinux_2_36_aarch64```
It looks like the dependency is packaged for x86-64 architecture, but is running on aarch64 (aka arm) architecture. How are you running it?
Re the build failure: Pants will attempt to build a dependency if there's no appropriate pre-built wheel, but for dependencies that have platform-specific code this can result in something that won't work and is thus rejected (for pure-python dependencies, this process always works). It looks like PyYAML generally has a lot of pre-build wheels so it's a bit unexpected that it's attempting to build: https://pypi.org/project/PyYAML/6.0.1/#files I think it may be because the platform string isn't correct. I'm not sure of the right one, but what you have doesn't look right. In any case, 
platforms
is a dead-end that's soon to be deprecated in Pants, because they're a very lossy estimate of the target platform: complete platforms is the way to go, as that has far more accurate info. https://github.com/pantsbuild/pants/discussions/18756 has some info about how we do it.
2 Views
Open in Slack
PreviousNext
Powered by