https://pantsbuild.org/ logo
#general
Title
# general
a

agreeable-oyster-28981

05/20/2022, 7:46 PM
Hi everyone, I’m struggling to make my
flask
pex
works with
gunicorn
, having tried with
entry_points
and
PEX_SCRIPT
./pants run xxx
works perfectly fine Once I try to have it running with
gunicorn
(from local or from a docker image, which is the ultimate goal), I can’t seem to find the right syntax for the module/app
Copy code
COPY packages.hello_world/main.pex packages/hello_world/main.pex

CMD ./packages/hello_world/main.pex packages.hello_world.main:app
Will give me the following error:
Copy code
/pants run packages/hello_world    
/Users/quentin/.cache/pants/setup/bootstrap-Darwin-arm64/2.11.0_py39/bin/pants
15:34:54.78 [INFO] Canceled: Building docker image hello_world:latest
15:34:56.01 [INFO] Completed: Building docker image hello_world:latest
[2022-05-20 19:34:57 +0000] [8] [INFO] Starting gunicorn 20.1.0
[2022-05-20 19:34:57 +0000] [8] [INFO] Listening at: <http://127.0.0.1:8000> (8)
[2022-05-20 19:34:57 +0000] [8] [INFO] Using worker: sync
[2022-05-20 19:34:57 +0000] [11] [INFO] Booting worker with pid: 11
[2022-05-20 19:34:57 +0000] [11] [ERROR] Exception in worker process
Traceback (most recent call last):
  File "/root/.pex/venvs/s/4323ba33/venv/lib/python3.9/site-packages/gunicorn/arbiter.py", line 589, in spawn_worker
    worker.init_process()
  File "/root/.pex/venvs/s/4323ba33/venv/lib/python3.9/site-packages/gunicorn/workers/base.py", line 134, in init_process
    self.load_wsgi()
  File "/root/.pex/venvs/s/4323ba33/venv/lib/python3.9/site-packages/gunicorn/workers/base.py", line 146, in load_wsgi
    self.wsgi = self.app.wsgi()
  File "/root/.pex/venvs/s/4323ba33/venv/lib/python3.9/site-packages/gunicorn/app/base.py", line 67, in wsgi
    self.callable = self.load()
  File "/root/.pex/venvs/s/4323ba33/venv/lib/python3.9/site-packages/gunicorn/app/wsgiapp.py", line 58, in load
    return self.load_wsgiapp()
  File "/root/.pex/venvs/s/4323ba33/venv/lib/python3.9/site-packages/gunicorn/app/wsgiapp.py", line 48, in load_wsgiapp
    return util.import_app(self.app_uri)
  File "/root/.pex/venvs/s/4323ba33/venv/lib/python3.9/site-packages/gunicorn/util.py", line 359, in import_app
    mod = importlib.import_module(module)
  File "/usr/local/lib/python3.9/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 984, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'packages.hello_world.main'
[2022-05-20 19:34:57 +0000] [11] [INFO] Worker exiting (pid: 11)
[2022-05-20 19:34:57 +0000] [8] [INFO] Shutting down: Master
[2022-05-20 19:34:57 +0000] [8] [INFO] Reason: Worker failed to boot.
I’ve been trying multiple syntax, and I either get:
Failed to find attribute 'app' in 'packages.hello_world'.
or
ModuleNotFoundError: No module named 'packages.hello_world.main'
which tells me that • the package
packages.hello_world
exists • the package `packages.hellow_world.main`doesn’t exist • there is no
app
attribute in the package • there is no
main
attribute in the package folder structure is:
Copy code
/packages/hello_word
|- Build
|- Dockerfile
|- main.py
You can find a repo to reproduce the issue here: https://github.com/qventura/pants-flask/tree/main/packages/hello_world Thanks in advance, if anyone can help me finding the right way to start my app, I’ll be grateful!
👀 1
I actually can build directly with
pex
, with:
Copy code
pex -vv -D . -o ./main.pex -r requirements.txt
PEX_SCRIPT=gunicorn ./main.pex main:app
I looks like I’m just struggling with the naming…
w

wide-midnight-78598

05/20/2022, 8:50 PM
Just reading this quickly, I'm having a hard time understanding which part isn't working as expected? Do you want "dist/packages.hello_world/main.pex" to run? Or is it specifically inside the docker container that is causing the problems?
Also, are you adding your
:src
sources as a dependency into your pex_binary?
a

agreeable-oyster-28981

05/20/2022, 9:01 PM
both, actually. if I do:
Copy code
./pants package packages/hello_world:main-non-docker
./dist/packages.hello_world/main-non-docker.pex main:app
or
Copy code
./pants run package packages/hello_world:docker
They both don’t run, with error
ModuleNotFoundError: No module named 'main'
w

wide-midnight-78598

05/20/2022, 9:05 PM
Well, for starters, when you unzip your pex - are the appropriate files there?
And under what package structure?
So, like, unrelated to gunicorn - I have this running the flask dev server
Copy code
pex_binary(
  name="main",
  execution_mode="venv",
  entry_point="hello_world.main",
  dependencies=[":src"]
a

agreeable-oyster-28981

05/20/2022, 9:08 PM
Also, are you adding your
:src
sources as a dependency into your pex_binary?
Hum, no. So, when I use
entry_point="main.py",
the file is included automatically, but if my entry point is
gunicorn
I need to add the files?
I’ll give it a try
w

wide-midnight-78598

05/20/2022, 9:09 PM
Well, I know there is some level of dependency inference and all that, but personally, I like seeing explicit deps, to avoid any potential confusion. Also, make sure you have appropriate
__init__.py
files where needed
You can use
./pants dependencies
or
./pants peek
to see what will be included, but personally, I'm always explicit over implicit
If you're still struggling when I come back in a few hours, I can whip up a working example from one of my flask projects
h

happy-kitchen-89482

05/20/2022, 9:40 PM
Ignoring the Docker aspect for now, are you able to
./pants package packages/hello_world:main
and run the resulting pex?
w

wide-midnight-78598

05/21/2022, 12:44 AM
Okay, sorry for the delay here - was just out of commission. How's about this?
Copy code
# BUILD
python_sources(
    name="src"
)

pex_binary(
    name="main",
    script="gunicorn",
    dependencies=[":src", "//:root#gunicorn"]
)
And then
./pants package ::
And then from the correct dir:
Copy code
> ./main.pex hello_world.main:app

[2022-05-20 20:43:01 -0400] [78898] [INFO] Starting gunicorn 20.1.0
[2022-05-20 20:43:01 -0400] [78898] [INFO] Listening at: <http://127.0.0.1:8000> (78898)
[2022-05-20 20:43:01 -0400] [78898] [INFO] Using worker: sync
[2022-05-20 20:43:01 -0400] [78899] [INFO] Booting worker with pid: 78899
There may be other, better ways to do it - with more flexibility. In the past when I needed to do something similar to this, I used the
PEX_TOOLS=1
exported to a venv and then called that from
nginx unit
@enough-analyst-54434 would know better though
If you unzip the
main.pex
and check out
PEX_INFO
that should give you a better idea of what's going on
h

happy-kitchen-89482

05/22/2022, 10:31 PM
Looking at the repo you posted (thanks for that, it's very helpful for debugging), I see that
packages/helloworld:main
has
gunicorn
as an entry point, and gunicorn expects an argument
So
./dist/packages.hello_world/main.pex hello_world.main
would be the right way to invoke that
(Your
main.pex
is really a pex that contains gunicorn and your main code, but the entry point is generic gunicorn doesn't know about the main)
When I run that I get an error
ModuleNotFoundError: No module named 'hello_world'
That is because
main.py
has no imports from
hello_world
so pants has nothing to infer a dependency from
So I add
dependencies=[":src"]
to the pex_binary target
Now I get a Gunicorn error:
Failed to find attribute 'application' in 'hello_world.main'.
Which I think is legit
renaming
app
to
application
in
main.py
solved that and the server works for me
And re the problem of having to specify the entry point to the pex: At Toolchain we solve that by having a wrapper file that runs gunicorn against the appropriate entry point, which allows the pex to run without arguments. This in turn allows us to have a single Docker layer stack for all our pexes, with just one final layer for each service that differs by just one file - the pex
You could also maybe do this with config files, since gunicorn now lets you specify
wsgi_app =
in gunicorn.conf.py
Anyway, @agreeable-oyster-28981 let us know if this sorts things out!
a

agreeable-oyster-28981

05/24/2022, 12:13 PM
It totally does
renaming
application
to
app
is not 100% necessary, as the attribute name can be passed directly when calling unicorn with
hello_world.main:app
Thanks both of you @wide-midnight-78598 and @happy-kitchen-89482 it was 100% the sources not being included in the pex file -_-
👍 1
7 Views