Hello! I’ve been trying to get Celery running on d...
# general
b
Hello! I’ve been trying to get Celery running on docker in my repo, but no success, wanted to see if anyone could help me. I have this:
Copy code
docker_environment(name="python_slim", platform="linux_x86_64", image="python:3.11-slim") (this already works for my main backend)

python_sources(
    name="backend",
)

pex_binary(
    name="main",
    entry_point="main.py",
    environment="python_slim",
)

pex_binary(
    name="celery_app",
    # entry_point="celery_tasks.py", (already tried with and without this)
    environment="python_slim",
)
# Also tried this but no success
#python_source(
#     name="tasks",
#     source="celery_tasks.py",
#     dependencies=[
#         "3rdparty/python:requirements#celery",
#     ],
# )

# pex_binary(
#     name="celery_app",
#     script="celery",
#     dependencies=[
#         ":tasks",
#         ":backend",
#     ],
#     environment="python_slim",
#     execution_mode="venv",
#     layout="packed"
# )
My dockerfile:
Copy code
FROM python:3.11-slim

WORKDIR /app

COPY src.python.backend/celery_app.pex /bin/app
ENTRYPOINT [ "/bin/app" ]
CMD celery -A tasks worker --loglevel=info
And my celery_tasks.py
Copy code
# celery_tasks.py
from celery import Celery

app = Celery("tasks", broker="<amqp://admin:pass@rabbit//>")

@app.task  
def add(x, y):  
    return x + y
What am I missing?
h
Hmm, quite a few moving parts here. Could you create a toy github repo that reproduces the issue, with a README that explains exactly what to run, what you'd expect to happen, and what actually happens?
It's quite hard to pin down problems when we're not certain that we're looking at the same thing
E.g. pants.toml, requirements.txt...
b
Sure thing, will recreate one tomorrow and share here
🙏 2
Hello! Here’s the git repo https://github.com/williamlac/celery_pants_example/tree/main I tried keeping all the configs I have in case of it being a conflict problem. But I removed all the logic and kept just a fastapi main and celery_tasks files. My fastapi main is working fine but can’t get celery to open
🙏 1
👀 1
Worth noting, we have Tilt in our project mainly to be able to hot reload during local development. But I tried running celery both using it, straight in docker and a bunch of other combinations
Hey ! Have you managed to check my example repo? Still haven’t managed to get this working 😢
h
Hi, sorry, haven't had a chance to look, but it's on my todo list for early next week
b
Hey, finally managed to figure most of it out. Celery is now running, it’s complaining about relative imports but still haven’t looked into it. But so far I guess problem is solved!
🎉 1
h
I'll hold off diving in to this until you report back on that relative imports issue
What was the Celery issue?
b
Took a while to figure out how to properly set the commands to make it run, the whole env is a bit complex because of the docker_environment + pex + tilt for hot reloading