https://pantsbuild.org/ logo
Join Slack
Powered by
Hi all I'm facing an issue running the `Celery[sqs...
# general
w
Hi all I'm facing an issue running the 
Celery[sqs]
application using the Pants PEX file. I've added the 
celery[sqs]
requirement in the 
pyproject.toml
file and imported it into the 
BUILD
file as well. However, I'm encountering an import error: 
ImportError: The curl client requires the pycurl library.
I've also installed the 
libcurl4-openssl-dev
package on my Docker container. Can someone please guide me on how to run the Celery application using the Pants system? I've attached the error message and sample config files.
Copy code
# Error message
[2023-12-11 08:56:19,379: CRITICAL/MainProcess] Unrecoverable error: ImportError('The curl client requires the pycurl library.')
Traceback (most recent call last):
  File "/root/.pex/installed_wheels/345e4b287b068c44a6ee3b190418c9003480dc8245c2cd1f80ed3e7971e22942/kombu-5.3.4-py3-none-any.whl/kombu/asynchronous/http/__init__.py", line 25, in get_client
    return hub._current_http_client
AttributeError: 'Hub' object has no attribute '_current_http_client'

During handling of the above exception, another exception occurred:
Copy code
# pyproject.toml
[tool.poetry.group.pkg-celery_worker.dependencies]
celery = {extras = ["redis", "sqs"], version = "^5.3.6"}
pycurl = "^7.45.2"
Copy code
# BUILD file
python_sources(
    name="pkg",
    sources=["celery/**/*.py"],
    dependencies=[
        "3rdparty/python:poetry#boto3",
        "3rdparty/python:poetry#celery",
        "3rdparty/python:poetry#redis",
        "3rdparty/python:poetry#pycurl",
    ],
)
Copy code
# Dockerfile
FROM nvidia/cuda:11.7.1-runtime-ubuntu20.04

RUN apt-get update \
    && apt-get install --no-install-recommends -y \
    libcurl4-openssl-dev \
    python3.9 \
    python3-pip

WORKDIR /app
COPY app /app/test_app

RUN mkdir /home/var

CMD ["/bin/bash", "-c", "/app/test_app"]
h
Looks like you're not the only one, and this is not Pants-specific: https://github.com/celery/celery/issues/4654
There are some remedies to try there, including how to install pycurl
This might be particularly relevant (just guessing): https://github.com/celery/celery/issues/4654#issuecomment-435607252
To isolate whether this is a Pants-specific issue or not, try getting this to work outside of Pants first
w
Thank you so much! I will try this method
I have a question: If I use a Docker base image with Celery installed, will Pants PEX use the Celery and PycURL packages from the base Docker image?
h
Currently this is not supported by Pants. It may be in the future.
w
Okay
Hi, You're right; I tried installing the pycurl package from a Docker container without using pants initially. However, it threw some errors. After that, I added a few packages (
libcurl4-openssl-dev and libssl-dev
) to the Docker image, which allowed me to install pycurl successfully using shell commands. Unfortunately, pants is still encountering the same error. Is there a way to verify whether the package is installed in the pex file or not?
Hi I've created a simple reproducible code for this issue. I can able to install 
pycurl
on my machine. However 
pycurl
package is not added to PEX file. Can someone please help me to resolve this issue ? Commands:
Copy code
# to build pex file
pants package python/pkg/celery_worker

# to run pex file
python3 dist/python.pkg.celery_worker/celery_worker_app.pex
Error message
Copy code
raise ImportError('The curl client requires the pycurl library.')
ImportError: The curl client requires the pycurl library.
celery_example.zip
đź‘€ 1
c
Great with a repro. However we are missing the details about what command you run, what expected result is and what the actual result was. When I package the pex from your zip file, I do have 
pycurl
listed as a requirement in there…
Copy code
❯ jq . dist/python.pkg.celery_worker/celery_worker_app.pex/PEX-INFO
{
  "bootstrap_hash": "5be3b46f7d6634cba8393be7fbfd89c2df7b49f8",
  "build_properties": {
    "pex_version": "2.1.134"
  },
  "code_hash": "dc51a109e94a4fa40da171ac08c3d722fc285339",
  "distributions": {
    "amqp-2.6.1-py2.py3-none-any.whl": "f585d3b5381f5cdb2ce2ef0771afaa2ada88fcc89af402fc1eb5385364a5a4b4",
    "async_timeout-4.0.3-py3-none-any.whl": "a48ada7fca4f5376ea7cbba037a1368e365c38ca767cfa9b0ed6079b7d96699c",
    "billiard-3.6.4.0-py3-none-any.whl": "add2a9a3ddcf229de775f8f11500f5f5c97485cd7b543f2299affda34feba65f",
    "boto3-1.33.13-py3-none-any.whl": "202b05e91cdeb7954328ac77cf71260022947d77391de19e497ab63f3568e55c",
    "botocore-1.33.13-py3-none-any.whl": "dc9a26fb4e6ef37d18e88802165319aca2260f1590bc01c3511288e93e4b8b48",
    "celery-4.3.0-py2.py3-none-any.whl": "dbdc8ef8de1ae0b8a7d36fad29c7dcc04e1f4a61c377f7efb1ecaac40ef5f80d",
    "jmespath-1.0.1-py3-none-any.whl": "299c3a18595a39d4d54252e98a3c39343899b9bd1997e3eebfee63ea5757588b",
    "kombu-4.6.11-py2.py3-none-any.whl": "ff70c39d71fde327ab1efd009a7eee7c5743bc633fd6339b7f12c238c21ca5b8",
    "pycurl-7.19.3-cp310-cp310-macosx_13_0_arm64.whl": "69ccb3ee4c777228fd67aafbcb055188031cc0082d4c0165538061ef4699526b",
    "pycurl-7.19.3-cp39-cp39-macosx_12_0_arm64.whl": "dabbf9619c9fd347834330224cf93f5ad99cbc676ed5ec17c84fb1df065581de",
    "python_dateutil-2.8.2-py2.py3-none-any.whl": "202c27a293331dd8fa9d41d1fcdd5ba4f4d6d2de0a2f00fa8547adc7c1aac629",
    "pytz-2023.3.post1-py2.py3-none-any.whl": "d1961115ba2d0eab86256042387b168cd0c4e9ab9aa56fcbdd07ac22250a1a5f",
    "redis-5.0.1-py3-none-any.whl": "ba6d70502476928a977796fb1687efa7de89d901b778d0c000ab84d36b702b4e",
    "s3transfer-0.8.2-py3-none-any.whl": "19e453799be60b8598d50f447fb4f58186e195149fd571dfe7fcfb2776b12172",
    "six-1.16.0-py2.py3-none-any.whl": "3e1c439c88d2e7681372427bab751b3fc99969891e95a714fed9604bf7710213",
    "urllib3-1.26.18-py2.py3-none-any.whl": "ed00d5e1a65928de2c3f13a743310b00480a9fbbcb98d410596d10af364ab51c",
    "vine-1.3.0-py2.py3-none-any.whl": "ae8613cbb3bae2015a75ccf3f24aeb8d8f838d9e8bebb4a3179a86596951c6d6"
  },
  "emit_warnings": false,
  "entry_point": "celery_worker.sqs_celery_app_factory",
  "ignore_errors": false,
  "includes_tools": false,
  "inherit_path": "false",
  "inject_args": [],
  "inject_env": {},
  "interpreter_constraints": [
    "CPython<3.11,>=3.9"
  ],
  "pex_hash": "d855e18e13f255e96ab464e992a8ba89234ebd87",
  "pex_path": "",
  "pex_paths": [],
  "requirements": [
    "boto3<2.0.0,>=1.29.2",
    "celery[redis,sqs]==4.3.0",
    "pycurl==7.19.3"
  ],
  "strip_pex_env": true,
  "venv": false,
  "venv_bin_path": "false",
  "venv_copies": false,
  "venv_hermetic_scripts": true,
  "venv_site_packages_copies": false
}
While doing that, I noticed there’s a number of warnings, and esp the one about 
kombu.Queue
being unknown… perhaps this is just a side-effect of a reduced example from a larger project…?
Also observing a number of 
__init__.py
files outside of any package is a bit strange..
w
Yea. this is a reduced example of larger project. Please ignore __init.py files
I will share the expected and actual output
Expected Output
Copy code
whether broker connection retries are made during startup in Celery 6.0 and above.
If you wish to retain the existing behavior for retrying connections on startup,
you should set broker_connection_retry_on_startup to True.
  warnings.warn(
[2023-12-14 00:06:31,951: INFO/MainProcess] Found credentials in shared credentials file: ~/.aws/credentials
[2023-12-14 00:06:32,300: INFO/MainProcess] Connected to <sqs://localhost//>
[2023-12-14 00:06:32,300: WARNING/MainProcess] /Users/dinesh/Library/Caches/pypoetry/virtualenvs/asris-v2-v9dx1PXw-py3.9/lib/python3.9/site-packages/celery/worker/consumer/consumer.py:507: CPendingDeprecationWarning: The broker_connection_retry configuration setting will no longer determine
whether broker connection retries are made during startup in Celery 6.0 and above.
If you wish to retain the existing behavior for retrying connections on startup,
you should set broker_connection_retry_on_startup to True.
  warnings.warn(
[2023-12-14 00:06:32,328: INFO/MainProcess] Found credentials in shared credentials file: ~/.aws/credentials
[2023-12-14 00:06:32,516: INFO/MainProcess] celery@M1V34RHXHD ready.
Actual Output
Copy code
[2023-12-14 00:09:00,908: INFO/MainProcess] Found credentials in shared credentials file: ~/.aws/credentials
[2023-12-14 00:09:01,230: INFO/MainProcess] Connected to <sqs://localhost//>
[2023-12-14 00:09:01,241: INFO/MainProcess] Found credentials in shared credentials file: ~/.aws/credentials
[2023-12-14 00:09:01,444: INFO/MainProcess] celery@M1V34RHXHD ready.
[2023-12-14 00:09:01,500: CRITICAL/MainProcess] Unrecoverable error: ImportError('The curl client requires the pycurl library.')
Traceback (most recent call last):
  File "/Users/dinesh/.pex/installed_wheels/ff70c39d71fde327ab1efd009a7eee7c5743bc633fd6339b7f12c238c21ca5b8/kombu-4.6.11-py2.py3-none-any.whl/kombu/asynchronous/http/__init__.py", line 20, in get_client
    return hub._current_http_client
AttributeError: 'Hub' object has no attribute '_current_http_client'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/dinesh/.pex/installed_wheels/dbdc8ef8de1ae0b8a7d36fad29c7dcc04e1f4a61c377f7efb1ecaac40ef5f80d/celery-4.3.0-py2.py3-none-any.whl/celery/worker/worker.py", line 205, in start
    self.blueprint.start(self)
  File "/Users/dinesh/.pex/installed_wheels/dbdc8ef8de1ae0b8a7d36fad29c7dcc04e1f4a61c377f7efb1ecaac40ef5f80d/celery-4.3.0-py2.py3-none-any.whl/celery/bootsteps.py", line 119, in start
    step.start(parent)
  File "/Users/dinesh/.pex/installed_wheels/dbdc8ef8de1ae0b8a7d36fad29c7dcc04e1f4a61c377f7efb1ecaac40ef5f80d/celery-4.3.0-py2.py3-none-any.whl/celery/bootsteps.py", line 369, in start
    return self.obj.start()
  File "/Users/dinesh/.pex/installed_wheels/dbdc8ef8de1ae0b8a7d36fad29c7dcc04e1f4a61c377f7efb1ecaac40ef5f80d/celery-4.3.0-py2.py3-none-any.whl/celery/worker/consumer/consumer.py", line 318, in start
    blueprint.start(self)
  File "/Users/dinesh/.pex/installed_wheels/dbdc8ef8de1ae0b8a7d36fad29c7dcc04e1f4a61c377f7efb1ecaac40ef5f80d/celery-4.3.0-py2.py3-none-any.whl/celery/bootsteps.py", line 119, in start
    step.start(parent)
  File "/Users/dinesh/.pex/installed_wheels/dbdc8ef8de1ae0b8a7d36fad29c7dcc04e1f4a61c377f7efb1ecaac40ef5f80d/celery-4.3.0-py2.py3-none-any.whl/celery/worker/consumer/consumer.py", line 596, in start
    c.loop(*c.loop_args())
  File "/Users/dinesh/.pex/installed_wheels/dbdc8ef8de1ae0b8a7d36fad29c7dcc04e1f4a61c377f7efb1ecaac40ef5f80d/celery-4.3.0-py2.py3-none-any.whl/celery/worker/loops.py", line 91, in asynloop
    next(loop)
  File "/Users/dinesh/.pex/installed_wheels/ff70c39d71fde327ab1efd009a7eee7c5743bc633fd6339b7f12c238c21ca5b8/kombu-4.6.11-py2.py3-none-any.whl/kombu/asynchronous/hub.py", line 301, in create_loop
    item()
  File "/Users/dinesh/.pex/installed_wheels/ae8613cbb3bae2015a75ccf3f24aeb8d8f838d9e8bebb4a3179a86596951c6d6/vine-1.3.0-py2.py3-none-any.whl/vine/promises.py", line 170, in __call__
    return self.throw()
  File "/Users/dinesh/.pex/installed_wheels/ae8613cbb3bae2015a75ccf3f24aeb8d8f838d9e8bebb4a3179a86596951c6d6/vine-1.3.0-py2.py3-none-any.whl/vine/promises.py", line 167, in __call__
    retval = fun(*final_args, **final_kwargs)
  File "/Users/dinesh/.pex/installed_wheels/ff70c39d71fde327ab1efd009a7eee7c5743bc633fd6339b7f12c238c21ca5b8/kombu-4.6.11-py2.py3-none-any.whl/kombu/transport/SQS.py", line 390, in _schedule_queue
    self._get_bulk_async(
  File "/Users/dinesh/.pex/installed_wheels/ff70c39d71fde327ab1efd009a7eee7c5743bc633fd6339b7f12c238c21ca5b8/kombu-4.6.11-py2.py3-none-any.whl/kombu/transport/SQS.py", line 407, in _get_bulk_async
    return self._get_async(queue, maxcount, callback=callback)
  File "/Users/dinesh/.pex/installed_wheels/ff70c39d71fde327ab1efd009a7eee7c5743bc633fd6339b7f12c238c21ca5b8/kombu-4.6.11-py2.py3-none-any.whl/kombu/transport/SQS.py", line 417, in _get_async
    qname, count=count, connection=self.asynsqs(queue=qname),
  File "/Users/dinesh/.pex/installed_wheels/ff70c39d71fde327ab1efd009a7eee7c5743bc633fd6339b7f12c238c21ca5b8/kombu-4.6.11-py2.py3-none-any.whl/kombu/transport/SQS.py", line 569, in asynsqs
    c = self._asynsqs = AsyncSQSConnection(
  File "/Users/dinesh/.pex/installed_wheels/ff70c39d71fde327ab1efd009a7eee7c5743bc633fd6339b7f12c238c21ca5b8/kombu-4.6.11-py2.py3-none-any.whl/kombu/asynchronous/aws/sqs/connection.py", line 23, in __init__
    AsyncAWSQueryConnection.__init__(
  File "/Users/dinesh/.pex/installed_wheels/ff70c39d71fde327ab1efd009a7eee7c5743bc633fd6339b7f12c238c21ca5b8/kombu-4.6.11-py2.py3-none-any.whl/kombu/asynchronous/aws/connection.py", line 197, in __init__
    AsyncConnection.__init__(self, sqs_connection, http_client,
  File "/Users/dinesh/.pex/installed_wheels/ff70c39d71fde327ab1efd009a7eee7c5743bc633fd6339b7f12c238c21ca5b8/kombu-4.6.11-py2.py3-none-any.whl/kombu/asynchronous/aws/connection.py", line 151, in __init__
    self._httpclient = http_client or get_client()
  File "/Users/dinesh/.pex/installed_wheels/ff70c39d71fde327ab1efd009a7eee7c5743bc633fd6339b7f12c238c21ca5b8/kombu-4.6.11-py2.py3-none-any.whl/kombu/asynchronous/http/__init__.py", line 22, in get_client
    client = hub._current_http_client = Client(hub, **kwargs)
  File "/Users/dinesh/.pex/installed_wheels/ff70c39d71fde327ab1efd009a7eee7c5743bc633fd6339b7f12c238c21ca5b8/kombu-4.6.11-py2.py3-none-any.whl/kombu/asynchronous/http/__init__.py", line 13, in Client
    return CurlClient(hub, **kwargs)
  File "/Users/dinesh/.pex/installed_wheels/ff70c39d71fde327ab1efd009a7eee7c5743bc633fd6339b7f12c238c21ca5b8/kombu-4.6.11-py2.py3-none-any.whl/kombu/asynchronous/http/curl.py", line 43, in __init__
    raise ImportError('The curl client requires the pycurl library.')
ImportError: The curl client requires the pycurl library.
If you have any AWS SQS queue, please use that queue name.
c
what command did you run?
w
Copy code
# to build pex file
pants package python/pkg/celery_worker

# to run pex file
python3 dist/python.pkg.celery_worker/celery_worker_app.pex
these two commands
c
Ah, so for me it doesn’t get to the failure scenario due to not having the expected services running it wants to connect to. Do you get the same error if you comment out the 
layout="packed"
on the 
pex_binary
target and run it as 
./dist/python.pkg.celery_worker/celery_worker_app.pex
? (you may have to remove the directory before packaging as it doesn’t seem to clean up that first)
w
Sure, Let me try this
I'm still getting the same error
If you are still unable to reproduce the error, could we schedule a call to discuss it further?
I will run the example code and explain the issue
c
sorry, I’m not available real-time, also this is not my area of expertise. We’re getting into the particulars of Python packaging and pycurl in particular. I tried 
import curl
from a venv created from your pex file, and it failed with:
Copy code
ImportError: pycurl: libcurl link-time ssl backend (none/other) is different from compile-time ssl backend (openssl)
If this works for you outside of pants and fails when using pants, we need to figure out what is different between the two cases.. but if this fails also outside of pants, I can’t help you further. Hope you find enough crumbs here to guide you…
w
Sure, Thank you so much!
16 Views
Open in Slack
PreviousNext
Powered by