https://pantsbuild.org/ logo
Join Slack
Powered by
Running into a weird issue with a python requireme...
# general
e
Running into a weird issue with a python requirement with an extra: 
celery[sqs]
Copy code
Engine traceback:
  in select
  in pants.core.goals.test.run_tests
  in pants.backend.python.goals.pytest_runner.run_python_test (src/util/tests/test_s3.py)
  in pants.backend.python.goals.pytest_runner.setup_pytest_for_target
  in pants.backend.python.util_rules.pex.create_pex
  in pants.backend.python.util_rules.pex.build_pex (requirements.pex)
  in pants.engine.process.fallible_to_exec_result_or_raise
Traceback (most recent call last):
  File "/home/mpcusack/.cache/pants/setup/bootstrap-Linux-x86_64/2.9.0+git57344e8f_py39/lib/python3.9/site-packages/pants/engine/process.py", line 282, in fallible_to_exec_result_or_raise
    raise ProcessExecutionFailure(
pants.engine.process.ProcessExecutionFailure: Process 'Building requirements.pex with 15 requirements: boto3, botocore, celery[sqs], decorator, django, honeycomb-beeline, memoize@, moto[ec2,s3,sqs,ssm]==2.3.1, pytest, pytz, redis, requests, text-unidecode' failed with exit code 1.
stdout:

stderr:
WARNING: Discarding <https://files.pythonhosted.org/packages/ef/05/4b773f74f830a90a326b06f9b24e65506302ab049e825a3c0b60b1a6e26a/pycurl-7.43.0.5.tar.gz#sha256=ec7dd291545842295b7b56c12c90ffad2976cc7070c98d7b1517b7b6cd5994b3> (from <https://pypi.org/simple/pycurl/>). Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
ERROR: Could not find a version that satisfies the requirement pycurl==7.43.0.5; extra == "sqs" (from celery[sqs])
ERROR: No matching distribution found for pycurl==7.43.0.5; extra == "sqs"
pid 2170396 -> /home/mpcusack/.cache/pants/named_caches/pex_root/venvs/e9a17d974385339dffaabf39686f310560143805/3d1364f53acc96cacfc8b68eb497c10e7797cb4f/pex --disable-pip-version-check --no-python-version-warning --exists-action a --isolated -q --cache-dir /home/mpcusack/.cache/pants/named_caches/pex_root --log /tmp/process-executionkxBOiT/.tmp/tmp96az5yn4/pip.log download --dest /tmp/process-executionkxBOiT/.tmp/tmp7cg_pj88/home.mpcusack..pyenv.versions.3.8.12.bin.python3.8 --constraint constraints.lock boto3 botocore celery[sqs] decorator django honeycomb-beeline moto[ec2,s3,sqs,ssm]==2.3.1 pytest pytz redis requests text-unidecode --index-url <https://pypi.org/simple/> --find-links <https://binaries.pantsbuild.org/wheels/pantsbuild.pants/57344e8f8315b47b47d27f9ef36e61d42caabb3f/2.9.0%2Bgit57344e8f/index.html> --retries 5 --timeout 15 exited with 1 and STDERR:
None
ERROR: Could not find a version that satisfies the requirement pycurl==7.43.0.5; extra == "sqs" (from celery[sqs]) ERROR: No matching distribution found for pycurl==7.43.0.5; extra == "sqs"
h
Well it clearly exists
h
Hey Michael, are you able to share the 
[python]
(formerly 
[python-setup]
) section of your 
pants.toml
? I'm curious how you're using things like constraints files
h
When I've run into this, it's because of python version requirements or something similar.
e
Copy code
mpcusack@mpcp15:~/dev/color$ grep celery build/python-packages/external-requirements.txt 
celery[sqs]
mpcusack@mpcp15:~/dev/color$ grep pycurl build/python-packages/external-requirements.txt mpcusack@mpcp15:~/dev/color$ grep celery constraints.lock 
celery==4.4.7
mpcusack@mpcp15:~/dev/color$ grep pycurl constraints.lock 
pycurl==7.43.0.5
What's weird is the the 
sqs
extra is on the 
celery
package, but its somehow propagating down to the 
pycurl
dependency
Copy code
[python]
interpreter_constraints = [
    "CPython==3.8.12",
]
requirement_constraints = "constraints.lock"
resolve_all_constraints = false
h
it looks like when you say 
celery[sqs]
, that adds a dep on `kombu[sqs]`: https://github.com/celery/celery/blob/master/requirements/extras/sqs.txt (also see the 
setup.py
file in that repo to see how I figured that out) Then 
kombu[sqs]
explains the pycurl dep: https://github.com/celery/kombu/blob/master/requirements/extras/sqs.txt But yeah, your'e right that I don't get why 
pycurl
has an 
sqs
extra...
it could be useful to try 
[pex].verbosity
, such as setting it to 3: https://www.pantsbuild.org/docs/reference-pex#section-verbosity Another debugging tip is to try installing this with PEX directly, remove Pants from the equation. You can use 
-ldebug
to get the original command Pants is running, then run it directly w/ PEX
e
Ahh yes i expect the pycurl dep. I'm locked to 4.4.7 for celery to its even more explicit: https://github.com/celery/celery/blob/v4.4.7/requirements/extras/sqs.txt
👍 1
Hmm, trying again with 
$ PANTS_PEX_VERBOSITY=9 ./pants -ldebug test src/util/tests/test_s3.py
Shows me this error before the warning about discarding:
Copy code
Traceback (most recent call last):
      File "/tmp/process-executionda9GA5/.tmp/pip-download-qox5siqt/pycurl_0d3c56d3a0fe4d10b96b0750a85be177/setup.py", line 233, in configure_unix
        p = subprocess.Popen((self.curl_config(), '--version'),
      File "/home/mpcusack/.pyenv/versions/3.8.12/lib/python3.8/subprocess.py", line 858, in __init__
        self._execute_child(args, executable, preexec_fn, close_fds,
      File "/home/mpcusack/.pyenv/versions/3.8.12/lib/python3.8/subprocess.py", line 1704, in _execute_child
        raise child_exception_type(errno_num, err_msg, err_filename)
    FileNotFoundError: [Errno 2] No such file or directory: 'curl-config'

    During handling of the above exception, another exception occurred:

    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/process-executionda9GA5/.tmp/pip-download-qox5siqt/pycurl_0d3c56d3a0fe4d10b96b0750a85be177/setup.py", line 961, in <module>
        ext = get_extension(sys.argv, split_extension_source=split_extension_source)
      File "/tmp/process-executionda9GA5/.tmp/pip-download-qox5siqt/pycurl_0d3c56d3a0fe4d10b96b0750a85be177/setup.py", line 623, in get_extension
        ext_config = ExtensionConfiguration(argv)
      File "/tmp/process-executionda9GA5/.tmp/pip-download-qox5siqt/pycurl_0d3c56d3a0fe4d10b96b0750a85be177/setup.py", line 101, in __init__
        self.configure()
      File "/tmp/process-executionda9GA5/.tmp/pip-download-qox5siqt/pycurl_0d3c56d3a0fe4d10b96b0750a85be177/setup.py", line 238, in configure_unix
        raise ConfigurationError(msg)
    __main__.ConfigurationError: Could not run curl-config: [Errno 2] No such file or directory: 'curl-config'
👀 1
So error message is a little confusing, but the 
[sqs]
is not being applied to pycurl
h
e
And this is fixed by running 
sudo apt install libcurl4-openssl-dev
confirmed
🚀 1
Thanks for the PANTS_PEX_VERBOSITY tip!
❤️ 1
Open in Slack
PreviousNext
Powered by