pantsbuild #general

Channels

welcome

github-notifications

announce

dry-analyst-73584

01/27/2022, 7:29 PM

It looks like pants is correctly inferring all our deps except one (pymongo), for reasons I do not understand. my requirements.txt has

pymongo==3.7.1

and it's being imported as

import pymongo

in a bunch of places, but for some reason when I run tests I'm getting

Copy code

ImportError while importing test module '/private/var/folders/yv/1kp6r8h57p91lxsszfdppwsw0000gn/T/process-executionGbSusG/test/python/foursquare_test/mongo/lib/handles_test.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
test/python/foursquare_test/mongo/lib/handles_test.py:10: in <module>
    from foursquare.mongo import mongo_ops
src/python/foursquare/mongo/mongo_ops.py:13: in <module>
    import pymongo
E   ImportError: No module named pymongo

It seems to be specific to that

mongo_ops.py

file. Everything that depends on mongo_ops.py fails with the import error. But every other file that has

import pymongo

is working fine. I can fix it by editing the BUILD file:

Copy code

python_sources(
    dependencies = [
        "3rdparty/python:pymongo",
    ],
)

But it seems extremely odd that I would need to.

high-yak-85899

01/27/2022, 9:31 PM

Curious on opinions about organizing dependencies like files for testdata. Let's say I have a bunch of

*.py

and

*_test.py

modules under

src/app

that I'm working with.

tailor

will conveniently make me a BUILD file with

python_sources

and

python_tests

declared that expands everything for me. Let's say I have one test in there,

some_test.py

that needs files stored in

src/app/testdata

. My suspicion is that if I were to add something like

Copy code

files(name="testdata", sources=["testdata/**"])

to the BUILD file and change

python_tests()

python_tests(dependencies=[":testdata"])

, any change to what is in

testdata

would invalidate a bunch of caching and cause all the tests to get rerun even though I only needed it in

some_test.py

eager-dress-66405

01/27/2022, 9:46 PM

Running into a weird issue with a python requirement with an extra:

celery[sqs]

eager-dress-66405

01/27/2022, 9:48 PM

Copy code

Engine traceback:
  in select
  in pants.core.goals.test.run_tests
  in pants.backend.python.goals.pytest_runner.run_python_test (src/util/tests/test_s3.py)
  in pants.backend.python.goals.pytest_runner.setup_pytest_for_target
  in pants.backend.python.util_rules.pex.create_pex
  in pants.backend.python.util_rules.pex.build_pex (requirements.pex)
  in pants.engine.process.fallible_to_exec_result_or_raise
Traceback (most recent call last):
  File "/home/mpcusack/.cache/pants/setup/bootstrap-Linux-x86_64/2.9.0+git57344e8f_py39/lib/python3.9/site-packages/pants/engine/process.py", line 282, in fallible_to_exec_result_or_raise
    raise ProcessExecutionFailure(
pants.engine.process.ProcessExecutionFailure: Process 'Building requirements.pex with 15 requirements: boto3, botocore, celery[sqs], decorator, django, honeycomb-beeline, memoize@, moto[ec2,s3,sqs,ssm]==2.3.1, pytest, pytz, redis, requests, text-unidecode' failed with exit code 1.
stdout:

stderr:
WARNING: Discarding <https://files.pythonhosted.org/packages/ef/05/4b773f74f830a90a326b06f9b24e65506302ab049e825a3c0b60b1a6e26a/pycurl-7.43.0.5.tar.gz#sha256=ec7dd291545842295b7b56c12c90ffad2976cc7070c98d7b1517b7b6cd5994b3> (from <https://pypi.org/simple/pycurl/>). Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
ERROR: Could not find a version that satisfies the requirement pycurl==7.43.0.5; extra == "sqs" (from celery[sqs])
ERROR: No matching distribution found for pycurl==7.43.0.5; extra == "sqs"
pid 2170396 -> /home/mpcusack/.cache/pants/named_caches/pex_root/venvs/e9a17d974385339dffaabf39686f310560143805/3d1364f53acc96cacfc8b68eb497c10e7797cb4f/pex --disable-pip-version-check --no-python-version-warning --exists-action a --isolated -q --cache-dir /home/mpcusack/.cache/pants/named_caches/pex_root --log /tmp/process-executionkxBOiT/.tmp/tmp96az5yn4/pip.log download --dest /tmp/process-executionkxBOiT/.tmp/tmp7cg_pj88/home.mpcusack..pyenv.versions.3.8.12.bin.python3.8 --constraint constraints.lock boto3 botocore celery[sqs] decorator django honeycomb-beeline moto[ec2,s3,sqs,ssm]==2.3.1 pytest pytz redis requests text-unidecode --index-url <https://pypi.org/simple/> --find-links <https://binaries.pantsbuild.org/wheels/pantsbuild.pants/57344e8f8315b47b47d27f9ef36e61d42caabb3f/2.9.0%2Bgit57344e8f/index.html> --retries 5 --timeout 15 exited with 1 and STDERR:
None

eager-dress-66405

01/27/2022, 9:48 PM

ERROR: Could not find a version that satisfies the requirement pycurl==7.43.0.5; extra == "sqs" (from celery[sqs]) ERROR: No matching distribution found for pycurl==7.43.0.5; extra == "sqs"

high-yak-85899

01/27/2022, 9:49 PM

Well it clearly exists

hundreds-father-404

01/27/2022, 9:49 PM

Hey Michael, are you able to share the

[python]

(formerly

[python-setup]

) section of your

pants.toml

? I'm curious how you're using things like constraints files

high-yak-85899

01/27/2022, 9:49 PM

When I've run into this, it's because of python version requirements or something similar.

eager-dress-66405

01/27/2022, 9:50 PM

Copy code

mpcusack@mpcp15:~/dev/color$ grep celery build/python-packages/external-requirements.txt 
celery[sqs]
mpcusack@mpcp15:~/dev/color$ grep pycurl build/python-packages/external-requirements.txt mpcusack@mpcp15:~/dev/color$ grep celery constraints.lock 
celery==4.4.7
mpcusack@mpcp15:~/dev/color$ grep pycurl constraints.lock 
pycurl==7.43.0.5

eager-dress-66405

01/27/2022, 9:52 PM

What's weird is the the

sqs

extra is on the

celery

package, but its somehow propagating down to the

pycurl

dependency

eager-dress-66405

01/27/2022, 9:52 PM

Copy code

[python]
interpreter_constraints = [
    "CPython==3.8.12",
]
requirement_constraints = "constraints.lock"
resolve_all_constraints = false

hundreds-father-404

01/27/2022, 9:59 PM

it looks like when you say

celery[sqs]

, that adds a dep on `kombu[sqs]`: https://github.com/celery/celery/blob/master/requirements/extras/sqs.txt (also see the

setup.py

file in that repo to see how I figured that out) Then

kombu[sqs]

explains the pycurl dep: https://github.com/celery/kombu/blob/master/requirements/extras/sqs.txt But yeah, your'e right that I don't get why

pycurl

has an

sqs

extra...

hundreds-father-404

01/27/2022, 10:00 PM

it could be useful to try

[pex].verbosity

, such as setting it to 3: https://www.pantsbuild.org/docs/reference-pex#section-verbosity Another debugging tip is to try installing this with PEX directly, remove Pants from the equation. You can use

-ldebug

to get the original command Pants is running, then run it directly w/ PEX

eager-dress-66405

01/27/2022, 10:01 PM

Ahh yes i expect the pycurl dep. I'm locked to 4.4.7 for celery to its even more explicit: https://github.com/celery/celery/blob/v4.4.7/requirements/extras/sqs.txt

👍 1

eager-dress-66405

01/27/2022, 10:06 PM

Hmm, trying again with

$ PANTS_PEX_VERBOSITY=9 ./pants -ldebug test src/util/tests/test_s3.py

eager-dress-66405

01/27/2022, 10:07 PM

Shows me this error before the warning about discarding:

Copy code

Traceback (most recent call last):
      File "/tmp/process-executionda9GA5/.tmp/pip-download-qox5siqt/pycurl_0d3c56d3a0fe4d10b96b0750a85be177/setup.py", line 233, in configure_unix
        p = subprocess.Popen((self.curl_config(), '--version'),
      File "/home/mpcusack/.pyenv/versions/3.8.12/lib/python3.8/subprocess.py", line 858, in __init__
        self._execute_child(args, executable, preexec_fn, close_fds,
      File "/home/mpcusack/.pyenv/versions/3.8.12/lib/python3.8/subprocess.py", line 1704, in _execute_child
        raise child_exception_type(errno_num, err_msg, err_filename)
    FileNotFoundError: [Errno 2] No such file or directory: 'curl-config'

    During handling of the above exception, another exception occurred:

    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/process-executionda9GA5/.tmp/pip-download-qox5siqt/pycurl_0d3c56d3a0fe4d10b96b0750a85be177/setup.py", line 961, in <module>
        ext = get_extension(sys.argv, split_extension_source=split_extension_source)
      File "/tmp/process-executionda9GA5/.tmp/pip-download-qox5siqt/pycurl_0d3c56d3a0fe4d10b96b0750a85be177/setup.py", line 623, in get_extension
        ext_config = ExtensionConfiguration(argv)
      File "/tmp/process-executionda9GA5/.tmp/pip-download-qox5siqt/pycurl_0d3c56d3a0fe4d10b96b0750a85be177/setup.py", line 101, in __init__
        self.configure()
      File "/tmp/process-executionda9GA5/.tmp/pip-download-qox5siqt/pycurl_0d3c56d3a0fe4d10b96b0750a85be177/setup.py", line 238, in configure_unix
        raise ConfigurationError(msg)
    __main__.ConfigurationError: Could not run curl-config: [Errno 2] No such file or directory: 'curl-config'

👀 1

eager-dress-66405

01/27/2022, 10:07 PM

So error message is a little confusing, but the

[sqs]

is not being applied to pycurl

hundreds-father-404

01/27/2022, 10:08 PM

Okay great, that's progress. Which a quick duckduckgo search takes me to: https://stackoverflow.com/questions/23937933/could-not-run-curl-config-errno-2-no-such-file-or-directory-when-installing

eager-dress-66405

01/27/2022, 10:08 PM

And this is fixed by running

sudo apt install libcurl4-openssl-dev

eager-dress-66405

01/27/2022, 10:09 PM

confirmed

🚀 1

eager-dress-66405

01/27/2022, 10:09 PM

Thanks for the PANTS_PEX_VERBOSITY tip!

❤️ 1

mysterious-action-44812

01/27/2022, 10:25 PM

hi I’m hitting the

Exception message: Could not establish identity of /System/Library/Frameworks/Python.framework/Versions/2.7/Resources/pythonwrapper.app/Contents/MacOS/pythonwrapper

that I believe was mentioned elsewhere (see e.g https://github.com/pantsbuild/pants/issues/13451). I am on pants 1.27.0 (yes I know it’s ancient we cannot upgrade to pants 2 as it doesn’t support our languages yet). this happens specifically with

./pants export

which the intellij plugin needs in order to work.

./pants compile

and friends on cmdline work fine. any way we can get the fix mentioned in those PRs, or some other workaround, to resolve this? cc @enough-analyst-54434

dry-analyst-73584

01/27/2022, 11:20 PM

When I run a pants command and then run it a second time with no changes, sometimes it hangs about for ~10 seconds apparently rebuilding its graph, and sometimes it finishes instantly. I'm curious what makes the difference?

hundreds-father-404

01/27/2022, 11:23 PM

Do you see a message saying "Initializing scheduler..."?

dry-analyst-73584

01/27/2022, 11:24 PM

yes

dry-analyst-73584

01/27/2022, 11:25 PM

But then it does "find all targets in project" which is the bit that seems to take a while

hundreds-father-404

01/27/2022, 11:25 PM

and that's in-between the two runs? If so, that means that Pantsd (daemon) restarted so you lost all memoization You can look at

.pants.d/pants.log

to see why

dry-analyst-73584

01/27/2022, 11:25 PM

👍

dry-analyst-73584

01/27/2022, 11:26 PM

Yeah, I just ran it six times in a row, and it initialized scheduler four times

dry-analyst-73584

01/27/2022, 11:27 PM

Copy code

18:25:22.86 [INFO] handling request: `--pants-bin-name=./v2pants --pants-version=2.9.0 test test/python/foursquare_test/fs::`
18:25:40.86 [INFO] request completed: `--pants-bin-name=./v2pants --pants-version=2.9.0 test test/python/foursquare_test/fs::`
18:25:41.18 [INFO] notify invalidation: cleared 0 and dirtied 0 nodes for: {".git", ".git/index.lock"}
18:25:41.19 [INFO] notify invalidation: cleared 1 and dirtied 45 nodes for: {".git", ".git/index.lock"}
18:25:42.08 [INFO] notify invalidation: cleared 0 and dirtied 0 nodes for: {".git/index.lock", ".git"}
18:25:42.08 [INFO] notify invalidation: cleared 0 and dirtied 0 nodes for: {".git", ".git/index.lock"}
18:25:42.09 [INFO] notify invalidation: cleared 0 and dirtied 0 nodes for: {".git/index.lock", ".git"}
18:25:42.10 [INFO] notify invalidation: cleared 0 and dirtied 0 nodes for: {".git/index.lock", ".git"}
18:26:43.63 [INFO] Extending leases
18:26:45.37 [INFO] Done extending leases