Hello, I'm having an issue with using the

googleapiclient

Python package when running

./pants test ...

. It seems to be an issue with the underlying

pyparsing

module. I was wondering if anyone has seen anything like this before. The error I'm getting is as follows.

AttributeError: module 'pyparsing' has no attribute 'downcaseTokens'

Here's the part of the stack trace that seems relevant.

front_porch/modules/common/google_drive/google_drive.py:11: in <module>
    from googleapiclient.discovery import build
/Users/hughhan/.cache/pants/named_caches/pex_root/venvs/8dccf2108884fe3898a8bc04279cb74a330a1bbd/e83365d7bd577d2fac6d4ec70eb77f4d4607d634/lib/python3.9/site-packages/googleapiclient/discovery.py:48: in <module>
    import httplib2
/Users/hughhan/.cache/pants/named_caches/pex_root/venvs/8dccf2108884fe3898a8bc04279cb74a330a1bbd/e83365d7bd577d2fac6d4ec70eb77f4d4607d634/lib/python3.9/site-packages/httplib2/__init__.py:52: in <module>
    from . import auth
/Users/hughhan/.cache/pants/named_caches/pex_root/venvs/8dccf2108884fe3898a8bc04279cb74a330a1bbd/e83365d7bd577d2fac6d4ec70eb77f4d4607d634/lib/python3.9/site-packages/httplib2/auth.py:20: in <module>
    auth_param_name = token.copy().setName("auth-param-name").addParseAction(pp.downcaseTokens)
E   AttributeError: module 'pyparsing' has no attribute 'downcaseTokens'

I have not seen this one. I wonder if

googleapiclient

is not correctly setting its metadata for the version of

pyparsing

it needs What pants version are you on? Are you using lock files?

➜  ./pants --version
2.10.0

I am using a

Pipfile.lock

, which is generated from a

Pipfile

that I define. Here is what my root-level

BUILD

file looks like.

python_sources(name = "root")

pipenv_requirements(
    name = "requirements",
    module_mapping = {
        "analytics-python": ["analytics"],
        "factory-boy": ["factory"],
        "google-api-python-client": ["google"],
        "graphql-core": ["graphql"],
        "iso-639": ["iso639"],
        "launchdarkly-server-sdk": ["ldclient"],
        "plaid-python": ["plaid"],
        "pytest-xdist": ["xdist"],
        "python-dateutil": ["dateutil"],
    },
    overrides = {
        "google-cloud-pubsub": { "dependencies": [":requirements#setuptools"] },
        "iso-639": { "dependencies": [":requirements#setuptools"] },
    },
    source = "Pipfile.lock",
    type_stubs_module_mapping = {
        "types-Flask": ["flask"],
        "types-PyMySQL": ["pymysql"],
        "types-python-dateutil": ["python-dateutil"],
        "types-pytz": ["pytz"],
        "types-PyYAML": ["pyyaml"],
        "types-requests": ["requests"],
        "types-simplejson": ["simplejson"],
        "types-Werkzeug": ["werkzeug"],
    },
)

Cool, thanks.

I am using a Pipfile.lock

So that won't be consued by Pants. See https://www.pantsbuild.org/docs/python-third-party-dependencies#lockfiles for how you can use a lockfile with Pants. It might shed some light on this problem by telling us which version of pyparsing is being used -- Even if you do not use a lock file for now, I suspect that the fix might be the same. Find the version of pyparser that is being used in your poetry lock file, as it sounds like that one is working. Then, add this to your BUILD file:

python_requirement(
    name="pyparsing",
    requirements=["pyparsing==<insert version>"]
)

Finally add to your

overrrides

that

google-api-python-client

depends on

:pyparsing

, similar to what you do w/ setuptools

So that won't be consumed by Pants.

I followed the reference at https://www.pantsbuild.org/docs/reference-pipenv_requirements. Are

Pipfile.lock

files not consumed by Pants? It seems that the corresponding dependencies are properly installed when running other tests that do not need

pyparsing

?

Are Pipfile.lock files not consumed by Pants?

They are not. You must set up a lock file as instructed at https://www.pantsbuild.org/docs/python-third-party-dependencies#lockfiles if you want the benefits of lockfiles like reproducibility

It seems that the corresponding dependencies are properly installed when running other tests that do not need pyparsing?

Based on the stack trace, it looks like pyparsing is a transitive dependency of google api client, right? do you ever directly use

pyparsing

?

Based on the stack trace, it looks like pyparsing is a transitive dependency of google api client, right? do you ever directly use

pyparsing

?

Yup, that's correct

Is the version of pyparsing different between pants's lock file and your poetry.lock?

Just to clarify, I am using Pipfile.lock (not poetry.lock). The version of

pyparsing

is the same as far as I can tell.

➜  pipenv graph
...
        - pyparsing [required: >=2.0.2,!=3.0.5, installed: 2.4.7]
...

Ah thanks for clarifying! Hmmm bummer, so then the version of pyparsing is not the problem 🤔

Is there anything else that you suspect it could be?

I'm trying to think of other hypotheses. The first one that comes to my mind is if a different code path is somehow being encountered due to the absence of transitive deps google normally has access to. As explained in the opening paragraphs of https://www.pantsbuild.org/docs/python-third-party-dependencies, Pants only runs with the subset of deps a certain task needs you can try to set this to

true

to test that hypothesis: https://www.pantsbuild.org/docs/reference-python#section-run-against-entire-lockfile

I added the following to

pants.toml

, and then reran

./pants test ...

, and am still seeing the same error.

[python]
...
run_against_entire_lockfile = true

have you seen this? https://stackoverflow.com/questions/69780351/attributeerror-module-pyparsing-has-no-attribute-downcasetokens-rasa

Will try the proposed solutions in this post and let you know how it goes... Thanks for sending this over.

I am not certain. It seems like we have disproved my two hypothesis 1) version of pyparsing is different 2) using the entire lockfile vs a subseet makes a difference Could you compare other dependencies in the lock file? For example, it sounds like

httplib2

is relevant. It would be helpful to see where the differences are

From `3rdparty/python/default.lock`:

httplib2==0.19.1 \
    --hash=sha256:2ad195faf9faf079723f6714926e9a9061f694d07724b846658ce08d40f522b4 \
    --hash=sha256:0b12617eeca7433d4c396a100eaecfa4b08ee99aa881e6df6e257a7aad5d533d

From `Pipfile.lock`:

"httplib2": {
    "hashes": [
        "sha256:0b12617eeca7433d4c396a100eaecfa4b08ee99aa881e6df6e257a7aad5d533d",
        "sha256:2ad195faf9faf079723f6714926e9a9061f694d07724b846658ce08d40f522b4"
    ],
    "version": "==0.19.1"
},

Order of hashes does not make a difference. Were any other relevant dependencies different?

It's hard for me to tell... the

3rdparty/python/default.lock

and

Pipfile.lock

files are quite large...

found this https://github.com/GoogleCloudPlatform/cloud-profiler-python/issues/104

Hmm, I already have

2.4.7

in my

Pipfile.lock

...

"pyparsing": {
    "hashes": [
        "sha256:c203ec8783bf771a155b207279b9bccb8dea02d8f0c9e5f8ead507bc3246ecc1",
        "sha256:ef9d7589ef3c200abe66653d3f1ab1033c3c419ae9b9bdb1240a85b024efc88b"
    ],
    "markers": "python_version >= '2.6' and python_version not in '3.0, 3.1, 3.2, 3.3'",
    "version": "==2.4.7"
},

you have it in pipfile.lock, but remindre pants ignores that

Correct me if I'm wrong, but the answer in this Github issue seems to be implying the solution that didn't work previously, right?

python_sources(name = "root")

python_requirement(
    name = "pyparsing",
    requirements = ["pyparsing==2.4.7"]
)

pipenv_requirements(
    name = "requirements",
    overrides = {
        "google-api-python-client": { "dependencies": [":pyparsing#pyparsing"] },
        ...
    },
    source = "Pipfile.lock",
    ...
)

That might be worth trying. Is httplib2 directly in your pipfile though as a direct dependency?

httplib2

is not in my Pipfile as a direct dependency.

It uses pex which uses pip, but the newer resolver

Huh…