https://pantsbuild.org/ logo
#general
Title
# general
c

cold-soccer-63228

05/02/2022, 4:02 PM
Hello, I'm having an issue with using the
googleapiclient
Python package when running
./pants test ...
. It seems to be an issue with the underlying
pyparsing
module. I was wondering if anyone has seen anything like this before. The error I'm getting is as follows.
Copy code
AttributeError: module 'pyparsing' has no attribute 'downcaseTokens'
Here's the part of the stack trace that seems relevant.
Copy code
front_porch/modules/common/google_drive/google_drive.py:11: in <module>
    from googleapiclient.discovery import build
/Users/hughhan/.cache/pants/named_caches/pex_root/venvs/8dccf2108884fe3898a8bc04279cb74a330a1bbd/e83365d7bd577d2fac6d4ec70eb77f4d4607d634/lib/python3.9/site-packages/googleapiclient/discovery.py:48: in <module>
    import httplib2
/Users/hughhan/.cache/pants/named_caches/pex_root/venvs/8dccf2108884fe3898a8bc04279cb74a330a1bbd/e83365d7bd577d2fac6d4ec70eb77f4d4607d634/lib/python3.9/site-packages/httplib2/__init__.py:52: in <module>
    from . import auth
/Users/hughhan/.cache/pants/named_caches/pex_root/venvs/8dccf2108884fe3898a8bc04279cb74a330a1bbd/e83365d7bd577d2fac6d4ec70eb77f4d4607d634/lib/python3.9/site-packages/httplib2/auth.py:20: in <module>
    auth_param_name = token.copy().setName("auth-param-name").addParseAction(pp.downcaseTokens)
E   AttributeError: module 'pyparsing' has no attribute 'downcaseTokens'
h

hundreds-father-404

05/02/2022, 4:04 PM
I have not seen this one. I wonder if
googleapiclient
is not correctly setting its metadata for the version of
pyparsing
it needs What pants version are you on? Are you using lock files?
c

cold-soccer-63228

05/02/2022, 4:06 PM
Copy code
➜  ./pants --version
2.10.0
I am using a
Pipfile.lock
, which is generated from a
Pipfile
that I define. Here is what my root-level
BUILD
file looks like.
Copy code
python_sources(name = "root")

pipenv_requirements(
    name = "requirements",
    module_mapping = {
        "analytics-python": ["analytics"],
        "factory-boy": ["factory"],
        "google-api-python-client": ["google"],
        "graphql-core": ["graphql"],
        "iso-639": ["iso639"],
        "launchdarkly-server-sdk": ["ldclient"],
        "plaid-python": ["plaid"],
        "pytest-xdist": ["xdist"],
        "python-dateutil": ["dateutil"],
    },
    overrides = {
        "google-cloud-pubsub": { "dependencies": [":requirements#setuptools"] },
        "iso-639": { "dependencies": [":requirements#setuptools"] },
    },
    source = "Pipfile.lock",
    type_stubs_module_mapping = {
        "types-Flask": ["flask"],
        "types-PyMySQL": ["pymysql"],
        "types-python-dateutil": ["python-dateutil"],
        "types-pytz": ["pytz"],
        "types-PyYAML": ["pyyaml"],
        "types-requests": ["requests"],
        "types-simplejson": ["simplejson"],
        "types-Werkzeug": ["werkzeug"],
    },
)
Is there anything I can do with respect to the
overrides
, specifically to tell
googleapiclient
that it needs a specific version of
pyparsing
? I'm not sure how automatic module mapping works, but alternatively would I have to add
googleapiclient
to the values list for
google-api-python-client
? For example:
Copy code
"google-api-python-client": ["google", "googleapiclient"]
I wouldn't think that I need to do the latter because normally I would have expected a
ModuleNotFoundError
in the cases that I need to explicitly define the module mapping.
h

hundreds-father-404

05/02/2022, 4:18 PM
Cool, thanks.
I am using a Pipfile.lock
So that won't be consued by Pants. See https://www.pantsbuild.org/docs/python-third-party-dependencies#lockfiles for how you can use a lockfile with Pants. It might shed some light on this problem by telling us which version of pyparsing is being used -- Even if you do not use a lock file for now, I suspect that the fix might be the same. Find the version of pyparser that is being used in your poetry lock file, as it sounds like that one is working. Then, add this to your BUILD file:
Copy code
python_requirement(
    name="pyparsing",
    requirements=["pyparsing==<insert version>"]
)
Finally add to your
overrrides
that
google-api-python-client
depends on
:pyparsing
, similar to what you do w/ setuptools
I wouldn't think that I need to do the latter because normally I would have expected a ModuleNotFoundError in the cases that I need to explicitly define the module mapping.
I agree
c

cold-soccer-63228

05/02/2022, 4:26 PM
So that won't be consumed by Pants.
I followed the reference at https://www.pantsbuild.org/docs/reference-pipenv_requirements. Are
Pipfile.lock
files not consumed by Pants? It seems that the corresponding dependencies are properly installed when running other tests that do not need
pyparsing
?
h

hundreds-father-404

05/02/2022, 4:29 PM
Are Pipfile.lock files not consumed by Pants?
They are not. You must set up a lock file as instructed at https://www.pantsbuild.org/docs/python-third-party-dependencies#lockfiles if you want the benefits of lockfiles like reproducibility
It seems that the corresponding dependencies are properly installed when running other tests that do not need pyparsing?
Based on the stack trace, it looks like pyparsing is a transitive dependency of google api client, right? do you ever directly use
pyparsing
?
c

cold-soccer-63228

05/02/2022, 4:36 PM
Based on the stack trace, it looks like pyparsing is a transitive dependency of google api client, right? do you ever directly use
pyparsing
?
Yup, that's correct
I went ahead and did what I interpreted as what you advised: 1. run
./pants generate-lockfiles
, 2. add a
pyparsing
requirement to the root-level
BUILD
file, 3. add
pyparsing
as a dependency to the overrides of
google-api-python-client
. Afterwards, I re-ran
./pants test ...
, and still am getting the same error. Here is what my
BUILD
looks like now.
Copy code
python_sources(name = "root")

python_requirement(
    name = "pyparsing",
    requirements = ["pyparsing==2.4.7"]
)

pipenv_requirements(
    name = "requirements",
    module_mapping = {
        "analytics-python": ["analytics"],
        "factory-boy": ["factory"],
        "google-api-python-client": ["google"],
        "graphql-core": ["graphql"],
        "iso-639": ["iso639"],
        "launchdarkly-server-sdk": ["ldclient"],
        "plaid-python": ["plaid"],
        "pytest-xdist": ["xdist"],
        "python-dateutil": ["dateutil"],
    },
    overrides = {
        "google-api-python-client": { "dependencies": [":pyparsing#pyparsing"] },
        "google-cloud-pubsub": { "dependencies": [":requirements#setuptools"] },
        "iso-639": { "dependencies": [":requirements#setuptools"] },
    },
    source = "Pipfile.lock",
    type_stubs_module_mapping = {
        "types-Flask": ["flask"],
        "types-PyMySQL": ["pymysql"],
        "types-python-dateutil": ["python-dateutil"],
        "types-pytz": ["pytz"],
        "types-PyYAML": ["pyyaml"],
        "types-requests": ["requests"],
        "types-simplejson": ["simplejson"],
        "types-Werkzeug": ["werkzeug"],
    },
)
Am I misinterpreting something?
h

hundreds-father-404

05/02/2022, 5:07 PM
Is the version of pyparsing different between pants's lock file and your poetry.lock?
c

cold-soccer-63228

05/02/2022, 5:08 PM
Just to clarify, I am using Pipfile.lock (not poetry.lock). The version of
pyparsing
is the same as far as I can tell.
Copy code
➜  pipenv graph
...
        - pyparsing [required: >=2.0.2,!=3.0.5, installed: 2.4.7]
...
h

hundreds-father-404

05/02/2022, 5:10 PM
Ah thanks for clarifying! Hmmm bummer, so then the version of pyparsing is not the problem 🤔
😢 1
c

cold-soccer-63228

05/02/2022, 5:11 PM
Is there anything else that you suspect it could be?
h

hundreds-father-404

05/02/2022, 5:13 PM
I'm trying to think of other hypotheses. The first one that comes to my mind is if a different code path is somehow being encountered due to the absence of transitive deps google normally has access to. As explained in the opening paragraphs of https://www.pantsbuild.org/docs/python-third-party-dependencies, Pants only runs with the subset of deps a certain task needs you can try to set this to
true
to test that hypothesis: https://www.pantsbuild.org/docs/reference-python#section-run-against-entire-lockfile
c

cold-soccer-63228

05/02/2022, 5:16 PM
I added the following to
pants.toml
, and then reran
./pants test ...
, and am still seeing the same error.
Copy code
[python]
...
run_against_entire_lockfile = true
👍 1
c

cold-soccer-63228

05/02/2022, 5:39 PM
Will try the proposed solutions in this post and let you know how it goes... Thanks for sending this over.
🤞 1
👍 1
Unfortunately, I can't upgrade
httplib2
to
0.2+
because of another dependency. And downgrading
pyparsing
to
2.4.2
didn't fix it for me 😞
Just to understand this a bit better, what is the reason that running
./pants test
causes this issue to be encountered, whereas running something like
pipenv run pytest
directly doesn't encounter this error?
h

hundreds-father-404

05/03/2022, 2:56 PM
I am not certain. It seems like we have disproved my two hypothesis 1) version of pyparsing is different 2) using the entire lockfile vs a subseet makes a difference Could you compare other dependencies in the lock file? For example, it sounds like
httplib2
is relevant. It would be helpful to see where the differences are
c

cold-soccer-63228

05/03/2022, 4:29 PM
From `3rdparty/python/default.lock`:
Copy code
httplib2==0.19.1 \
    --hash=sha256:2ad195faf9faf079723f6714926e9a9061f694d07724b846658ce08d40f522b4 \
    --hash=sha256:0b12617eeca7433d4c396a100eaecfa4b08ee99aa881e6df6e257a7aad5d533d
From `Pipfile.lock`:
Copy code
"httplib2": {
    "hashes": [
        "sha256:0b12617eeca7433d4c396a100eaecfa4b08ee99aa881e6df6e257a7aad5d533d",
        "sha256:2ad195faf9faf079723f6714926e9a9061f694d07724b846658ce08d40f522b4"
    ],
    "version": "==0.19.1"
},
It seems that the versions are hashes are the same—the only difference is the order that the hashes are listed between the two.
h

hundreds-father-404

05/03/2022, 4:30 PM
Order of hashes does not make a difference. Were any other relevant dependencies different?
c

cold-soccer-63228

05/03/2022, 4:31 PM
It's hard for me to tell... the
3rdparty/python/default.lock
and
Pipfile.lock
files are quite large...
1
h

hundreds-father-404

05/03/2022, 4:33 PM
c

cold-soccer-63228

05/03/2022, 4:34 PM
Hmm, I already have
2.4.7
in my
Pipfile.lock
...
Copy code
"pyparsing": {
    "hashes": [
        "sha256:c203ec8783bf771a155b207279b9bccb8dea02d8f0c9e5f8ead507bc3246ecc1",
        "sha256:ef9d7589ef3c200abe66653d3f1ab1033c3c419ae9b9bdb1240a85b024efc88b"
    ],
    "markers": "python_version >= '2.6' and python_version not in '3.0, 3.1, 3.2, 3.3'",
    "version": "==2.4.7"
},
Going to try regenerating the
default.lock
file just to sanity check...
h

hundreds-father-404

05/03/2022, 4:35 PM
you have it in pipfile.lock, but remindre pants ignores that
c

cold-soccer-63228

05/03/2022, 4:40 PM
Correct me if I'm wrong, but the answer in this Github issue seems to be implying the solution that didn't work previously, right?
Copy code
python_sources(name = "root")

python_requirement(
    name = "pyparsing",
    requirements = ["pyparsing==2.4.7"]
)

pipenv_requirements(
    name = "requirements",
    overrides = {
        "google-api-python-client": { "dependencies": [":pyparsing#pyparsing"] },
        ...
    },
    source = "Pipfile.lock",
    ...
)
Perhaps should it be the following instead (substituting
httplib2
for where we had
google-api-python-client
previously)?
Copy code
python_sources(name = "root")

python_requirement(
    name = "pyparsing",
    requirements = ["pyparsing==2.4.7"]
)

pipenv_requirements(
    name = "requirements",
    overrides = {
        "httplib2": { "dependencies": [":pyparsing#pyparsing"] },
        ...
    },
    source = "Pipfile.lock",
    ...
)
h

hundreds-father-404

05/03/2022, 4:41 PM
That might be worth trying. Is httplib2 directly in your pipfile though as a direct dependency?
c

cold-soccer-63228

05/03/2022, 4:44 PM
httplib2
is not in my Pipfile as a direct dependency.
I see this: https://github.com/httplib2/httplib2/issues/207#issuecomment-951282767
Old pip dependency resolver allowed installation of conflicting requirements. Installing httplib2 (with requirements pyparsing<3) would not downgrade existing pyparsing>=3.
pypa/pip#6969
pypa/pip#988
> pip 20.3 has been released, and it has the new resolver by default!
So upgrade pip might also help.
pip install 'pip>=20.3'
Does Pants use Pip to resolve dependenecies?
h

hundreds-father-404

05/03/2022, 4:51 PM
It uses pex which uses pip, but the newer resolver
c

cold-soccer-63228

05/03/2022, 4:51 PM
Huh…
So it's not an issue with the resolver then either
It's surprising that it works okay when installing via pipenv but not via Pants