https://pantsbuild.org/ logo
Join Slack
Powered by
Running into a weird issue with Pants which worked...
# general
b
Running into a weird issue with Pants which worked before. It’s unable to build requirements.pex. Failing with a requirement parsing error. Details in 🧵
Pants version 
2.10.0
Copy code
16:38:03.43 [ERROR] 1 Exception encountered:

Engine traceback:
  in select
  in pants.core.goals.test.run_tests
  in pants.backend.python.goals.pytest_runner.run_python_test (src/python/func-olorailsmenugetter/tests/test_app.py:../test_py38)
  in pants.backend.python.goals.pytest_runner.setup_pytest_for_target
  in pants.backend.python.util_rules.pex.create_pex
  in pants.backend.python.util_rules.pex.build_pex (requirements.pex)
  in pants.engine.process.fallible_to_exec_result_or_raise
Traceback (most recent call last):
  File "/Users/dmistry/.cache/pants/setup/bootstrap-Darwin-x86_64/pants.vwr5Ve/install/lib/python3.9/site-packages/pants/engine/process.py", line 285, in fallible_to_exec_result_or_raise
    raise ProcessExecutionFailure(
pants.engine.process.ProcessExecutionFailure: Process 'Building requirements.pex with 3 requirements: boto3, gh-core, ghlambda' failed with exit code 1.
stdout:

stderr:
pex: Resolving interpreters
pex: Resolving interpreters: 0.1ms
pex: Building pex
pex: Building pex :: Resolving distributions (boto3 gh-core ghlambda)
pex: Building pex :: Resolving distributions (boto3 gh-core ghlambda) :: Resolving requirements.
pex: Building pex :: Resolving distributions (boto3 gh-core ghlambda) :: Resolving requirements. :: Resolving for:
  /Users/dmistry/.pyenv/versions/3.8.13/bin/python3.8
pex: Spawning a maximum of 3 parallel jobs to process:
  /Users/dmistry/.pyenv/versions/3.8.13/bin/python3.8
pex: Hashing pex
pex: Hashing pex: 86.7ms
pex: Isolating pex
pex: Isolating pex: 0.1ms
    Running command python setup.py egg_info
    running egg_info
    creating /private/var/folders/z1/8_j4n7t17h15skx5y6h69whh0000gp/T/process-executionYS0bXh/.tmp/pip-pip-egg-info-m5r1z055/jaeger_client.egg-info
    writing /private/var/folders/z1/8_j4n7t17h15skx5y6h69whh0000gp/T/process-executionYS0bXh/.tmp/pip-pip-egg-info-m5r1z055/jaeger_client.egg-info/PKG-INFO
    writing dependency_links to /private/var/folders/z1/8_j4n7t17h15skx5y6h69whh0000gp/T/process-executionYS0bXh/.tmp/pip-pip-egg-info-m5r1z055/jaeger_client.egg-info/dependency_links.txt
    writing requirements to /private/var/folders/z1/8_j4n7t17h15skx5y6h69whh0000gp/T/process-executionYS0bXh/.tmp/pip-pip-egg-info-m5r1z055/jaeger_client.egg-info/requires.txt
    writing top-level names to /private/var/folders/z1/8_j4n7t17h15skx5y6h69whh0000gp/T/process-executionYS0bXh/.tmp/pip-pip-egg-info-m5r1z055/jaeger_client.egg-info/top_level.txt
    writing manifest file '/private/var/folders/z1/8_j4n7t17h15skx5y6h69whh0000gp/T/process-executionYS0bXh/.tmp/pip-pip-egg-info-m5r1z055/jaeger_client.egg-info/SOURCES.txt'
    reading manifest file '/private/var/folders/z1/8_j4n7t17h15skx5y6h69whh0000gp/T/process-executionYS0bXh/.tmp/pip-pip-egg-info-m5r1z055/jaeger_client.egg-info/SOURCES.txt'
    reading manifest template '<http://MANIFEST.in|MANIFEST.in>'
    warning: no files found matching '*.thrift' under directory 'jaeger_client'
    warning: no files found matching '*.yaml'
    warning: no files found matching '*.json' under directory 'config'
    warning: no files found matching '*.yaml' under directory 'config'
    warning: no files found matching '*.rst' under directory 'docs'
    writing manifest file '/private/var/folders/z1/8_j4n7t17h15skx5y6h69whh0000gp/T/process-executionYS0bXh/.tmp/pip-pip-egg-info-m5r1z055/jaeger_client.egg-info/SOURCES.txt'
ERROR: Exception:
Traceback (most recent call last):
  File "/Users/dmistry/.cache/pants/named_caches/pex_root/venvs/s/66ef3043/venv/lib/python3.8/site-packages/pip/_vendor/pkg_resources/__init__.py", line 3021, in _dep_map
    return self.__dep_map
  File "/Users/dmistry/.cache/pants/named_caches/pex_root/venvs/s/66ef3043/venv/lib/python3.8/site-packages/pip/_vendor/pkg_resources/__init__.py", line 2815, in __getattr__
    raise AttributeError(attr)
AttributeError: _DistInfoDistribution__dep_map

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/dmistry/.cache/pants/named_caches/pex_root/venvs/s/66ef3043/venv/lib/python3.8/site-packages/pip/_vendor/packaging/requirements.py", line 113, in __init__
    req = REQUIREMENT.parseString(requirement_string)
  File "/Users/dmistry/.cache/pants/named_caches/pex_root/venvs/s/66ef3043/venv/lib/python3.8/site-packages/pip/_vendor/pyparsing.py", line 1955, in parseString
    raise exc
  File "/Users/dmistry/.cache/pants/named_caches/pex_root/venvs/s/66ef3043/venv/lib/python3.8/site-packages/pip/_vendor/pyparsing.py", line 3814, in parseImpl
    raise ParseException(instring, loc, self.errmsg, self)
pip._vendor.pyparsing.ParseException: Expected stringEnd, found ','  (at char 23), (line:1, col:24)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/dmistry/.cache/pants/named_caches/pex_root/venvs/s/66ef3043/venv/lib/python3.8/site-packages/pip/_vendor/pkg_resources/__init__.py", line 3101, in __init__
    super(Requirement, self).__init__(requirement_string)
  File "/Users/dmistry/.cache/pants/named_caches/pex_root/venvs/s/66ef3043/venv/lib/python3.8/site-packages/pip/_vendor/packaging/requirements.py", line 115, in __init__
    raise InvalidRequirement(
pip._vendor.packaging.requirements.InvalidRequirement: Parse error at "','": Expected stringEnd

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/dmistry/.cache/pants/named_caches/pex_root/venvs/s/66ef3043/venv/lib/python3.8/site-packages/pip/_internal/cli/base_command.py", line 223, in _main
    status = self.run(options, args)
  File "/Users/dmistry/.cache/pants/named_caches/pex_root/venvs/s/66ef3043/venv/lib/python3.8/site-packages/pip/_internal/cli/req_command.py", line 180, in wrapper
    return func(self, options, args)
  File "/Users/dmistry/.cache/pants/named_caches/pex_root/venvs/s/66ef3043/venv/lib/python3.8/site-packages/pip/_internal/commands/download.py", line 130, in run
    requirement_set = resolver.resolve(
  File "/Users/dmistry/.cache/pants/named_caches/pex_root/venvs/s/66ef3043/venv/lib/python3.8/site-packages/pip/_internal/resolution/resolvelib/resolver.py", line 121, in resolve
    self._result = resolver.resolve(
  File "/Users/dmistry/.cache/pants/named_caches/pex_root/venvs/s/66ef3043/venv/lib/python3.8/site-packages/pip/_vendor/resolvelib/resolvers.py", line 453, in resolve
    state = resolution.resolve(requirements, max_rounds=max_rounds)
  File "/Users/dmistry/.cache/pants/named_caches/pex_root/venvs/s/66ef3043/venv/lib/python3.8/site-packages/pip/_vendor/resolvelib/resolvers.py", line 347, in resolve
    failure_causes = self._attempt_to_pin_criterion(name, criterion)
  File "/Users/dmistry/.cache/pants/named_caches/pex_root/venvs/s/66ef3043/venv/lib/python3.8/site-packages/pip/_vendor/resolvelib/resolvers.py", line 207, in _attempt_to_pin_criterion
    criteria = self._get_criteria_to_update(candidate)
  File "/Users/dmistry/.cache/pants/named_caches/pex_root/venvs/s/66ef3043/venv/lib/python3.8/site-packages/pip/_vendor/resolvelib/resolvers.py", line 198, in _get_criteria_to_update
    for r in self._p.get_dependencies(candidate):
  File "/Users/dmistry/.cache/pants/named_caches/pex_root/venvs/s/66ef3043/venv/lib/python3.8/site-packages/pip/_internal/resolution/resolvelib/provider.py", line 170, in get_dependencies
    return [
  File "/Users/dmistry/.cache/pants/named_caches/pex_root/venvs/s/66ef3043/venv/lib/python3.8/site-packages/pip/_internal/resolution/resolvelib/provider.py", line 170, in <listcomp>
    return [
  File "/Users/dmistry/.cache/pants/named_caches/pex_root/venvs/s/66ef3043/venv/lib/python3.8/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 247, in iter_dependencies
    requires = self.dist.requires() if with_requires else ()
  File "/Users/dmistry/.cache/pants/named_caches/pex_root/venvs/s/66ef3043/venv/lib/python3.8/site-packages/pip/_vendor/pkg_resources/__init__.py", line 2736, in requires
    dm = self._dep_map
  File "/Users/dmistry/.cache/pants/named_caches/pex_root/venvs/s/66ef3043/venv/lib/python3.8/site-packages/pip/_vendor/pkg_resources/__init__.py", line 3023, in _dep_map
    self.__dep_map = self._compute_dependencies()
  File "/Users/dmistry/.cache/pants/named_caches/pex_root/venvs/s/66ef3043/venv/lib/python3.8/site-packages/pip/_vendor/pkg_resources/__init__.py", line 3033, in _compute_dependencies
    reqs.extend(parse_requirements(req))
  File "/Users/dmistry/.cache/pants/named_caches/pex_root/venvs/s/66ef3043/venv/lib/python3.8/site-packages/pip/_vendor/pkg_resources/__init__.py", line 3094, in parse_requirements
    yield Requirement(line)
  File "/Users/dmistry/.cache/pants/named_caches/pex_root/venvs/s/66ef3043/venv/lib/python3.8/site-packages/pip/_vendor/pkg_resources/__init__.py", line 3103, in __init__
    raise RequirementParseError(str(e))
pip._vendor.pkg_resources.RequirementParseError: Parse error at "','": Expected stringEnd
pid 47076 -> /Users/dmistry/.cache/pants/named_caches/pex_root/venvs/44f0d229c64d262df3c9196eaa181a922a2a17cd/5190f2bb1c910d0c3977d742bb12392b7f02f9f9/pex --disable-pip-version-check --no-python-version-warning --exists-action a --isolated -vvv --cache-dir /Users/dmistry/.cache/pants/named_caches/pex_root --log /private/var/folders/z1/8_j4n7t17h15skx5y6h69whh0000gp/T/process-executionYS0bXh/.tmp/tmp5un1_0h6/pip.log download --dest /private/var/folders/z1/8_j4n7t17h15skx5y6h69whh0000gp/T/process-executionYS0bXh/.tmp/tmp6l7js_56/Users.dmistry..pyenv.versions.3.8.13.bin.python3.8 boto3 gh-core ghlambda --index-url <https://pypi.org/simple/> --extra-index-url <https://artifactory.grubhub.com/artifactory/api/pypi/pypi-repos/simple> --retries 5 --timeout 15 exited with 2 and STDERR:
The project only has 3 dependencies. 2 first party and 1 3rd party (boto3)
with this requirements macro -
Copy code
python_requirements(
    name="reqs",
    overrides={
        "ghlambda": {"dependencies": [":reqs#boto3"]},
    },
    module_mapping={
      "gh-core": ["gh"]
    },
)
If I remove the undeclared dependency (boto3), everything works
e
Worked before ... in time, before a recent Pants upgrade, ... any more info on what may have changed?
Also, what is gh-core? That does not appear to be on PyPI.
b
that is our in-house lib
Both gh-core and ghlambda
e
Can you share its requirements? Basically can you share the public PyPI root set of reqs?
The more you can share the better of course. Hard to debug without knowing what is being resolved.
Or is boto3 it? Do the 1st party actually depend on 0 3rdparty except for boto3?
b
They do depend on other 3rdparty libs. Let me share the full list
e
Thank you. That at least allows a quick rule in / out of Pex vs Pants ti whittle things down.
Pants 2.10.0 is Pex 2.1.71 and 
$ pex --python python3.8 pex==2.1.71 -cpex -- boto3 -o example.pex
works for example.
b
Here is the dep tree
Copy code
ghlambda==0.5.6
  - datadog-lambda [required: Any, installed: 3.60.0]
    - datadog [required: >=0.41.0,<0.42.0, installed: 0.41.0]
      - decorator [required: >=3.3.2, installed: 5.1.1]
      - requests [required: >=2.6.0, installed: 2.28.1]
        - certifi [required: >=2017.4.17, installed: 2022.6.15]
        - charset-normalizer [required: >=2,<3, installed: 2.1.0]
        - idna [required: >=2.5,<4, installed: 3.3]
        - urllib3 [required: >=1.21.1,<1.27, installed: 1.26.10]
    - ddtrace [required: >=0.61.1,<0.62.0, installed: 0.61.1]
      - attrs [required: >=19.2.0, installed: 21.4.0]
      - packaging [required: >=17.1, installed: 21.3]
        - pyparsing [required: >=2.0.2,!=3.0.5, installed: 3.0.9]
      - protobuf [required: >=3,<4, installed: 3.20.1]
      - six [required: >=1.12.0, installed: 1.16.0]
      - tenacity [required: >=5, installed: 8.0.1]
    - wrapt [required: >=1.11.2,<2.0.0, installed: 1.14.1]
  - gh-core [required: >=0.1.22, installed: 0.2.23]
    - boto3 [required: <1.24.0, installed: 1.23.10]
      - botocore [required: >=1.26.10,<1.27.0, installed: 1.26.10]
        - jmespath [required: >=0.7.1,<2.0.0, installed: 1.0.1]
        - python-dateutil [required: >=2.1,<3.0.0, installed: 2.8.2]
          - six [required: >=1.5, installed: 1.16.0]
        - urllib3 [required: >=1.25.4,<1.27, installed: 1.26.10]
      - jmespath [required: >=0.7.1,<2.0.0, installed: 1.0.1]
      - s3transfer [required: >=0.5.0,<0.6.0, installed: 0.5.2]
        - botocore [required: >=1.12.36,<2.0a.0, installed: 1.26.10]
          - jmespath [required: >=0.7.1,<2.0.0, installed: 1.0.1]
          - python-dateutil [required: >=2.1,<3.0.0, installed: 2.8.2]
            - six [required: >=1.5, installed: 1.16.0]
          - urllib3 [required: >=1.25.4,<1.27, installed: 1.26.10]
    - dataclasses-json [required: Any, installed: 0.5.7]
      - marshmallow [required: >=3.3.0,<4.0.0, installed: 3.17.0]
        - packaging [required: >=17.0, installed: 21.3]
          - pyparsing [required: >=2.0.2,!=3.0.5, installed: 3.0.9]
      - marshmallow-enum [required: >=1.5.1,<2.0.0, installed: 1.5.1]
        - marshmallow [required: >=2.0.0, installed: 3.17.0]
          - packaging [required: >=17.0, installed: 21.3]
            - pyparsing [required: >=2.0.2,!=3.0.5, installed: 3.0.9]
      - typing-inspect [required: >=0.4.0, installed: 0.7.1]
        - mypy-extensions [required: >=0.3.0, installed: 0.4.3]
        - typing-extensions [required: >=3.7.4, installed: 4.3.0]
    - datadog [required: >=0.10.0, installed: 0.41.0]
      - decorator [required: >=3.3.2, installed: 5.1.1]
      - requests [required: >=2.6.0, installed: 2.28.1]
        - certifi [required: >=2017.4.17, installed: 2022.6.15]
        - charset-normalizer [required: >=2,<3, installed: 2.1.0]
        - idna [required: >=2.5,<4, installed: 3.3]
        - urllib3 [required: >=1.21.1,<1.27, installed: 1.26.10]
    - dnspython [required: Any, installed: 2.2.1]
    - jaeger-client [required: >4.5.0, installed: 4.8.0]
      - opentracing [required: >=2.1,<3.0, installed: 2.4.0]
      - threadloop [required: >=1,<2, installed: 1.0.2]
        - tornado [required: Any, installed: 6.2]
      - thrift [required: Any, installed: 0.16.0]
        - six [required: >=1.7.2, installed: 1.16.0]
      - tornado [required: >=4.3, installed: 6.2]
    - jmespath [required: >=0.9.3, installed: 1.0.1]
    - marshmallow [required: Any, installed: 3.17.0]
      - packaging [required: >=17.0, installed: 21.3]
        - pyparsing [required: >=2.0.2,!=3.0.5, installed: 3.0.9]
    - opentracing [required: >=2.4.0, installed: 2.4.0]
    - pyformance [required: >=10.2.2, installed: 10.2.6]
      - mock [required: >=2.0.0, installed: 4.0.3]
      - six [required: Any, installed: 1.16.0]
    - python-dateutil [required: >=2.6.1, installed: 2.8.2]
      - six [required: >=1.5, installed: 1.16.0]
    - pytz [required: >=2016.6.1, installed: 2022.1]
    - PyYAML [required: >=5.1, installed: 6.0]
    - requests [required: >=2.21.0, installed: 2.28.1]
      - certifi [required: >=2017.4.17, installed: 2022.6.15]
      - charset-normalizer [required: >=2,<3, installed: 2.1.0]
      - idna [required: >=2.5,<4, installed: 3.3]
      - urllib3 [required: >=1.21.1,<1.27, installed: 1.26.10]
    - setuptools [required: Any, installed: 56.0.0]
e
This is a Poetry venv listing or Pip-based?
b
pip based
e
Can you provide a freeze instead, that's a bit easier to work with.
b
Sure.
Copy code
attrs==21.4.0
boto3==1.23.10
botocore==1.26.10
certifi==2022.6.15
charset-normalizer==2.1.0
dataclasses-json==0.5.7
datadog==0.41.0
datadog-lambda==3.60.0
ddtrace==0.61.1
decorator==5.1.1
dnspython==2.2.1
gh-core==0.2.23
ghlambda==0.5.6
idna==3.3
jaeger-client==4.8.0
jmespath==1.0.1
marshmallow==3.17.0
marshmallow-enum==1.5.1
mock==4.0.3
mypy-extensions==0.4.3
opentracing==2.4.0
packaging==21.3
pipdeptree==2.2.1
protobuf==3.20.1
pyformance==10.2.6
pyparsing==3.0.9
python-dateutil==2.8.2
pytz==2022.1
PyYAML==6.0
requests==2.28.1
s3transfer==0.5.2
six==1.16.0
tenacity==8.0.1
threadloop==1.0.2
thrift==0.16.0
tornado==6.2
typing-inspect==0.7.1
typing_extensions==4.3.0
urllib3==1.26.10
wrapt==1.14.1
e
Thanks!
While I check that, and back on what has changed - do you use 
python_distribution
targets for any of the internal dependencies?
OK, this works fine where 
rwquirements.txt
contains the freeze you provided:
Copy code
$ grep -v -E "gh-core|ghlambda|pyformance" requirements.txt | xargs ~/.venv/pex/bin/pex --python python3.8 pex==2.1.71 -cpex -- -otest-resolve.pex
b
Yes. We have a Pants based monorepo which publishes many python distributions
e
I had to exclude pyformance since:
Copy code
ERROR: Could not find a version that satisfies the requirement pyformance==10.2.6 (from -r requirements.txt (line 23)) (from versions: 0.2, 0.2.1, 0.2.3, 0.2.4, 0.2.5, 0.3.1, 0.3.2, 0.3.3, 0.3.4, 0.4)
ERROR: No matching distribution found for pyformance==10.2.6 (from -r requirements.txt (line 23))
b
Yeah, we use a forked version that hosted internally
e
Yes. We have a Pants based monorepo which publishes many python distributions
OK, so do any of those 
python_distribution
targets have recent changes? Do they use custom 
setup.py
with maybe bad 
install_requires
?
The parse error looks like an errant string list somewhere: 
"','"
b
There are recent changes. But nothing looks suspicious.
The puzzling thing is, if I remove the 
boto3
dependency from the project I’m getting the error in, it all works fine
e
Ok, and circling back, are there recent change to the Pants version or have you been on 2.10.0 when all this was working?
b
No changes to Pants version or the toml files
e
Can you get dirty and hack in a debug print?:
Copy code
Traceback (most recent call last):
  File "/Users/dmistry/.cache/pants/named_caches/pex_root/venvs/s/66ef3043/venv/lib/python3.8/site-packages/pip/_vendor/packaging/requirements.py", line 113, in __init__
    req = REQUIREMENT.parseString(requirement_string)
  File "/Users/dmistry/.cache/pants/named_caches/pex_root/venvs/s/66ef3043/venv/lib/python3.8/site-packages/pip/_vendor/pyparsing.py", line 1955, in parseString
    raise exc
  File "/Users/dmistry/.cache/pants/named_caches/pex_root/venvs/s/66ef3043/venv/lib/python3.8/site-packages/pip/_vendor/pyparsing.py", line 3814, in parseImpl
    raise ParseException(instring, loc, self.errmsg, self)
pip._vendor.pyparsing.ParseException: Expected stringEnd, found ','  (at char 23), (line:1, col:24)
That suggests adding a 
print(f">>> {requirement_string}", file=sys.stderr)
just above 
/Users/dmistry/.cache/pants/named_caches/pex_root/venvs/s/66ef3043/venv/lib/python3.8/site-packages/pip/_vendor/packaging/requirements.py
line 113.
At least we'd see the requirement string its choking on and that might help things along.
b
Good idea. Let me try that.
Ah, it’s the botocore dependency
Copy code
>>>>> Parsing requirement = boto3
>>>>> Parsing requirement = botocore>=1.3.0,<1.4.0',
e
Ok. Hopefully you can get a quick grep for that string!
I have no idea where that's coming from.
b
I think it’s because we don’t use lockfiles. So there is some backtracking going on in resolving the boto3 lib
e
So you hypothesize some old version of boto3 has botched Requires-Dist metadata with the stray trailing 
',
and Pip chokes on that during a backtrack?
b
e
Ah, excellent. That should really yank that.
b
I’ll look to start using lockfiles in that project.
Thanks for you help 👍
e
The lock file resolve will hit the same issue of course. Once the lock is created, you're good to go, but still the same root issue calculating the lock.
7 Views
Open in Slack
PreviousNext
Powered by