https://pantsbuild.org/ logo
b

big-xylophone-43403

07/07/2022, 10:02 PM
Running into a weird issue with Pants which worked before. It’s unable to build requirements.pex. Failing with a requirement parsing error. Details in 🧵
Pants version
2.10.0
Copy code
16:38:03.43 [ERROR] 1 Exception encountered:

Engine traceback:
  in select
  in pants.core.goals.test.run_tests
  in pants.backend.python.goals.pytest_runner.run_python_test (src/python/func-olorailsmenugetter/tests/test_app.py:../test_py38)
  in pants.backend.python.goals.pytest_runner.setup_pytest_for_target
  in pants.backend.python.util_rules.pex.create_pex
  in pants.backend.python.util_rules.pex.build_pex (requirements.pex)
  in pants.engine.process.fallible_to_exec_result_or_raise
Traceback (most recent call last):
  File "/Users/dmistry/.cache/pants/setup/bootstrap-Darwin-x86_64/pants.vwr5Ve/install/lib/python3.9/site-packages/pants/engine/process.py", line 285, in fallible_to_exec_result_or_raise
    raise ProcessExecutionFailure(
pants.engine.process.ProcessExecutionFailure: Process 'Building requirements.pex with 3 requirements: boto3, gh-core, ghlambda' failed with exit code 1.
stdout:

stderr:
pex: Resolving interpreters
pex: Resolving interpreters: 0.1ms
pex: Building pex
pex: Building pex :: Resolving distributions (boto3 gh-core ghlambda)
pex: Building pex :: Resolving distributions (boto3 gh-core ghlambda) :: Resolving requirements.
pex: Building pex :: Resolving distributions (boto3 gh-core ghlambda) :: Resolving requirements. :: Resolving for:
  /Users/dmistry/.pyenv/versions/3.8.13/bin/python3.8
pex: Spawning a maximum of 3 parallel jobs to process:
  /Users/dmistry/.pyenv/versions/3.8.13/bin/python3.8
pex: Hashing pex
pex: Hashing pex: 86.7ms
pex: Isolating pex
pex: Isolating pex: 0.1ms
    Running command python setup.py egg_info
    running egg_info
    creating /private/var/folders/z1/8_j4n7t17h15skx5y6h69whh0000gp/T/process-executionYS0bXh/.tmp/pip-pip-egg-info-m5r1z055/jaeger_client.egg-info
    writing /private/var/folders/z1/8_j4n7t17h15skx5y6h69whh0000gp/T/process-executionYS0bXh/.tmp/pip-pip-egg-info-m5r1z055/jaeger_client.egg-info/PKG-INFO
    writing dependency_links to /private/var/folders/z1/8_j4n7t17h15skx5y6h69whh0000gp/T/process-executionYS0bXh/.tmp/pip-pip-egg-info-m5r1z055/jaeger_client.egg-info/dependency_links.txt
    writing requirements to /private/var/folders/z1/8_j4n7t17h15skx5y6h69whh0000gp/T/process-executionYS0bXh/.tmp/pip-pip-egg-info-m5r1z055/jaeger_client.egg-info/requires.txt
    writing top-level names to /private/var/folders/z1/8_j4n7t17h15skx5y6h69whh0000gp/T/process-executionYS0bXh/.tmp/pip-pip-egg-info-m5r1z055/jaeger_client.egg-info/top_level.txt
    writing manifest file '/private/var/folders/z1/8_j4n7t17h15skx5y6h69whh0000gp/T/process-executionYS0bXh/.tmp/pip-pip-egg-info-m5r1z055/jaeger_client.egg-info/SOURCES.txt'
    reading manifest file '/private/var/folders/z1/8_j4n7t17h15skx5y6h69whh0000gp/T/process-executionYS0bXh/.tmp/pip-pip-egg-info-m5r1z055/jaeger_client.egg-info/SOURCES.txt'
    reading manifest template '<http://MANIFEST.in|MANIFEST.in>'
    warning: no files found matching '*.thrift' under directory 'jaeger_client'
    warning: no files found matching '*.yaml'
    warning: no files found matching '*.json' under directory 'config'
    warning: no files found matching '*.yaml' under directory 'config'
    warning: no files found matching '*.rst' under directory 'docs'
    writing manifest file '/private/var/folders/z1/8_j4n7t17h15skx5y6h69whh0000gp/T/process-executionYS0bXh/.tmp/pip-pip-egg-info-m5r1z055/jaeger_client.egg-info/SOURCES.txt'
ERROR: Exception:
Traceback (most recent call last):
  File "/Users/dmistry/.cache/pants/named_caches/pex_root/venvs/s/66ef3043/venv/lib/python3.8/site-packages/pip/_vendor/pkg_resources/__init__.py", line 3021, in _dep_map
    return self.__dep_map
  File "/Users/dmistry/.cache/pants/named_caches/pex_root/venvs/s/66ef3043/venv/lib/python3.8/site-packages/pip/_vendor/pkg_resources/__init__.py", line 2815, in __getattr__
    raise AttributeError(attr)
AttributeError: _DistInfoDistribution__dep_map

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/dmistry/.cache/pants/named_caches/pex_root/venvs/s/66ef3043/venv/lib/python3.8/site-packages/pip/_vendor/packaging/requirements.py", line 113, in __init__
    req = REQUIREMENT.parseString(requirement_string)
  File "/Users/dmistry/.cache/pants/named_caches/pex_root/venvs/s/66ef3043/venv/lib/python3.8/site-packages/pip/_vendor/pyparsing.py", line 1955, in parseString
    raise exc
  File "/Users/dmistry/.cache/pants/named_caches/pex_root/venvs/s/66ef3043/venv/lib/python3.8/site-packages/pip/_vendor/pyparsing.py", line 3814, in parseImpl
    raise ParseException(instring, loc, self.errmsg, self)
pip._vendor.pyparsing.ParseException: Expected stringEnd, found ','  (at char 23), (line:1, col:24)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/dmistry/.cache/pants/named_caches/pex_root/venvs/s/66ef3043/venv/lib/python3.8/site-packages/pip/_vendor/pkg_resources/__init__.py", line 3101, in __init__
    super(Requirement, self).__init__(requirement_string)
  File "/Users/dmistry/.cache/pants/named_caches/pex_root/venvs/s/66ef3043/venv/lib/python3.8/site-packages/pip/_vendor/packaging/requirements.py", line 115, in __init__
    raise InvalidRequirement(
pip._vendor.packaging.requirements.InvalidRequirement: Parse error at "','": Expected stringEnd

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/dmistry/.cache/pants/named_caches/pex_root/venvs/s/66ef3043/venv/lib/python3.8/site-packages/pip/_internal/cli/base_command.py", line 223, in _main
    status = self.run(options, args)
  File "/Users/dmistry/.cache/pants/named_caches/pex_root/venvs/s/66ef3043/venv/lib/python3.8/site-packages/pip/_internal/cli/req_command.py", line 180, in wrapper
    return func(self, options, args)
  File "/Users/dmistry/.cache/pants/named_caches/pex_root/venvs/s/66ef3043/venv/lib/python3.8/site-packages/pip/_internal/commands/download.py", line 130, in run
    requirement_set = resolver.resolve(
  File "/Users/dmistry/.cache/pants/named_caches/pex_root/venvs/s/66ef3043/venv/lib/python3.8/site-packages/pip/_internal/resolution/resolvelib/resolver.py", line 121, in resolve
    self._result = resolver.resolve(
  File "/Users/dmistry/.cache/pants/named_caches/pex_root/venvs/s/66ef3043/venv/lib/python3.8/site-packages/pip/_vendor/resolvelib/resolvers.py", line 453, in resolve
    state = resolution.resolve(requirements, max_rounds=max_rounds)
  File "/Users/dmistry/.cache/pants/named_caches/pex_root/venvs/s/66ef3043/venv/lib/python3.8/site-packages/pip/_vendor/resolvelib/resolvers.py", line 347, in resolve
    failure_causes = self._attempt_to_pin_criterion(name, criterion)
  File "/Users/dmistry/.cache/pants/named_caches/pex_root/venvs/s/66ef3043/venv/lib/python3.8/site-packages/pip/_vendor/resolvelib/resolvers.py", line 207, in _attempt_to_pin_criterion
    criteria = self._get_criteria_to_update(candidate)
  File "/Users/dmistry/.cache/pants/named_caches/pex_root/venvs/s/66ef3043/venv/lib/python3.8/site-packages/pip/_vendor/resolvelib/resolvers.py", line 198, in _get_criteria_to_update
    for r in self._p.get_dependencies(candidate):
  File "/Users/dmistry/.cache/pants/named_caches/pex_root/venvs/s/66ef3043/venv/lib/python3.8/site-packages/pip/_internal/resolution/resolvelib/provider.py", line 170, in get_dependencies
    return [
  File "/Users/dmistry/.cache/pants/named_caches/pex_root/venvs/s/66ef3043/venv/lib/python3.8/site-packages/pip/_internal/resolution/resolvelib/provider.py", line 170, in <listcomp>
    return [
  File "/Users/dmistry/.cache/pants/named_caches/pex_root/venvs/s/66ef3043/venv/lib/python3.8/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 247, in iter_dependencies
    requires = self.dist.requires() if with_requires else ()
  File "/Users/dmistry/.cache/pants/named_caches/pex_root/venvs/s/66ef3043/venv/lib/python3.8/site-packages/pip/_vendor/pkg_resources/__init__.py", line 2736, in requires
    dm = self._dep_map
  File "/Users/dmistry/.cache/pants/named_caches/pex_root/venvs/s/66ef3043/venv/lib/python3.8/site-packages/pip/_vendor/pkg_resources/__init__.py", line 3023, in _dep_map
    self.__dep_map = self._compute_dependencies()
  File "/Users/dmistry/.cache/pants/named_caches/pex_root/venvs/s/66ef3043/venv/lib/python3.8/site-packages/pip/_vendor/pkg_resources/__init__.py", line 3033, in _compute_dependencies
    reqs.extend(parse_requirements(req))
  File "/Users/dmistry/.cache/pants/named_caches/pex_root/venvs/s/66ef3043/venv/lib/python3.8/site-packages/pip/_vendor/pkg_resources/__init__.py", line 3094, in parse_requirements
    yield Requirement(line)
  File "/Users/dmistry/.cache/pants/named_caches/pex_root/venvs/s/66ef3043/venv/lib/python3.8/site-packages/pip/_vendor/pkg_resources/__init__.py", line 3103, in __init__
    raise RequirementParseError(str(e))
pip._vendor.pkg_resources.RequirementParseError: Parse error at "','": Expected stringEnd
pid 47076 -> /Users/dmistry/.cache/pants/named_caches/pex_root/venvs/44f0d229c64d262df3c9196eaa181a922a2a17cd/5190f2bb1c910d0c3977d742bb12392b7f02f9f9/pex --disable-pip-version-check --no-python-version-warning --exists-action a --isolated -vvv --cache-dir /Users/dmistry/.cache/pants/named_caches/pex_root --log /private/var/folders/z1/8_j4n7t17h15skx5y6h69whh0000gp/T/process-executionYS0bXh/.tmp/tmp5un1_0h6/pip.log download --dest /private/var/folders/z1/8_j4n7t17h15skx5y6h69whh0000gp/T/process-executionYS0bXh/.tmp/tmp6l7js_56/Users.dmistry..pyenv.versions.3.8.13.bin.python3.8 boto3 gh-core ghlambda --index-url <https://pypi.org/simple/> --extra-index-url <https://artifactory.grubhub.com/artifactory/api/pypi/pypi-repos/simple> --retries 5 --timeout 15 exited with 2 and STDERR:
The project only has 3 dependencies. 2 first party and 1 3rd party (boto3)
with this requirements macro -
Copy code
python_requirements(
    name="reqs",
    overrides={
        "ghlambda": {"dependencies": [":reqs#boto3"]},
    },
    module_mapping={
      "gh-core": ["gh"]
    },
)
If I remove the undeclared dependency (boto3), everything works
e

enough-analyst-54434

07/07/2022, 10:30 PM
Worked before ... in time, before a recent Pants upgrade, ... any more info on what may have changed?
Also, what is gh-core? That does not appear to be on PyPI.
b

big-xylophone-43403

07/07/2022, 10:34 PM
that is our in-house lib
Both gh-core and ghlambda
e

enough-analyst-54434

07/07/2022, 10:35 PM
Can you share its requirements? Basically can you share the public PyPI root set of reqs?
The more you can share the better of course. Hard to debug without knowing what is being resolved.
Or is boto3 it? Do the 1st party actually depend on 0 3rdparty except for boto3?
b

big-xylophone-43403

07/07/2022, 10:39 PM
They do depend on other 3rdparty libs. Let me share the full list
e

enough-analyst-54434

07/07/2022, 10:39 PM
Thank you. That at least allows a quick rule in / out of Pex vs Pants ti whittle things down.
Pants 2.10.0 is Pex 2.1.71 and
$ pex --python python3.8 pex==2.1.71 -cpex -- boto3 -o example.pex
works for example.
b

big-xylophone-43403

07/07/2022, 10:41 PM
Here is the dep tree
Copy code
ghlambda==0.5.6
  - datadog-lambda [required: Any, installed: 3.60.0]
    - datadog [required: >=0.41.0,<0.42.0, installed: 0.41.0]
      - decorator [required: >=3.3.2, installed: 5.1.1]
      - requests [required: >=2.6.0, installed: 2.28.1]
        - certifi [required: >=2017.4.17, installed: 2022.6.15]
        - charset-normalizer [required: >=2,<3, installed: 2.1.0]
        - idna [required: >=2.5,<4, installed: 3.3]
        - urllib3 [required: >=1.21.1,<1.27, installed: 1.26.10]
    - ddtrace [required: >=0.61.1,<0.62.0, installed: 0.61.1]
      - attrs [required: >=19.2.0, installed: 21.4.0]
      - packaging [required: >=17.1, installed: 21.3]
        - pyparsing [required: >=2.0.2,!=3.0.5, installed: 3.0.9]
      - protobuf [required: >=3,<4, installed: 3.20.1]
      - six [required: >=1.12.0, installed: 1.16.0]
      - tenacity [required: >=5, installed: 8.0.1]
    - wrapt [required: >=1.11.2,<2.0.0, installed: 1.14.1]
  - gh-core [required: >=0.1.22, installed: 0.2.23]
    - boto3 [required: <1.24.0, installed: 1.23.10]
      - botocore [required: >=1.26.10,<1.27.0, installed: 1.26.10]
        - jmespath [required: >=0.7.1,<2.0.0, installed: 1.0.1]
        - python-dateutil [required: >=2.1,<3.0.0, installed: 2.8.2]
          - six [required: >=1.5, installed: 1.16.0]
        - urllib3 [required: >=1.25.4,<1.27, installed: 1.26.10]
      - jmespath [required: >=0.7.1,<2.0.0, installed: 1.0.1]
      - s3transfer [required: >=0.5.0,<0.6.0, installed: 0.5.2]
        - botocore [required: >=1.12.36,<2.0a.0, installed: 1.26.10]
          - jmespath [required: >=0.7.1,<2.0.0, installed: 1.0.1]
          - python-dateutil [required: >=2.1,<3.0.0, installed: 2.8.2]
            - six [required: >=1.5, installed: 1.16.0]
          - urllib3 [required: >=1.25.4,<1.27, installed: 1.26.10]
    - dataclasses-json [required: Any, installed: 0.5.7]
      - marshmallow [required: >=3.3.0,<4.0.0, installed: 3.17.0]
        - packaging [required: >=17.0, installed: 21.3]
          - pyparsing [required: >=2.0.2,!=3.0.5, installed: 3.0.9]
      - marshmallow-enum [required: >=1.5.1,<2.0.0, installed: 1.5.1]
        - marshmallow [required: >=2.0.0, installed: 3.17.0]
          - packaging [required: >=17.0, installed: 21.3]
            - pyparsing [required: >=2.0.2,!=3.0.5, installed: 3.0.9]
      - typing-inspect [required: >=0.4.0, installed: 0.7.1]
        - mypy-extensions [required: >=0.3.0, installed: 0.4.3]
        - typing-extensions [required: >=3.7.4, installed: 4.3.0]
    - datadog [required: >=0.10.0, installed: 0.41.0]
      - decorator [required: >=3.3.2, installed: 5.1.1]
      - requests [required: >=2.6.0, installed: 2.28.1]
        - certifi [required: >=2017.4.17, installed: 2022.6.15]
        - charset-normalizer [required: >=2,<3, installed: 2.1.0]
        - idna [required: >=2.5,<4, installed: 3.3]
        - urllib3 [required: >=1.21.1,<1.27, installed: 1.26.10]
    - dnspython [required: Any, installed: 2.2.1]
    - jaeger-client [required: >4.5.0, installed: 4.8.0]
      - opentracing [required: >=2.1,<3.0, installed: 2.4.0]
      - threadloop [required: >=1,<2, installed: 1.0.2]
        - tornado [required: Any, installed: 6.2]
      - thrift [required: Any, installed: 0.16.0]
        - six [required: >=1.7.2, installed: 1.16.0]
      - tornado [required: >=4.3, installed: 6.2]
    - jmespath [required: >=0.9.3, installed: 1.0.1]
    - marshmallow [required: Any, installed: 3.17.0]
      - packaging [required: >=17.0, installed: 21.3]
        - pyparsing [required: >=2.0.2,!=3.0.5, installed: 3.0.9]
    - opentracing [required: >=2.4.0, installed: 2.4.0]
    - pyformance [required: >=10.2.2, installed: 10.2.6]
      - mock [required: >=2.0.0, installed: 4.0.3]
      - six [required: Any, installed: 1.16.0]
    - python-dateutil [required: >=2.6.1, installed: 2.8.2]
      - six [required: >=1.5, installed: 1.16.0]
    - pytz [required: >=2016.6.1, installed: 2022.1]
    - PyYAML [required: >=5.1, installed: 6.0]
    - requests [required: >=2.21.0, installed: 2.28.1]
      - certifi [required: >=2017.4.17, installed: 2022.6.15]
      - charset-normalizer [required: >=2,<3, installed: 2.1.0]
      - idna [required: >=2.5,<4, installed: 3.3]
      - urllib3 [required: >=1.21.1,<1.27, installed: 1.26.10]
    - setuptools [required: Any, installed: 56.0.0]
e

enough-analyst-54434

07/07/2022, 10:41 PM
This is a Poetry venv listing or Pip-based?
b

big-xylophone-43403

07/07/2022, 10:41 PM
pip based
e

enough-analyst-54434

07/07/2022, 10:42 PM
Can you provide a freeze instead, that's a bit easier to work with.
b

big-xylophone-43403

07/07/2022, 10:43 PM
Sure.
Copy code
attrs==21.4.0
boto3==1.23.10
botocore==1.26.10
certifi==2022.6.15
charset-normalizer==2.1.0
dataclasses-json==0.5.7
datadog==0.41.0
datadog-lambda==3.60.0
ddtrace==0.61.1
decorator==5.1.1
dnspython==2.2.1
gh-core==0.2.23
ghlambda==0.5.6
idna==3.3
jaeger-client==4.8.0
jmespath==1.0.1
marshmallow==3.17.0
marshmallow-enum==1.5.1
mock==4.0.3
mypy-extensions==0.4.3
opentracing==2.4.0
packaging==21.3
pipdeptree==2.2.1
protobuf==3.20.1
pyformance==10.2.6
pyparsing==3.0.9
python-dateutil==2.8.2
pytz==2022.1
PyYAML==6.0
requests==2.28.1
s3transfer==0.5.2
six==1.16.0
tenacity==8.0.1
threadloop==1.0.2
thrift==0.16.0
tornado==6.2
typing-inspect==0.7.1
typing_extensions==4.3.0
urllib3==1.26.10
wrapt==1.14.1
e

enough-analyst-54434

07/07/2022, 10:55 PM
Thanks!
While I check that, and back on what has changed - do you use
python_distribution
targets for any of the internal dependencies?
OK, this works fine where
rwquirements.txt
contains the freeze you provided:
Copy code
$ grep -v -E "gh-core|ghlambda|pyformance" requirements.txt | xargs ~/.venv/pex/bin/pex --python python3.8 pex==2.1.71 -cpex -- -otest-resolve.pex
b

big-xylophone-43403

07/07/2022, 11:07 PM
Yes. We have a Pants based monorepo which publishes many python distributions
e

enough-analyst-54434

07/07/2022, 11:07 PM
I had to exclude pyformance since:
Copy code
ERROR: Could not find a version that satisfies the requirement pyformance==10.2.6 (from -r requirements.txt (line 23)) (from versions: 0.2, 0.2.1, 0.2.3, 0.2.4, 0.2.5, 0.3.1, 0.3.2, 0.3.3, 0.3.4, 0.4)
ERROR: No matching distribution found for pyformance==10.2.6 (from -r requirements.txt (line 23))
b

big-xylophone-43403

07/07/2022, 11:08 PM
Yeah, we use a forked version that hosted internally
e

enough-analyst-54434

07/07/2022, 11:08 PM
Yes. We have a Pants based monorepo which publishes many python distributions
OK, so do any of those
python_distribution
targets have recent changes? Do they use custom
setup.py
with maybe bad
install_requires
?
The parse error looks like an errant string list somewhere:
"','"
b

big-xylophone-43403

07/07/2022, 11:11 PM
There are recent changes. But nothing looks suspicious.
The puzzling thing is, if I remove the
boto3
dependency from the project I’m getting the error in, it all works fine
e

enough-analyst-54434

07/07/2022, 11:12 PM
Ok, and circling back, are there recent change to the Pants version or have you been on 2.10.0 when all this was working?
b

big-xylophone-43403

07/07/2022, 11:12 PM
No changes to Pants version or the toml files
e

enough-analyst-54434

07/07/2022, 11:23 PM
Can you get dirty and hack in a debug print?:
Copy code
Traceback (most recent call last):
  File "/Users/dmistry/.cache/pants/named_caches/pex_root/venvs/s/66ef3043/venv/lib/python3.8/site-packages/pip/_vendor/packaging/requirements.py", line 113, in __init__
    req = REQUIREMENT.parseString(requirement_string)
  File "/Users/dmistry/.cache/pants/named_caches/pex_root/venvs/s/66ef3043/venv/lib/python3.8/site-packages/pip/_vendor/pyparsing.py", line 1955, in parseString
    raise exc
  File "/Users/dmistry/.cache/pants/named_caches/pex_root/venvs/s/66ef3043/venv/lib/python3.8/site-packages/pip/_vendor/pyparsing.py", line 3814, in parseImpl
    raise ParseException(instring, loc, self.errmsg, self)
pip._vendor.pyparsing.ParseException: Expected stringEnd, found ','  (at char 23), (line:1, col:24)
That suggests adding a
print(f">>> {requirement_string}", file=sys.stderr)
just above
/Users/dmistry/.cache/pants/named_caches/pex_root/venvs/s/66ef3043/venv/lib/python3.8/site-packages/pip/_vendor/packaging/requirements.py
line 113.
At least we'd see the requirement string its choking on and that might help things along.
b

big-xylophone-43403

07/07/2022, 11:47 PM
Good idea. Let me try that.
Ah, it’s the botocore dependency
Copy code
>>>>> Parsing requirement = boto3
>>>>> Parsing requirement = botocore>=1.3.0,<1.4.0',
e

enough-analyst-54434

07/08/2022, 12:07 AM
Ok. Hopefully you can get a quick grep for that string!
I have no idea where that's coming from.
b

big-xylophone-43403

07/08/2022, 2:13 PM
I think it’s because we don’t use lockfiles. So there is some backtracking going on in resolving the boto3 lib
e

enough-analyst-54434

07/08/2022, 2:38 PM
So you hypothesize some old version of boto3 has botched Requires-Dist metadata with the stray trailing
',
and Pip chokes on that during a backtrack?
e

enough-analyst-54434

07/08/2022, 3:39 PM
Ah, excellent. That should really yank that.
b

big-xylophone-43403

07/08/2022, 3:39 PM
I’ll look to start using lockfiles in that project.
Thanks for you help 👍
e

enough-analyst-54434

07/08/2022, 3:54 PM
The lock file resolve will hit the same issue of course. Once the lock is created, you're good to go, but still the same root issue calculating the lock.
7 Views