https://pantsbuild.org/ logo
Join Slack
Powered by
Hey! I have a question on pip/requirements.txt dep...
# general
c
Hey! I have a question on pip/requirements.txt dependency resolution. We have a (primarily Python) monorepo with several modules, all with their own requirements.txt and constrained by a top-level constraints.txt. These modules are both intended to be self-contained so some degree, but also get imported from other modules. We're running into problems specifically when doing a top-level 
./pants test ::
(or other goals) where it attempts to resolve all requirements from the requirements.txt, but because some modules have common dependencies, it errors out with:
Copy code
AssertionError: More than one direct requirement is satisfied by cbor2 5.3.0:
0. cbor2
1. cbor2<6.0,>=5.0.1
This should never happen since Pip fails when more than one requirement for a given project name key is supplied and applies for a given target interpreter environment.
We can get it to work if all requirements.txt exactly match, however when using a constraints file, this shouldn't be necessary
This appears to be an issue with how pants is feeding requirements into pex without doing any deduplication, which is problematic for modularized monorepos
e
To baseline this, what version of Pants are you using?
c
2.4.0 and 2.3.0. Appears to work fine on 2.2.0, which implies it's related to the pip dependency resolving change
e
I don't think it will be relevant, but what do you have configured for 
[python-setup] resolver_version
if anything? That changed from a default of "pip-legacy-resolver" in 2.2 and 2.3 to "pip-2020-resolver" in 2.4.
c
it's unconfigured
e
Ok, I need to pop to a meeting but I'll investigate ~30 minutes from now.
👍 1
c
Let me know if you need any info from me
e
Actually - some extra info would be great. Can you run this failure scenario adding 
-ldebug
to the command line and then supply the full output?
For context, although it sounds like you figured this out, this is what happens when you feed the same project name in more than one requirement to Pex:
Copy code
$ pex cbor2 "cbor2<6.0,>=5.0.1"
ERROR: Double requirement given: cbor2<6.0,>=5.0.1 (already in cbor2, name='cbor2')
pid 13787 -> /home/jsirois/.pex/venvs/8a893dcb01b3ee032bd99c2db556cfa3d015123d/832e95f85413646e8b7b056ab4e60414f83d7397/pex --disable-pip-version-check --no-python-version-warning --exists-action a --use-deprecated legacy-resolver --isolated -q --cache-dir /home/jsirois/.pex download --dest /tmp/tmphrmpjbl1/home.jsirois..venv.pex.bin.python3 cbor2 cbor2<6.0,>=5.0.1 --index-url <https://pypi.org/simple> --retries 5 --timeout 15 exited with 1 and STDERR:
None
I.E. You get a Pip error and that error happens before the assertion you see in your error. That's what the assertion is referring to. So, Pants must be passing Pex requirements in a more interesting way.
Aha - OK, got it. No need to run experiments. You correctly called the issue. Given:
Copy code
$ cat constraints.txt 
cbor2==5.0.1
These are fine:
Copy code
$ pex --constraint constraints.txt "cbor2<6.0,>=5.0.1"  -- -c 'import cbor2; print(cbor2.__file__)'
/home/jsirois/.pex/installed_wheels/cf07ef3d37b0f7c502152caf2610cbd6c8a82e8f/cbor2-5.0.1-cp39-cp39-linux_x86_64.whl/cbor2/__init__.py
$ pex --constraint constraints.txt "cbor2"  -- -c 'import cbor2; print(cbor2.__file__)'
/home/jsirois/.pex/installed_wheels/cf07ef3d37b0f7c502152caf2610cbd6c8a82e8f/cbor2-5.0.1-cp39-cp39-linux_x86_64.whl/cbor2/__init__.py
But this gives the failure you see:
Copy code
$ pex --constraint constraints.txt "cbor2" "cbor2<6.0,>=5.0.1"  -- -c 'import cbor2; print(cbor2.__file__)'
Traceback (most recent call last):
  File "/home/jsirois/.venv/pex/bin/pex", line 8, in <module>
    sys.exit(main())
  File "/home/jsirois/.venv/pex/lib/python3.9/site-packages/pex/bin/pex.py", line 1145, in main
    pex_builder = build_pex(options.requirements, options, cache=ENV.PEX_ROOT)
  File "/home/jsirois/.venv/pex/lib/python3.9/site-packages/pex/bin/pex.py", line 1023, in build_pex
    resolveds = resolve_multi(
  File "/home/jsirois/.venv/pex/lib/python3.9/site-packages/pex/resolver.py", line 1054, in resolve_multi
    build_and_install_request.install_distributions(
  File "/home/jsirois/.venv/pex/lib/python3.9/site-packages/pex/resolver.py", line 718, in install_distributions
    raise AssertionError(
AssertionError: More than one direct requirement is satisfied by cbor2 5.0.1:
0. cbor2
1. cbor2<6.0,>=5.0.1
This should never happen since Pip fails when more than one requirement for a given project name key is supplied and applies for a given target interpreter environment.
Without the --constraints option, Pex fails earlier in a 
pip download
call:
Copy code
$ pex "cbor2" "cbor2<6.0,>=5.0.1"  -- -c 'import cbor2; print(cbor2.__file__)'
ERROR: Double requirement given: cbor2<6.0,>=5.0.1 (already in cbor2, name='cbor2')
pid 16310 -> /home/jsirois/.pex/venvs/8a893dcb01b3ee032bd99c2db556cfa3d015123d/832e95f85413646e8b7b056ab4e60414f83d7397/pex --disable-pip-version-check --no-python-version-warning --exists-action a --use-deprecated legacy-resolver --isolated -q --cache-dir /home/jsirois/.pex download --dest /tmp/tmps12q8x38/home.jsirois..venv.pex.bin.python3 cbor2 cbor2<6.0,>=5.0.1 --index-url <https://pypi.org/simple> --retries 5 --timeout 15 exited with 1 and STDERR:
None
So - either way - as you say - Pants is doing the wrong thing here and not deduping requirements or else not failing fast enough with a clear enough message that it thinks you have things configured wrongly.
Alright - now I understand what's going on a bit better, but let me restate what's going on and see if you concur @calm-ambulance-65371: You have a consistent resolve globally in your repo; thus your single global constraints file works for all projects and their requirements files. Pants assumes, in this case, that you have corresponding single requirements file. As such, it does not need to perform any logic on the requirements since the global constraints prove they are unique. Clearly not so here. So: the fact you have a single global constraints file means you could in fact have a single global requirements file. You don't though for whatever reasons.
c
That is correct, we have a single top-level constraints file and several requirements.txt files, typically 1 for each module
We do not have a global requirements.txt to correspond to the constraints.txt
e
Ok. Yeah - that messes with Pants view of the world currently. With Pants you could have 1 requirements file and Pants will correctly subset that to be an exact fit for any given binary (test or PEX or distribution).
We currently have no logic to actually parse requirements and do anything with them like de-dup.
Thinking....
c
That's interesting. Would that apply if you specified the 
python_requirements_library
directly for two separate modules with overlapping 3rd party dependencies where one `import`ed the other?
e
The only rule is you can only specify a given dependency once, wherever that is done. No dups or triples, etc.
c
That seems problematic for multi-project or complex monorepos
even moreso if you're packaging modules
e
How so? Given a consistent global resolve - which you have, it works unless I'm missing something.
We do have workd pending to support inconsistent resolves with multiple lock files.
Back to the problem at hand, do you have 
[python-setup] resolve_all_constraints
configured?
c
I thought we did, but it appears we do not
e
Aha. If your constraints is globally consistent and all pinned (effectively a lock file) you want that.
That should clear this up for your case.
When you set that Pants will resolve the whole constraints file once and then subresolve out from that locally when it goes to resolve tests, PEXes, etc.
That's generally a speed hack with no downsides. The final subsetted resolve is still what you'd get resolving each test PEX file, etc individually.
And that speed hack should step around your layout with dup requirements.
c
oh cool, it does appear to have fixed the issue
e
Ok. If its not clear what's going on here I can explain more. Fire away with questions if so.
You've still found a bug though. We need to do better for folks with your multiple requirements.txt layout when they do not have a globally consistent resolve. I'll file an issue here in a bit.
👍 2
c
What's the intent behind the requirement for requirements.txt to be a subset of the constraints.txt listings in order for 
resolve_all_constraints
to be honored? It seems I spoke too soon before, my tests eventually break (they got further though) on the same issue with a different package even with 
resolve_all_constraints
set to 'always'. I'm guessing this has something to do with it:
Copy code
[WARN] Ignoring `[python_setup].resolve_all_constraints` option because constraints file does not cover all requirements.
For further context, we cannot put all of our packages in 
constraints.txt
as they are using a 
git+https://...
format, which is illegal for 
constraints.txt
Oh, the ignore messages is probably just because it has to re-resolve for that given module
e
Yeah. So is the break due to unexpected resolved requirements at runtime in the tests or is the actual resolve failing?
But also - I think you can cover 
git+https://
in constraints by just adding the unqualified project name. So, for either of these forms of git requirement:
Copy code
git+<https://git.example.com/foo.git@da39a3ee5e6b4b0d3255bfef95601890afd80709#egg=bar>
bar @ git+<https://git.example.com/foo.git@da39a3ee5e6b4b0d3255bfef95601890afd80709>
Just add 
bar
to constraints.txt.
Ah, right, that doesn't work. Sorry.
@calm-ambulance-65371 before proceeding I'll wait for an answer to:
Yeah. So is the break due to unexpected resolved requirements at runtime in the tests or is the actual resolve failing?
c
The actual resolve is failing
it fails when building the pex for a given subset of tests (some pass through fine)
Copy code
09:24:13.78 [INFO] Completed: Building requirements.pex with 9 requirements: PyJWT==1.7.1, matplotlib==3.4.1, numpy==1.18.5, protobuf~=3.11.1, pydantic==1.8.1, requests<3,>=2.24.0, requests==2.24.0, scipy==1.6.3, typing_extensions... (9 characters truncated)
09:24:13.80 [ERROR] Exception caught: (pants.engine.internals.scheduler.ExecutionError)
  File "/home/noah/.cache/pants/setup/bootstrap-Linux-x86_64/2.4.0_py38/lib/python3.8/site-packages/pants/bin/local_pants_runner.py", line 229, in _run_inner
    return self._perform_run(goals)
  File "/home/noah/.cache/pants/setup/bootstrap-Linux-x86_64/2.4.0_py38/lib/python3.8/site-packages/pants/bin/local_pants_runner.py", line 168, in _perform_run
    return self._perform_run_body(goals, poll=False)
  File "/home/noah/.cache/pants/setup/bootstrap-Linux-x86_64/2.4.0_py38/lib/python3.8/site-packages/pants/bin/local_pants_runner.py", line 185, in _perform_run_body
    return self.graph_session.run_goal_rules(
  File "/home/noah/.cache/pants/setup/bootstrap-Linux-x86_64/2.4.0_py38/lib/python3.8/site-packages/pants/init/engine_initializer.py", line 135, in run_goal_rules
    exit_code = self.scheduler_session.run_goal_rule(
  File "/home/noah/.cache/pants/setup/bootstrap-Linux-x86_64/2.4.0_py38/lib/python3.8/site-packages/pants/engine/internals/scheduler.py", line 530, in run_goal_rule
    self._raise_on_error([t for _, t in throws])
  File "/home/noah/.cache/pants/setup/bootstrap-Linux-x86_64/2.4.0_py38/lib/python3.8/site-packages/pants/engine/internals/scheduler.py", line 498, in _raise_on_error
    raise ExecutionError(

Exception message: 1 Exception encountered:

  ProcessExecutionFailure: Process 'Building requirements.pex with 9 requirements: PyJWT==1.7.1, matplotlib==3.4.1, numpy==1.18.5, protobuf~=3.11.1, pydantic==1.8.1, requests<3,>=2.24.0, requests==2.24.0, scipy==1.6.3, typing_extensions>=3.7.4.3' failed with exit code 1.
stdout:

stderr:
Traceback (most recent call last):
  File "/home/noah/.cache/pants/named_caches/pex_root/unzipped_pexes/fe5a6e33c47ab8f54825dc7dd841561189270730/.bootstrap/pex/pex.py", line 489, in execute
    exit_value = self._wrap_coverage(self._wrap_profiling, self._execute)
  File "/home/noah/.cache/pants/named_caches/pex_root/unzipped_pexes/fe5a6e33c47ab8f54825dc7dd841561189270730/.bootstrap/pex/pex.py", line 406, in _wrap_coverage
    return runner(*args)
  File "/home/noah/.cache/pants/named_caches/pex_root/unzipped_pexes/fe5a6e33c47ab8f54825dc7dd841561189270730/.bootstrap/pex/pex.py", line 437, in _wrap_profiling
    return runner(*args)
  File "/home/noah/.cache/pants/named_caches/pex_root/unzipped_pexes/fe5a6e33c47ab8f54825dc7dd841561189270730/.bootstrap/pex/pex.py", line 545, in _execute
    return self.execute_entry(self._pex_info.entry_point)
  File "/home/noah/.cache/pants/named_caches/pex_root/unzipped_pexes/fe5a6e33c47ab8f54825dc7dd841561189270730/.bootstrap/pex/pex.py", line 661, in execute_entry
    return self.execute_pkg_resources(entry_point)
  File "/home/noah/.cache/pants/named_caches/pex_root/unzipped_pexes/fe5a6e33c47ab8f54825dc7dd841561189270730/.bootstrap/pex/pex.py", line 693, in execute_pkg_resources
    return runner()
  File "/home/noah/.cache/pants/named_caches/pex_root/unzipped_pexes/fe5a6e33c47ab8f54825dc7dd841561189270730/.deps/pex-2.1.35-py2.py3-none-any.whl/pex/bin/pex.py", line 1088, in main
    pex_builder = build_pex(options.requirements, options, cache=ENV.PEX_ROOT)
  File "/home/noah/.cache/pants/named_caches/pex_root/unzipped_pexes/fe5a6e33c47ab8f54825dc7dd841561189270730/.deps/pex-2.1.35-py2.py3-none-any.whl/pex/bin/pex.py", line 966, in build_pex
    resolveds = resolve_multi(
  File "/home/noah/.cache/pants/named_caches/pex_root/unzipped_pexes/fe5a6e33c47ab8f54825dc7dd841561189270730/.deps/pex-2.1.35-py2.py3-none-any.whl/pex/resolver.py", line 1049, in resolve_multi
    build_and_install_request.install_distributions(
  File "/home/noah/.cache/pants/named_caches/pex_root/unzipped_pexes/fe5a6e33c47ab8f54825dc7dd841561189270730/.deps/pex-2.1.35-py2.py3-none-any.whl/pex/resolver.py", line 715, in install_distributions
    raise AssertionError(
AssertionError: More than one direct requirement is satisfied by requests 2.24.0:
0. requests<3,>=2.24.0
1. requests==2.24.0
This should never happen since Pip fails when more than one requirement for a given project name key is supplied and applies for a given target interpreter environment.
e
Ok. I'm headed out for about 8 days so someone else will need to help you through this. It might be good to file an issue with pertinent details and then ask folks to look there.
5 Views
Open in Slack
PreviousNext
Powered by