Hey I have a question on pip requirements txt dependency res

Question

Hey I have a question on pip requirements txt dependency resolution We have a primarily Python monorepo with several modules all with their own requirements txt and constrained by a top level constraints txt These modules are both intended to be self contained so some degree but also get imported from other modules We re running into problems specifically when doing a top level ` pants test ` or other goals where it attempts to resolve all requirements from the requirements txt but because some modules have common dependencies it errors out with ```AssertionError More than one direct requirement is satisfied by cbor2 5 3 0 0 cbor2 1 cbor2 lt 6 0 gt =5 0 1 This should never happen since Pip fails when more than one requirement for a given project name key is supplied and applies for a given target interpreter environment ``` We can get it to work if all requirements txt exactly match however when using a constraints file this shouldn t be necessary

calm-ambulance-65371 · Answer

This appears to be an issue with how pants is feeding requirements into pex without doing any deduplication which is problematic for modularized monorepos

enough-analyst-54434 · Answer

To baseline this what version of Pants are you using

calm-ambulance-65371 · Answer

2 4 0 and 2 3 0 Appears to work fine on 2 2 0 which implies it s related to the pip dependency resolving change

enough-analyst-54434 · Answer

I don t think it will be relevant but what do you have configured for ` python setup resolver version` if anything That changed from a default of pip legacy resolver in 2 2 and 2 3 to pip 2020 resolver in 2 4

calm-ambulance-65371 · Answer

it s unconfigured

enough-analyst-54434 · Answer

Ok I need to pop to a meeting but I ll investigate ~30 minutes from now

calm-ambulance-65371 · Answer

Let me know if you need any info from me

enough-analyst-54434 · Answer

Actually some extra info would be great Can you run this failure scenario adding ` ldebug` to the command line and then supply the full output

enough-analyst-54434 · Answer

For context although it sounds like you figured this out this is what happens when you feed the same project name in more than one requirement to Pex ```$ pex cbor2 cbor2 lt 6 0 gt =5 0 1 ERROR Double requirement given cbor2 lt 6 0 gt =5 0 1 already in cbor2 name= cbor2 pid 13787 gt home jsirois pex venvs 8a893dcb01b3ee032bd99c2db556cfa3d015123d 832e95f85413646e8b7b056ab4e60414f83d7397 pex disable pip version check no python version warning exists action a use deprecated legacy resolver isolated q cache dir home jsirois pex download dest tmp tmphrmpjbl1 home jsirois venv pex bin python3 cbor2 cbor2 lt 6 0 gt =5 0 1 index url <https pypi org simple> retries 5 timeout 15 exited with 1 and STDERR None``` I E You get a Pip error and that error happens before the assertion you see in your error That s what the assertion is referring to So Pants must be passing Pex requirements in a more interesting way

enough-analyst-54434 · Answer

Aha OK got it No need to run experiments You correctly called the issue Given ```$ cat constraints txt cbor2==5 0 1``` These are fine ```$ pex constraint constraints txt cbor2 lt 6 0 gt =5 0 1 c import cbor2 print cbor2 file home jsirois pex installed wheels cf07ef3d37b0f7c502152caf2610cbd6c8a82e8f cbor2 5 0 1 cp39 cp39 linux x86 64 whl cbor2 init py $ pex constraint constraints txt cbor2 c import cbor2 print cbor2 file home jsirois pex installed wheels cf07ef3d37b0f7c502152caf2610cbd6c8a82e8f cbor2 5 0 1 cp39 cp39 linux x86 64 whl cbor2 init py``` But this gives the failure you see ```$ pex constraint constraints txt cbor2 cbor2 lt 6 0 gt =5 0 1 c import cbor2 print cbor2 file Traceback most recent call last File home jsirois venv pex bin pex line 8 in lt module gt sys exit main File home jsirois venv pex lib python3 9 site packages pex bin pex py line 1145 in main pex builder = build pex options requirements options cache=ENV PEX ROOT File home jsirois venv pex lib python3 9 site packages pex bin pex py line 1023 in build pex resolveds = resolve multi File home jsirois venv pex lib python3 9 site packages pex resolver py line 1054 in resolve multi build and install request install distributions File home jsirois venv pex lib python3 9 site packages pex resolver py line 718 in install distributions raise AssertionError AssertionError More than one direct requirement is satisfied by cbor2 5 0 1 0 cbor2 1 cbor2 lt 6 0 gt =5 0 1 This should never happen since Pip fails when more than one requirement for a given project name key is supplied and applies for a given target interpreter environment ``` Without the constraints option Pex fails earlier in a `pip download` call ```$ pex cbor2 cbor2 lt 6 0 gt =5 0 1 c import cbor2 print cbor2 file ERROR Double requirement given cbor2 lt 6 0 gt =5 0 1 already in cbor2 name= cbor2 pid 16310 gt home jsirois pex venvs 8a893dcb01b3ee032bd99c2db556cfa3d015123d 832e95f85413646e8b7b056ab4e60414f83d7397 pex disable pip version check no python version warning exists action a use deprecated legacy resolver isolated q cache dir home jsirois pex download dest tmp tmps12q8x38 home jsirois venv pex bin python3 cbor2 cbor2 lt 6 0 gt =5 0 1 index url <https pypi org simple> retries 5 timeout 15 exited with 1 and STDERR None``` So either way as you say Pants is doing the wrong thing here and not deduping requirements or else not failing fast enough with a clear enough message that it thinks you have things configured wrongly

enough-analyst-54434 · Answer

Alright now I understand what s going on a bit better but let me restate what s going on and see if you concur < calm ambulance 65371> You have a consistent resolve globally in your repo thus your single global constraints file works for all projects and their requirements files Pants assumes in this case that you have corresponding single requirements file As such it does not need to perform any logic on the requirements since the global constraints prove they are unique Clearly not so here So the fact you have a single global constraints file means you could in fact have a single global requirements file You don t though for whatever reasons

calm-ambulance-65371 · Answer

That is correct we have a single top level constraints file and several requirements txt files typically 1 for each module

calm-ambulance-65371 · Answer

We do not have a global requirements txt to correspond to the constraints txt

enough-analyst-54434 · Answer

Ok Yeah that messes with Pants view of the world currently With Pants you could have 1 requirements file and Pants will correctly subset that to be an exact fit for any given binary test or PEX or distribution

enough-analyst-54434 · Answer

We currently have no logic to actually parse requirements and do anything with them like de dup

enough-analyst-54434 · Answer

Thinking

calm-ambulance-65371 · Answer

That s interesting Would that apply if you specified the `python requirements library` directly for two separate modules with overlapping 3rd party dependencies where one `import`ed the other

enough-analyst-54434 · Answer

The only rule is you can only specify a given dependency once wherever that is done No dups or triples etc

calm-ambulance-65371 · Answer

That seems problematic for multi project or complex monorepos

calm-ambulance-65371 · Answer

even moreso if you re packaging modules

enough-analyst-54434 · Answer

How so Given a consistent global resolve which you have it works unless I m missing something

enough-analyst-54434 · Answer

We do have workd pending to support inconsistent resolves with multiple lock files

enough-analyst-54434 · Answer

Back to the problem at hand do you have ` python setup resolve all constraints` configured

calm-ambulance-65371 · Answer

I thought we did but it appears we do not

enough-analyst-54434 · Answer

Aha If your constraints is globally consistent and all pinned effectively a lock file you want that

enough-analyst-54434 · Answer

That should clear this up for your case

enough-analyst-54434 · Answer

When you set that Pants will resolve the whole constraints file once and then subresolve out from that locally when it goes to resolve tests PEXes etc

enough-analyst-54434 · Answer

That s generally a speed hack with no downsides The final subsetted resolve is still what you d get resolving each test PEX file etc individually

enough-analyst-54434 · Answer

And that speed hack should step around your layout with dup requirements

calm-ambulance-65371 · Answer

oh cool it does appear to have fixed the issue

enough-analyst-54434 · Answer

Ok If its not clear what s going on here I can explain more Fire away with questions if so

enough-analyst-54434 · Answer

You ve still found a bug though We need to do better for folks with your multiple requirements txt layout when they do not have a globally consistent resolve I ll file an issue here in a bit

calm-ambulance-65371 · Answer

What s the intent behind the requirement for requirements txt to be a subset of the constraints txt listings in order for `resolve all constraints` to be honored It seems I spoke too soon before my tests eventually break they got further though on the same issue with a different package even with `resolve all constraints` set to always I m guessing this has something to do with it ``` WARN Ignoring ` python setup resolve all constraints` option because constraints file does not cover all requirements ```

calm-ambulance-65371 · Answer

For further context we cannot put all of our packages in `constraints txt` as they are using a `git+https ` format which is illegal for `constraints txt`

calm-ambulance-65371 · Answer

Oh the ignore messages is probably just because it has to re resolve for that given module

enough-analyst-54434 · Answer

Yeah So is the break due to unexpected resolved requirements at runtime in the tests or is the actual resolve failing

enough-analyst-54434 · Answer

But also I think you can cover `git+https ` in constraints by just adding the unqualified project name So for either of these forms of git requirement ```git+ bar git+``` Just add `bar` to constraints txt

enough-analyst-54434 · Answer

Ah right that doesn t work Sorry

enough-analyst-54434 · Answer

< calm ambulance 65371> before proceeding I ll wait for an answer to gt Yeah So is the break due to unexpected resolved requirements at runtime in the tests or is the actual resolve failing

calm-ambulance-65371 · Answer

The actual resolve is failing

calm-ambulance-65371 · Answer

it fails when building the pex for a given subset of tests some pass through fine

calm-ambulance-65371 · Answer

```09 24 13 78 INFO Completed Building requirements pex with 9 requirements PyJWT==1 7 1 matplotlib==3 4 1 numpy==1 18 5 protobuf~=3 11 1 pydantic==1 8 1 requests lt 3 gt =2 24 0 requests==2 24 0 scipy==1 6 3 typing extensions 9 characters truncated 09 24 13 80 ERROR Exception caught pants engine internals scheduler ExecutionError File home noah cache pants setup bootstrap Linux x86 64 2 4 0 py38 lib python3 8 site packages pants bin local pants runner py line 229 in run inner return self perform run goals File home noah cache pants setup bootstrap Linux x86 64 2 4 0 py38 lib python3 8 site packages pants bin local pants runner py line 168 in perform run return self perform run body goals poll=False File home noah cache pants setup bootstrap Linux x86 64 2 4 0 py38 lib python3 8 site packages pants bin local pants runner py line 185 in perform run body return self graph session run goal rules File home noah cache pants setup bootstrap Linux x86 64 2 4 0 py38 lib python3 8 site packages pants init engine initializer py line 135 in run goal rules exit code = self scheduler session run goal rule File home noah cache pants setup bootstrap Linux x86 64 2 4 0 py38 lib python3 8 site packages pants engine internals scheduler py line 530 in run goal rule self raise on error t for t in throws File home noah cache pants setup bootstrap Linux x86 64 2 4 0 py38 lib python3 8 site packages pants engine internals scheduler py line 498 in raise on error raise ExecutionError Exception message 1 Exception encountered ProcessExecutionFailure Process Building requirements pex with 9 requirements PyJWT==1 7 1 matplotlib==3 4 1 numpy==1 18 5 protobuf~=3 11 1 pydantic==1 8 1 requests lt 3 gt =2 24 0 requests==2 24 0 scipy==1 6 3 typing extensions gt =3 7 4 3 failed with exit code 1 stdout stderr Traceback most recent call last File home noah cache pants named caches pex root unzipped pexes fe5a6e33c47ab8f54825dc7dd841561189270730 bootstrap pex pex py line 489 in execute exit value = self wrap coverage self wrap profiling self execute File home noah cache pants named caches pex root unzipped pexes fe5a6e33c47ab8f54825dc7dd841561189270730 bootstrap pex pex py line 406 in wrap coverage return runner args File home noah cache pants named caches pex root unzipped pexes fe5a6e33c47ab8f54825dc7dd841561189270730 bootstrap pex pex py line 437 in wrap profiling return runner args File home noah cache pants named caches pex root unzipped pexes fe5a6e33c47ab8f54825dc7dd841561189270730 bootstrap pex pex py line 545 in execute return self execute entry self pex info entry point File home noah cache pants named caches pex root unzipped pexes fe5a6e33c47ab8f54825dc7dd841561189270730 bootstrap pex pex py line 661 in execute entry return self execute pkg resources entry point File home noah cache pants named caches pex root unzipped pexes fe5a6e33c47ab8f54825dc7dd841561189270730 bootstrap pex pex py line 693 in execute pkg resources return runner File home noah cache pants named caches pex root unzipped pexes fe5a6e33c47ab8f54825dc7dd841561189270730 deps pex 2 1 35 py2 py3 none any whl pex bin pex py line 1088 in main pex builder = build pex options requirements options cache=ENV PEX ROOT File home noah cache pants named caches pex root unzipped pexes fe5a6e33c47ab8f54825dc7dd841561189270730 deps pex 2 1 35 py2 py3 none any whl pex bin pex py line 966 in build pex resolveds = resolve multi File home noah cache pants named caches pex root unzipped pexes fe5a6e33c47ab8f54825dc7dd841561189270730 deps pex 2 1 35 py2 py3 none any whl pex resolver py line 1049 in resolve multi build and install request install distributions File home noah cache pants named caches pex root unzipped pexes fe5a6e33c47ab8f54825dc7dd841561189270730 deps pex 2 1 35 py2 py3 none any whl pex resolver py line 715 in install distributions raise AssertionError AssertionError More than one direct requirement is satisfied by requests 2 24 0 0 requests lt 3 gt =2 24 0 1 requests==2 24 0 This should never happen since Pip fails when more than one requirement for a given project name key is supplied and applies for a given target interpreter environment ```

enough-analyst-54434 · Answer

Ok I m headed out for about 8 days so someone else will need to help you through this It might be good to file an issue with pertinent details and then ask folks to look there