Hey wave not sure If I m doing this correctly but I m gettin

Question

Hey wave not sure If I m doing this correctly but I m getting warnings about a deps conflict issues my use case ```python requirement name= tensorflow requirements= tensorflow==2 3 0 modules= tensorflow numpy scipy h5py resources name= config files sources= python sources name= lib dependencies= c extensions config files numpy tensorflow ``` and the warning ```18 41 06 42 WARN The target my target legacy imports `numpy` but Pants cannot safely infer a dependency because more than one target owns this module so it is ambiguous which to use numpy my target tensorflow Please explicitly include the dependency you want in the `dependencies` field of my target legacy or ignore the ones you do not want by prefixing with ` ` or ` ` so that one or no targets are left Alternatively you can remove the ambiguity by deleting changing some of the targets so that only 1 target owns this module Refer to <https www pantsbuild org v2 9 docs troubleshooting import errors and missing dependencies> ```

rapid-crayon-8232 · Answer

I ve excluded the global `numpy` dep but it still inferred when running the `test` goal for example

enough-analyst-54434 · Answer

Does the tensorflow distribution actually contain all those top level modules packages

enough-analyst-54434 · Answer

Those look like modules packages provided by its transitive dependencies

enough-analyst-54434 · Answer

I think you maybe just typed too much

rapid-crayon-8232 · Answer

ah yes I understood modules incorrectly then sweat smile

enough-analyst-54434 · Answer

So does this work ```python requirement name= tensorflow requirements= tensorflow==2 3 0 resources name= config files sources= python sources name= lib dependencies= c extensions config files ``` I E Why do you want to exclude numpy if tensorflow depends on it transitively

enough-analyst-54434 · Answer

For a bit too much info on tensorflow here are its top level modules and its dependencies ```^jsirois gill ~ Downloads $ zipinfo tensorflow 2 3 0 cp38 cp38 win amd64 whl | tail rw rw rw 2 0 fat 1332 b defN 20 Jul 24 04 19 tensorflow xla aot runtime src tensorflow core platform default dynamic annotations h rw rw rw 2 0 fat 1269 b defN 20 Jul 24 04 19 tensorflow xla aot runtime src tensorflow core platform default integral types h rw rw rw 2 0 fat 2399 b defN 20 Jul 24 04 19 tensorflow xla aot runtime src tensorflow core platform windows env time cc rw rw rw 2 0 fat 11419 b defN 20 Jul 24 04 19 tensorflow 2 3 0 dist info LICENSE rw rw rw 2 0 fat 2648 b defN 20 Jul 24 04 19 tensorflow 2 3 0 dist info METADATA rw rw rw 2 0 fat 105 b defN 20 Jul 24 04 19 tensorflow 2 3 0 dist info WHEEL rw rw rw 2 0 fat 469 b defN 20 Jul 24 04 19 tensorflow 2 3 0 dist info entry points txt rw rw rw 2 0 fat 11 b defN 20 Jul 24 04 19 tensorflow 2 3 0 dist info top level txt rw rw r 2 0 fat 925731 b defN 20 Jul 24 04 20 tensorflow 2 3 0 dist info RECORD 7502 files 979933102 bytes uncompressed 340951205 bytes compressed 65 2 ^jsirois gill ~ Downloads $ unzip qc tensorflow 2 3 0 cp38 cp38 win amd64 whl tensorflow 2 3 0 dist info top level txt tensorflow ^jsirois gill ~ Downloads $ unzip qc tensorflow 2 3 0 cp38 cp38 win amd64 whl tensorflow 2 3 0 dist info METADATA | grep Requires Requires Dist absl py gt =0 7 0 Requires Dist astunparse ==1 6 3 Requires Dist gast ==0 3 3 Requires Dist google pasta gt =0 1 8 Requires Dist h5py lt 2 11 0 gt =2 10 0 Requires Dist keras preprocessing lt 1 2 gt =1 1 1 Requires Dist numpy lt 1 19 0 gt =1 16 0 Requires Dist opt einsum gt =2 3 2 Requires Dist protobuf gt =3 9 2 Requires Dist tensorboard lt 3 gt =2 3 0 Requires Dist tensorflow estimator lt 2 4 0 gt =2 3 0 Requires Dist termcolor gt =1 1 0 Requires Dist wrapt gt =1 11 1 Requires Dist wheel gt =0 26 Requires Dist six gt =1 12 0 Requires Dist scipy ==1 4 1 Requires Dist grpcio gt =1 8 6 ```

rapid-crayon-8232 · Answer

I have a global numpy used by other modules 1 22 0 and tensorflow==2 3 0 only supports numpy lt 1 19 0 so with that config I get ``` The conflict is caused by The user requested numpy lt 2 0 0 and gt =1 22 3 tensorflow 2 3 0 depends on numpy lt 1 19 0 and gt =1 16 0 To fix this you could try to 1 loosen the range of package versions you ve specified 2 remove package versions to allow pip attempt to solve the dependency conflict```

enough-analyst-54434 · Answer

Aha Here < hundreds father 404> will probably want to guide you through multiple resolves There are other ways to support conflicting dependencies like this in other words on the bleeding edge of Pants

rapid-crayon-8232 · Answer

I ve tested with ```python requirement name= tensorflow requirements= tensorflow==2 3 0 python requirement name= numpy requirements= numpy==1 18 5 python sources name= lib dependencies= c extensions config files numpy tensorflow ``` But I still get the warning because I import numpy in my tests and dependency inference picks up both of them

rapid-crayon-8232 · Answer

To be more precise the tests build correctly and pass using the above but I get the warning specifically on the test files where `numpy` is imported not the rest so it s probably a bug with dependency inference logging incorrect warnings

rapid-crayon-8232 · Answer

~The following seems to avoid warnings ~ ```python requirement name= tensorflow requirements= tensorflow==2 3 0 python requirement name= numpy requirements= numpy==1 18 5 python sources name= lib dependencies= c extensions config files numpy tensorflow numpy ```

enough-analyst-54434 · Answer

That looks about right for the current stable state of the art

rapid-crayon-8232 · Answer

I talked a bit too fast apparently it was just a case of the built requirements being cached and reused so the warnings didn t show sweat with no cache I can reproduce the warnings with the above config

hundreds-father-404 · Answer

Indeed This sounds like a use case for multiple resolves ie multiple lockfiles which allows you to have conflicting versions of dependencies like NumPy It has its soft launch in Pants 2 10 soft due to the limitations explained at <https www pantsbuild org v2 10 docs reference python section enable resolves> I m still rewriting the general Python third party docs on this but that option and the next option explain how this new feature works

rapid-crayon-8232 · Answer

ah that s very cool indeed star struck I ll wait for the stable release of 2 10 for the switch I see it s around the corner

hundreds-father-404 · Answer

Our top voted issue in the January 2022 survey which we are blogging about shortly gt I ll wait for the stable release of 2 10 for the switch If you are willing to create a branch that updates to 2 10 but not yet land it necessarily that is super super helpful to us to have a more stable release

rapid-crayon-8232 · Answer

Yes I have abranch running with pants 2 10 I m note sure I understand this warnings thought when running ` pants update build files fix python macros` ``` `poetry requirements` in BUILD add `name= poetry 09 57 31 02 ERROR You must manually add the `name=` field to the following targets This is not done automatically by the `update build files` goal `poetry requirements` in BUILD add `name= poetry 09 57 31 02 ERROR You must manually add the `name=` field to the following targets This is not done automatically by the `update build files` goal `poetry requirements` in BUILD add `name= poetry 09 57 31 02 ERROR You must manually add the `name=` field to the following targets This is not done automatically by the `update build files` goal `poetry requirements` in BUILD add `name= poetry Updated BUILD Update references to targets generated by `poetry requirements```` `poetry requireents` doesn t have a field `name` no

rapid-crayon-8232 · Answer

Another question If I understood correctly if I want to set 2 lock files resolves `default` and `legacy` for example I can t have the `legacy` `poetry requirements` only containing the subset of dependencies that are different from the `default` one and for it to be resolved automatically with dep inference example getting requests from `default` pyproject toml and `numpy` from legacu pyproject toml when building the `python sources`

rapid-crayon-8232 · Answer

gt Yes I have abranch running with pants 2 10 I m note sure I understand this warnings thought when running ` pants update build files fix python macros` gt Fixed it was just an issue of when to toggle `use deprecated python macros` set to `true` before running the `update build files` and then set to `false` before fixing the files based on the errors

hundreds-father-404 · Answer

gt poetry requireents doesn t have a field name no It does if you set `use deprecated python macros = false` because `poetry requirements` becomes a proper target e g shows up in ` pants help targets`

hundreds-father-404 · Answer

gt I can t have the legacy poetry requirements only containing the subset of dependencies that are different from the default one and for it to be resolved automatically with dep inference Pants 2 11 will make this easier through a new parametrize feature ```poetry requirements name= poetry All entries from this pyproject toml will be usable by both resolves resolve=parametrize default legacy ``` Parametrize will create 2 `python requirement` targets for each entry one with the `default` resolve and another with the `legacy` resolve In the meantime you would create two distinct target generators here which have have roughly the same effect ```poetry requirements name= poetry default resolve= default poetry requirements name= poetry legacy resolve= legacy ``` Does that make sense

rapid-crayon-8232 · Answer

yeah makes perfect sense I m doing the same and working nice ^^ On another note we unfortunately had to disable the `generate lockfile` goal because we use some custom indexes apart from Pypi switched to shell commands to generate lock files ``` bin bash set eou pipefail POETRY LOCK DIR=$ POETRY LOCK DIR echo Resoving dependencies in $POETRY LOCK DIR cd $POETRY LOCK DIR poetry lock no update echo Exporting poetry lock to a requirements txt format poetry export without hashes f requirements txt o lock txt echo Done ``` ```experimental run shell command name= lock command= monorepo build support poetry lock sh experimental run shell command name= lock legacy command= POETRY LOCK DIR= monorepo legacy monorepo build support poetry lock sh ```

hundreds-father-404 · Answer

We re working on fixing that this week in 2 11 by switching to Pex for lockfile generation very close to being enabled slightly smiling face In the meantime did you see the flag ` python resolves generate lockfiles = false` so that you can still use `generate lockfiles` goal for tool lockfiles

rapid-crayon-8232 · Answer

gt In the meantime did you see the flag ` python resolves generate lockfiles = false` so that you can still use `generate lockfiles` goal for tool lockfiles Yep that s the what I did ^^ I m struggling a bit again with 3rd party deps If I can have your help pray I have the following BUILD file ```poetry requirements name= poetry module mapping= javaobj py3 javaobj overrides= fastapi dependencies poetry itsdangerous tensorboard dependencies poetry wheel poetry setuptools pytorch lightning dependencies poetry wheel poetry setuptools ``` When I try to run a pex binary that depends on the pytorch lightning I get the error ```17 50 01 57 INFO Completed Installing lock txt for the resolve `default` 17 50 02 57 INFO Completed Extracting 3 requirements to build cli pex from default lockfile pex pytorch lightning lt 2 0 0 gt =1 5 10 setuptools==59 5 0 wheel lt 0 38 0 gt =0 37 1 17 50 02 57 ERROR 1 Exception encountered ProcessExecutionFailure Process Extracting 3 requirements to build cli pex from default lockfile pex pytorch lightning lt 2 0 0 gt =1 5 10 setuptools==59 5 0 wheel lt 0 38 0 gt =0 37 1 failed with exit code 1 stdout stderr Failed to resolve requirements from PEX environment home obendidi cache pants named caches pex root unzipped pexes 257b2a9c82961e432f412c83fef2fb3ddbe8ed7b Needed cp38 cp38 manylinux 2 31 x86 64 compatible dependencies for 1 setuptools gt =41 0 0 Required by FingerprintedDistribution distribution=tensorboard 2 8 0 home obendidi cache pants named caches pex root installed wheels 8aac3016bf8745198e406b5a8ffad4f30454cc0b tensorboard 2 8 0 py3 none any whl fingerprint= 6d11e0a4273464e01a0d11f770f96302ddf6f720 But this pex had no setuptools distributions 2 wheel gt =0 26 Required by FingerprintedDistribution distribution=tensorboard 2 8 0 home obendidi cache pants named caches pex root installed wheels 8aac3016bf8745198e406b5a8ffad4f30454cc0b tensorboard 2 8 0 py3 none any whl fingerprint= 6d11e0a4273464e01a0d11f770f96302ddf6f720 But this pex had no wheel distributions 3 setuptools==59 5 0 Required by FingerprintedDistribution distribution=pytorch lightning 1 5 10 home obendidi cache pants named caches pex root installed wheels 1b178d9142a72ecbd1ca4ff6f71df50367efeed9 pytorch lightning 1 5 10 py3 none any whl fingerprint= d548245c67fb01318d04b12a747e35e3165f0f94 But this pex had no setuptools distributions 4 wheel lt 0 38 0 gt =0 37 1 But this pex had no wheel distributions Use ` no process cleanup` to preserve process chroots for inspection ```

hundreds-father-404 · Answer

My first thought would be if `default lock py` is missing setuptools amp wheel in it Note that this is failing after Pants has already installed default lock txt and then now Pants is trying to extract a subset of the PEX so that you have finer grained caching When you install default lock txt Pants Pex does so naively whatever you have it in Pants goes with No validation particularly because this lockfile is being manually managed

rapid-crayon-8232 · Answer

yes the setuptools and wheel are not included in the poetry export There is a fix but it is only available on poetry 1 2 0 alpha release I ll probably try to set them as `python requiremnt` directly to avoid this

hundreds-father-404 · Answer

gt I ll probably try to set them as python requiremnt directly to avoid this I don t think that will help you with manual lockfile management confused The best we can do is install the lock you give us We don t install anything extra you ask because that would be violating the premise of a lock that we only install the exact artifacts the lock contains I think that s actually good feedback for this question it s hard to get manual lockfile management right <https pantsbuild slack com archives C0D7TNJHL p1646844293306559>

hundreds-father-404 · Answer

I think your options might be 1 manually add `setuptools` and `wheel` to the lock These don t have transitive deps so you only should need to add those You will need ` hash` for this to work properly assuming you keep ` hash` for everything else 2 Use alpha Poetry 3 figure out if you don t need the `setuptools` and `wheel` deps like if you can use a new version of `pytorch lightning` 4 try out Pants 2 11 with Pex lockfile generation hopefully released at the end of this week or next week

rapid-crayon-8232 · Answer

alpha poetry it is then I ll try out pants 2 11 when the first rc release is out ^^