I have another dependency issue While running tests `pants`

Question

I have another dependency issue While running tests `pants` can t figure out dependencies generated by `poetry requirements ` using `pyproject toml` I have listed all the relevant targets which have been generated by pants These are just regular python packages like `pandas` and `numpy` Not sure what I am doing wrong There are BUILD files everywhere

happy-kitchen-89482 · Answer

Do you have the same requirements in multiple pyproject toml files in multiple projects

refined-addition-53644 · Answer

Yes some python packages like pandas etc are common But it doesn t complain about that but complains it can t find pandas at all

happy-kitchen-89482 · Answer

Can you post the logs you re getting If you have multiple targets providing the same symbols dep inference cannot know which one you mean but it should WARN about that

happy-kitchen-89482 · Answer

cc < hundreds father 404> who is working on support for multiple lockfiles

happy-kitchen-89482 · Answer

Presumably you have multiple pyproject toml files for your current build tooling Pants does not require that you can have just one `requirements txt` or `pyproject toml` that lists the universe of allowed 3rdparty deps and then Pants picks just the ones you actually need in any given situation

hundreds-father-404 · Answer

indeed explained at <https www pantsbuild org docs python third party dependencies warning multiple versions of the same dependency>

refined-addition-53644 · Answer

```23 00 23 02 ERROR Completed Run Pytest src pyfleet vehicle spec test api py tests failed exit code 2 ==================================== ERRORS ==================================== ERROR collecting test api py ImportError while importing test module private var folders 9w 9 4n35r57d707zkrk86rvh9w0000gn T process executionC6rjpx src pyfleet vehicle spec test api py Hint make sure your test modules packages have valid Python names Traceback Users developer pyenv versions 3 7 9 lib python3 7 importlib init py 127 in import module return bootstrap gcd import name level package level src pyfleet vehicle spec test api py 5 in lt module gt import pandas as pd E ModuleNotFoundError No module named pandas generated xml file private var folders 9w 9 4n35r57d707zkrk86rvh9w0000gn T process executionC6rjpx src pyfleet vehicle spec test api py tests xml =========================== short test summary info ============================ ERROR src pyfleet vehicle spec test api py Interrupted 1 error during collection 1 error in 0 12s src pyfleet vehicle spec test api py tests failed ``` The BUILD file inside `src pyfleet vehicle spec ` ```poetry requirements python tests name= tests ```

refined-addition-53644 · Answer

I am trying to maintain separate `pyproject toml` for each project inside `src`

happy-kitchen-89482 · Answer

Can you post the Pants logs from the beginning of the run until that error

refined-addition-53644 · Answer

That was almost complete It was missing the pants command Here this is all with the pants command ``` pants test src pyfleet vehicle spec tests 23 08 05 88 ERROR Completed Run Pytest src pyfleet vehicle spec test api py tests failed exit code 2 ==================================== ERRORS ==================================== ERROR collecting test api py ImportError while importing test module private var folders 9w 9 4n35r57d707zkrk86rvh9w0000gn T process executionf1PMNO src pyfleet vehicle spec test api py Hint make sure your test modules packages have valid Python names Traceback Users developer pyenv versions 3 7 9 lib python3 7 importlib init py 127 in import module return bootstrap gcd import name level package level src pyfleet vehicle spec test api py 5 in lt module gt import pandas as pd E ModuleNotFoundError No module named pandas generated xml file private var folders 9w 9 4n35r57d707zkrk86rvh9w0000gn T process executionf1PMNO src pyfleet vehicle spec test api py tests xml =========================== short test summary info ============================ ERROR src pyfleet vehicle spec test api py Interrupted 1 error during collection 1 error in 0 12s src pyfleet vehicle spec test api py tests failed ```

happy-kitchen-89482 · Answer

Hmm I was hoping to see WARNings about multiple providers of the same symbol but this appears to be not that

happy-kitchen-89482 · Answer

What does this show ` pants dependencies src pyfleet vehicle spec tests`

refined-addition-53644 · Answer

Earlier there was dependency on `src pyfleet vehicle spec test api py` which was actually not a test file but just some script starting with `test` I have removed the `test api py` Now all tests related files live inside the `tests` directory inside `src pyfleet vehicle spec` Now it can t figure out any dependencies when I run It s just blank screen ``` pants dependencies src pyfleet vehicle spec tests``` Then when I run ` pants test src pyfleet vehicle spec tests` this is what I get ``` pants test src pyfleet vehicle spec tests 13 17 25 61 WARN No applicable files or targets matched The `test` goal works with these target types python test However you only specified targets with these target types python tests Please specify relevant files and or targets Run ` pants filter target type=python test ` to find all applicable targets in your project or run ` pants filter target type=python test | xargs pants filedeps` to find all applicable files ``` This `tests` target is defined in a BUILD file under `src pyfleet vehicle spec` which looks like this ```poetry requirements python tests name= tests ``` Not sure how to define dependencies for `tests` as target now It s this issue which I keep running into when we put stuff inside some directory which has further nested directories which don t depend on each other All these directories have a BUILD file generated by ` pants tailor` This is the layout of `tests` directory ```tests init py app authentication py BUILD resources utils ```

refined-addition-53644 · Answer

So after adding `sources= tests test py ` to `python tests` these are the dependencies ``` pants dependencies src pyfleet vehicle spec tests src pyfleet vehicle spec tests app routes test vehicle specs py tests src pyfleet vehicle spec tests test panion ds vehicle specs py tests src pyfleet vehicle spec tests utils test nlp py tests src pyfleet vehicle spec tests utils test regex py tests``` But I still get the `pandas` not found error ``` pants test src pyfleet vehicle spec tests 13 55 26 15 INFO Completed Run Pytest src pyfleet vehicle spec tests test panion ds vehicle specs py tests succeeded 13 55 26 86 ERROR Completed Run Pytest src pyfleet vehicle spec tests init py tests failed exit code 5 generated xml file private var folders 9w 9 4n35r57d707zkrk86rvh9w0000gn T process executionM2FBOj src pyfleet vehicle spec tests init py tests xml no tests ran in 0 02s 13 55 26 86 ERROR Completed Run Pytest src pyfleet vehicle spec tests app routes init py tests failed exit code 5 generated xml file private var folders 9w 9 4n35r57d707zkrk86rvh9w0000gn T process executionEMEotU src pyfleet vehicle spec tests app routes init py tests xml no tests ran in 0 02s 13 55 26 95 ERROR Completed Run Pytest src pyfleet vehicle spec tests app init py tests failed exit code 5 generated xml file private var folders 9w 9 4n35r57d707zkrk86rvh9w0000gn T process execution8FTEDZ src pyfleet vehicle spec tests app init py tests xml no tests ran in 0 03s 13 55 27 02 ERROR Completed Run Pytest src pyfleet vehicle spec tests utils init py tests failed exit code 5 generated xml file private var folders 9w 9 4n35r57d707zkrk86rvh9w0000gn T process executionyb3PqD src pyfleet vehicle spec tests utils init py tests xml no tests ran in 0 03s 13 55 27 43 ERROR Completed Run Pytest src pyfleet vehicle spec tests utils test regex py tests failed exit code 2 ==================================== ERRORS ==================================== ERROR collecting tests utils test regex py ImportError while importing test module private var folders 9w 9 4n35r57d707zkrk86rvh9w0000gn T process executionBhV2TF src pyfleet vehicle spec tests utils test regex py Hint make sure your test modules packages have valid Python names Traceback Users developer pyenv versions 3 7 9 lib python3 7 importlib init py 127 in import module return bootstrap gcd import name level package level src pyfleet vehicle spec tests utils test regex py 1 in lt module gt import numpy as np E ModuleNotFoundError No module named numpy generated xml file private var folders 9w 9 4n35r57d707zkrk86rvh9w0000gn T process executionBhV2TF src pyfleet vehicle spec tests utils test regex py tests xml =========================== short test summary info ============================ ERROR src pyfleet vehicle spec tests utils test regex py Interrupted 1 error during collection 1 error in 0 30s 13 55 27 62 ERROR Completed Run Pytest src pyfleet vehicle spec tests utils test nlp py tests failed exit code 2 ==================================== ERRORS ==================================== ERROR collecting tests utils test nlp py ImportError while importing test module private var folders 9w 9 4n35r57d707zkrk86rvh9w0000gn T process executiondq4iJ5 src pyfleet vehicle spec tests utils test nlp py Hint make sure your test modules packages have valid Python names Traceback Users developer pyenv versions 3 7 9 lib python3 7 importlib init py 127 in import module return bootstrap gcd import name level package level src pyfleet vehicle spec tests utils test nlp py 4 in lt module gt import pandas as pd tox env pyfleet git pyfleet vehicle spec pants test src pyfleet vehicle spec tests 13 59 56 09 INFO Completed Run Pytest src pyfleet vehicle spec tests test panion ds vehicle specs py tests succeeded 13 59 56 85 ERROR Completed Run Pytest src pyfleet vehicle spec tests utils test regex py tests failed exit code 2 ==================================== ERRORS ==================================== ERROR collecting tests utils test regex py ImportError while importing test module private var folders 9w 9 4n35r57d707zkrk86rvh9w0000gn T process executionSv4gKd src pyfleet vehicle spec tests utils test regex py Hint make sure your test modules packages have valid Python names Traceback Users developer pyenv versions 3 7 9 lib python3 7 importlib init py 127 in import module return bootstrap gcd import name level package level src pyfleet vehicle spec tests utils test regex py 1 in lt module gt import numpy as np E ModuleNotFoundError No module named numpy generated xml file private var folders 9w 9 4n35r57d707zkrk86rvh9w0000gn T process executionSv4gKd src pyfleet vehicle spec tests utils test regex py tests xml =========================== short test summary info ============================ ERROR src pyfleet vehicle spec tests utils test regex py Interrupted 1 error during collection 1 error in 0 13s 13 59 57 19 ERROR Completed Run Pytest src pyfleet vehicle spec tests utils test nlp py tests failed exit code 2 ==================================== ERRORS ==================================== ERROR collecting tests utils test nlp py ImportError while importing test module private var folders 9w 9 4n35r57d707zkrk86rvh9w0000gn T process execution1qM560 src pyfleet vehicle spec tests utils test nlp py Hint make sure your test modules packages have valid Python names Traceback Users developer pyenv versions 3 7 9 lib python3 7 importlib init py 127 in import module return bootstrap gcd import name level package level src pyfleet vehicle spec tests utils test nlp py 4 in lt module gt import pandas as pd E ModuleNotFoundError No module named pandas generated xml file private var folders 9w 9 4n35r57d707zkrk86rvh9w0000gn T process execution1qM560 src pyfleet vehicle spec tests utils test nlp py tests xml =========================== short test summary info ============================ ERROR src pyfleet vehicle spec tests utils test nlp py Interrupted 1 error during collection 1 error in 0 37s 13 59 57 85 ERROR Completed Run Pytest src pyfleet vehicle spec tests app routes test vehicle specs py tests failed exit code 2 ==================================== ERRORS ==================================== ERROR collecting tests app routes test vehicle specs py ImportError while importing test module private var folders 9w 9 4n35r57d707zkrk86rvh9w0000gn T process execution3rSptN src pyfleet vehicle spec tests app routes test vehicle specs py Hint make sure your test modules packages have valid Python names Traceback Users developer pyenv versions 3 7 9 lib python3 7 importlib init py 127 in import module return bootstrap gcd import name level package level src pyfleet vehicle spec tests app routes test vehicle specs py 5 in lt module gt from pyfleet vehicle spec main import authenticate main src pyfleet vehicle spec pyfleet vehicle spec main py 11 in lt module gt from pyfleet vehicle spec app routes api import router as api router src pyfleet vehicle spec pyfleet vehicle spec app routes api py 2 in lt module gt from pyfleet vehicle spec app routes import configuration healthcheck vehicle specs src pyfleet vehicle spec pyfleet vehicle spec app routes configuration py 3 in lt module gt import pandas as pd E ModuleNotFoundError No module named pandas generated xml file private var folders 9w 9 4n35r57d707zkrk86rvh9w0000gn T process execution3rSptN src pyfleet vehicle spec tests app routes test vehicle specs py tests xml =========================== short test summary info ============================ ERROR src pyfleet vehicle spec tests app routes test vehicle specs py Interrupted 1 error during collection 1 error in 0 78s src pyfleet vehicle spec tests test panion ds vehicle specs py tests succeeded src pyfleet vehicle spec tests app routes test vehicle specs py tests failed src pyfleet vehicle spec tests utils test nlp py tests failed src pyfleet vehicle spec tests utils test regex py tests failed ```

refined-addition-53644 · Answer

So the issue seems to be originating from the multiple versions of the same 3rd party dependency I had removed the project completely from the mono repo which shared same dependencies and it worked fine So what would be easy way to generate some common 3rd party dependencies files This would mean every time we add a new project we would have to manually take care of this Any elegant solution

happy-kitchen-89482 · Answer

Ah OK so the issue was what I suspected you have multiple targets providing the same dep so Pants can t know which one you meant

happy-kitchen-89482 · Answer

That and also that the default sources for the `python tests ` target are just the tests in the same dir so you need to set `sources=` explicitly to cover subdirs which you ve discovered

happy-kitchen-89482 · Answer

So what we have here is a mismatch between the legacy need to have a pyproject toml per project because python tooling such as poetry has no intrinsic monorepo support and the Pants way where you can have a single set of requirements for the entire repo and pants subsets them for you

happy-kitchen-89482 · Answer

So important question do you have conflicting versions of deps or could you in theory have one consistent set of requirements for the entire repo

happy-kitchen-89482 · Answer

This will help determine the easiest way to set things up

refined-addition-53644 · Answer

In theory we can force to keep one version of a given dependency At the moment we don t have any requirement where we need to maintain separate versions

happy-kitchen-89482 · Answer

Great in that case having one global requirements txt or pyproject toml the former is probably easier will work

happy-kitchen-89482 · Answer

So the question I guess is how to create one

refined-addition-53644 · Answer

Create maintain update

refined-addition-53644 · Answer

Whether it s changing version for a pre existing or adding a new common dependency

happy-kitchen-89482 · Answer

Well updating seems easy enough you just edit the global requirements txt

happy-kitchen-89482 · Answer

So I m not sure I understand the question

refined-addition-53644 · Answer

I was just adding further requirement to how to create But for now I suppose it s fine just to figure out how to create one

happy-kitchen-89482 · Answer

So the initial one I guess you could create by manually copy pasting the requirements from your various pyproject toml into a requirements txt sorting alphabetically and removing the dups