https://pantsbuild.org/ logo
r

refined-addition-53644

12/07/2021, 6:00 PM
I have another dependency issue. While running tests
pants
can't figure out dependencies generated by
poetry_requirements()
using
pyproject.toml
. I have listed all the relevant targets which have been generated by pants. These are just regular python packages like
pandas
and
numpy
. Not sure what I am doing wrong. There are BUILD files everywhere
h

happy-kitchen-89482

12/07/2021, 6:55 PM
Do you have the same requirements in multiple pyproject.toml files in multiple projects?
r

refined-addition-53644

12/07/2021, 8:35 PM
Yes, some python packages like pandas etc are common. But it doesn't complain about that but complains it can't find pandas at all.
h

happy-kitchen-89482

12/07/2021, 9:19 PM
Can you post the logs you're getting? If you have multiple targets providing the same symbols, dep inference cannot know which one you mean, but it should WARN about that.
šŸ‘ 1
cc @hundreds-father-404, who is working on support for multiple lockfiles
Presumably you have multiple pyproject.toml files for your current build tooling? Pants does not require that, you can have just one
requirements.txt
or
pyproject.toml
that lists the universe of allowed 3rdparty deps, and then Pants picks just the ones you actually need in any given situation.
r

refined-addition-53644

12/07/2021, 10:10 PM
Copy code
23:00:23.02 [ERROR] Completed: Run Pytest - src/pyfleet-vehicle-spec/test_api.py:tests failed (exit code 2).

==================================== ERRORS ====================================
_________________________ ERROR collecting test_api.py _________________________
ImportError while importing test module '/private/var/folders/9w/9_4n35r57d707zkrk86rvh9w0000gn/T/process-executionC6rjpx/src/pyfleet-vehicle-spec/test_api.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
/Users/developer/.pyenv/versions/3.7.9/lib/python3.7/importlib/__init__.py:127: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
src/pyfleet-vehicle-spec/test_api.py:5: in <module>
    import pandas as pd
E   ModuleNotFoundError: No module named 'pandas'
- generated xml file: /private/var/folders/9w/9_4n35r57d707zkrk86rvh9w0000gn/T/process-executionC6rjpx/src.pyfleet-vehicle-spec.test_api.py.tests.xml -
=========================== short test summary info ============================
ERROR src/pyfleet-vehicle-spec/test_api.py
!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!
1 error in 0.12s



š„‚ src/pyfleet-vehicle-spec/test_api.py:tests failed.
The BUILD file inside
src/pyfleet-vehicle-spec/
Copy code
poetry_requirements()

python_tests(
    name="tests",
)
I am trying to maintain separate
pyproject.toml
for each project inside
src
.
h

happy-kitchen-89482

12/07/2021, 10:28 PM
Can you post the Pants logs from the beginning of the run until that error?
r

refined-addition-53644

12/08/2021, 9:20 AM
That was almost complete. It was missing the pants command. Here this is all with the pants command
Copy code
./pants test src/pyfleet-vehicle-spec:tests
23:08:05.88 [ERROR] Completed: Run Pytest - src/pyfleet-vehicle-spec/test_api.py:tests failed (exit code 2).

==================================== ERRORS ====================================
_________________________ ERROR collecting test_api.py _________________________
ImportError while importing test module '/private/var/folders/9w/9_4n35r57d707zkrk86rvh9w0000gn/T/process-executionf1PMNO/src/pyfleet-vehicle-spec/test_api.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
/Users/developer/.pyenv/versions/3.7.9/lib/python3.7/importlib/__init__.py:127: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
src/pyfleet-vehicle-spec/test_api.py:5: in <module>
    import pandas as pd
E   ModuleNotFoundError: No module named 'pandas'
- generated xml file: /private/var/folders/9w/9_4n35r57d707zkrk86rvh9w0000gn/T/process-executionf1PMNO/src.pyfleet-vehicle-spec.test_api.py.tests.xml -
=========================== short test summary info ============================
ERROR src/pyfleet-vehicle-spec/test_api.py
!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!
1 error in 0.12s



š„‚ src/pyfleet-vehicle-spec/test_api.py:tests failed.
h

happy-kitchen-89482

12/08/2021, 9:40 PM
Hmm, I was hoping to see WARNings about multiple providers of the same symbol, but this appears to be not that
āž• 1
What does this show?
./pants dependencies src/pyfleet-vehicle-spec:tests
r

refined-addition-53644

12/09/2021, 12:29 PM
Earlier there was dependency on
src/pyfleet-vehicle-spec/test_api.py
which was actually not a test file but just some script starting with
test
. I have removed the
test_api.py
Now all tests related files live inside the
tests
directory inside
src/pyfleet-vehicle-spec
. Now it can't figure out any dependencies when I run . It's just blank screen.
Copy code
./pants dependencies src/pyfleet-vehicle-spec:tests
Then when I run
./pants test src/pyfleet-vehicle-spec:tests
, this is what I get
Copy code
./pants test src/pyfleet-vehicle-spec:tests  
13:17:25.61 [WARN] No applicable files or targets matched. The `test` goal works with these target types:

  * python_test

However, you only specified targets with these target types:

  * python_tests

Please specify relevant files and/or targets. Run `./pants filter --target-type=python_test ::` to find all applicable targets in your project, or run `./pants filter --target-type=python_test :: | xargs ./pants filedeps` to find all applicable files.
This
tests
target is defined in a BUILD file under
src/pyfleet-vehicle-spec
which looks like this
Copy code
poetry_requirements()


python_tests(
    name="tests",
)
Not sure how to define dependencies for
tests
as target now. It's this issue which I keep running into when we put stuff inside some directory which has further nested directories which don't depend on each other. All these directories have a BUILD file generated by
./pants tailor
This is the layout of
tests
directory
Copy code
tests
ā”œā”€ā”€ __init__.py
ā”œā”€ā”€ app/
ā”œā”€ā”€ authentication.py
ā”œā”€ā”€ BUILD
ā”œā”€ā”€ resources/
ā””ā”€ā”€ utils/
So after adding
sources=["tests/**/test_*.py"]
to
python_tests
, these are the dependencies
Copy code
./pants dependencies src/pyfleet-vehicle-spec:tests 
src/pyfleet-vehicle-spec/tests/app/routes/test_vehicle_specs.py:../../../tests
src/pyfleet-vehicle-spec/tests/test_panion_ds_vehicle_specs.py:../tests
src/pyfleet-vehicle-spec/tests/utils/test_nlp.py:../../tests
src/pyfleet-vehicle-spec/tests/utils/test_regex.py:../../tests
But I still get the
pandas
not found error
Copy code
/pants test src/pyfleet-vehicle-spec:tests        
13:55:26.15 [INFO] Completed: Run Pytest - src/pyfleet-vehicle-spec/tests/test_panion_ds_vehicle_specs.py:../tests succeeded.
13:55:26.86 [ERROR] Completed: Run Pytest - src/pyfleet-vehicle-spec/tests/__init__.py:../tests failed (exit code 5).

- generated xml file: /private/var/folders/9w/9_4n35r57d707zkrk86rvh9w0000gn/T/process-executionM2FBOj/src.pyfleet-vehicle-spec.tests.__init__.py@tests.xml -
no tests ran in 0.02s


13:55:26.86 [ERROR] Completed: Run Pytest - src/pyfleet-vehicle-spec/tests/app/routes/__init__.py:../../../tests failed (exit code 5).

- generated xml file: /private/var/folders/9w/9_4n35r57d707zkrk86rvh9w0000gn/T/process-executionEMEotU/src.pyfleet-vehicle-spec.tests.app.routes.__init__.py@@@tests.xml -
no tests ran in 0.02s


13:55:26.95 [ERROR] Completed: Run Pytest - src/pyfleet-vehicle-spec/tests/app/__init__.py:../../tests failed (exit code 5).

- generated xml file: /private/var/folders/9w/9_4n35r57d707zkrk86rvh9w0000gn/T/process-execution8FTEDZ/src.pyfleet-vehicle-spec.tests.app.__init__.py@@tests.xml -
no tests ran in 0.03s


13:55:27.02 [ERROR] Completed: Run Pytest - src/pyfleet-vehicle-spec/tests/utils/__init__.py:../../tests failed (exit code 5).

- generated xml file: /private/var/folders/9w/9_4n35r57d707zkrk86rvh9w0000gn/T/process-executionyb3PqD/src.pyfleet-vehicle-spec.tests.utils.__init__.py@@tests.xml -
no tests ran in 0.03s


13:55:27.43 [ERROR] Completed: Run Pytest - src/pyfleet-vehicle-spec/tests/utils/test_regex.py:../../tests failed (exit code 2).

==================================== ERRORS ====================================
__________________ ERROR collecting tests/utils/test_regex.py __________________
ImportError while importing test module '/private/var/folders/9w/9_4n35r57d707zkrk86rvh9w0000gn/T/process-executionBhV2TF/src/pyfleet-vehicle-spec/tests/utils/test_regex.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
/Users/developer/.pyenv/versions/3.7.9/lib/python3.7/importlib/__init__.py:127: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
src/pyfleet-vehicle-spec/tests/utils/test_regex.py:1: in <module>
    import numpy as np
E   ModuleNotFoundError: No module named 'numpy'
- generated xml file: /private/var/folders/9w/9_4n35r57d707zkrk86rvh9w0000gn/T/process-executionBhV2TF/src.pyfleet-vehicle-spec.tests.utils.test_regex.py@@tests.xml -
=========================== short test summary info ============================
ERROR src/pyfleet-vehicle-spec/tests/utils/test_regex.py
!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!
1 error in 0.30s


13:55:27.62 [ERROR] Completed: Run Pytest - src/pyfleet-vehicle-spec/tests/utils/test_nlp.py:../../tests failed (exit code 2).

==================================== ERRORS ====================================
___________________ ERROR collecting tests/utils/test_nlp.py ___________________
ImportError while importing test module '/private/var/folders/9w/9_4n35r57d707zkrk86rvh9w0000gn/T/process-executiondq4iJ5/src/pyfleet-vehicle-spec/tests/utils/test_nlp.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
/Users/developer/.pyenv/versions/3.7.9/lib/python3.7/importlib/__init__.py:127: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
src/pyfleet-vehicle-spec/tests/utils/test_nlp.py:4: in <module>
    import pandas as pd
(tox-env) āžœ  pyfleet git:(pyfleet_vehicle_spec) āœ— ./pants test src/pyfleet-vehicle-spec:tests
13:59:56.09 [INFO] Completed: Run Pytest - src/pyfleet-vehicle-spec/tests/test_panion_ds_vehicle_specs.py:../tests succeeded.
13:59:56.85 [ERROR] Completed: Run Pytest - src/pyfleet-vehicle-spec/tests/utils/test_regex.py:../../tests failed (exit code 2).

==================================== ERRORS ====================================
__________________ ERROR collecting tests/utils/test_regex.py __________________
ImportError while importing test module '/private/var/folders/9w/9_4n35r57d707zkrk86rvh9w0000gn/T/process-executionSv4gKd/src/pyfleet-vehicle-spec/tests/utils/test_regex.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
/Users/developer/.pyenv/versions/3.7.9/lib/python3.7/importlib/__init__.py:127: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
src/pyfleet-vehicle-spec/tests/utils/test_regex.py:1: in <module>
    import numpy as np
E   ModuleNotFoundError: No module named 'numpy'
- generated xml file: /private/var/folders/9w/9_4n35r57d707zkrk86rvh9w0000gn/T/process-executionSv4gKd/src.pyfleet-vehicle-spec.tests.utils.test_regex.py@@tests.xml -
=========================== short test summary info ============================
ERROR src/pyfleet-vehicle-spec/tests/utils/test_regex.py
!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!
1 error in 0.13s


13:59:57.19 [ERROR] Completed: Run Pytest - src/pyfleet-vehicle-spec/tests/utils/test_nlp.py:../../tests failed (exit code 2).

==================================== ERRORS ====================================
___________________ ERROR collecting tests/utils/test_nlp.py ___________________
ImportError while importing test module '/private/var/folders/9w/9_4n35r57d707zkrk86rvh9w0000gn/T/process-execution1qM560/src/pyfleet-vehicle-spec/tests/utils/test_nlp.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
/Users/developer/.pyenv/versions/3.7.9/lib/python3.7/importlib/__init__.py:127: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
src/pyfleet-vehicle-spec/tests/utils/test_nlp.py:4: in <module>
    import pandas as pd
E   ModuleNotFoundError: No module named 'pandas'
- generated xml file: /private/var/folders/9w/9_4n35r57d707zkrk86rvh9w0000gn/T/process-execution1qM560/src.pyfleet-vehicle-spec.tests.utils.test_nlp.py@@tests.xml -
=========================== short test summary info ============================
ERROR src/pyfleet-vehicle-spec/tests/utils/test_nlp.py
!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!
1 error in 0.37s


13:59:57.85 [ERROR] Completed: Run Pytest - src/pyfleet-vehicle-spec/tests/app/routes/test_vehicle_specs.py:../../../tests failed (exit code 2).

==================================== ERRORS ====================================
___________ ERROR collecting tests/app/routes/test_vehicle_specs.py ____________
ImportError while importing test module '/private/var/folders/9w/9_4n35r57d707zkrk86rvh9w0000gn/T/process-execution3rSptN/src/pyfleet-vehicle-spec/tests/app/routes/test_vehicle_specs.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
/Users/developer/.pyenv/versions/3.7.9/lib/python3.7/importlib/__init__.py:127: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
src/pyfleet-vehicle-spec/tests/app/routes/test_vehicle_specs.py:5: in <module>
    from pyfleet_vehicle_spec.main import authenticate, main
src/pyfleet-vehicle-spec/pyfleet_vehicle_spec/main.py:11: in <module>
    from pyfleet_vehicle_spec.app.routes.api import router as api_router
src/pyfleet-vehicle-spec/pyfleet_vehicle_spec/app/routes/api.py:2: in <module>
    from pyfleet_vehicle_spec.app.routes import configuration, healthcheck, vehicle_specs
src/pyfleet-vehicle-spec/pyfleet_vehicle_spec/app/routes/configuration.py:3: in <module>
    import pandas as pd
E   ModuleNotFoundError: No module named 'pandas'
- generated xml file: /private/var/folders/9w/9_4n35r57d707zkrk86rvh9w0000gn/T/process-execution3rSptN/src.pyfleet-vehicle-spec.tests.app.routes.test_vehicle_specs.py@@@tests.xml -
=========================== short test summary info ============================
ERROR src/pyfleet-vehicle-spec/tests/app/routes/test_vehicle_specs.py
!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!
1 error in 0.78s



āœ“ src/pyfleet-vehicle-spec/tests/test_panion_ds_vehicle_specs.py:../tests succeeded.
š„‚ src/pyfleet-vehicle-spec/tests/app/routes/test_vehicle_specs.py:../../../tests failed.
š„‚ src/pyfleet-vehicle-spec/tests/utils/test_nlp.py:../../tests failed.
š„‚ src/pyfleet-vehicle-spec/tests/utils/test_regex.py:../../tests failed.
So the issue seems to be originating from the multiple versions of the same 3rd party dependency. I had removed the project completely from the mono-repo which shared same dependencies and it worked fine. So what would be "easy" way to generate some common 3rd party dependencies files? This would mean every-time we add a new project we would have to manually take care of this. Any elegant solution?
h

happy-kitchen-89482

12/09/2021, 6:24 PM
Ah OK, so the issue was what I suspected - you have multiple targets providing the same dep, so Pants can't know which one you meant.
That, and also that the default sources for the
python_tests()
target are just the tests in the same dir, so you need to set
sources=
explicitly to cover subdirs, which you've discovered.
So what we have here is a mismatch between the legacy need to have a pyproject.toml per-project (because python tooling such as poetry has no intrinsic monorepo support) and the "Pants way", where you can have a single set of requirements for the entire repo, and pants subsets them for you.
So, important question - do you have conflicting versions of deps, or could you in theory have one consistent set of requirements for the entire repo?
This will help determine the easiest way to set things up
r

refined-addition-53644

12/09/2021, 6:27 PM
In theory we can force to keep one version of a given dependency. At the moment, we don't have any requirement where we need to maintain separate versions.
h

happy-kitchen-89482

12/09/2021, 8:31 PM
Great, in that case having one global requirements.txt or pyproject.toml (the former is probably easier) will work
So the question I guess is how to create one
r

refined-addition-53644

12/09/2021, 9:17 PM
Create, maintain/update
Whether it's changing version for a pre-existing or adding a new common dependency
h

happy-kitchen-89482

12/09/2021, 11:08 PM
Well, updating seems easy enough, you just edit the global requirements.txt
So I'm not sure I understand the question?
r

refined-addition-53644

12/09/2021, 11:38 PM
I was just adding further requirement to "how to create". But for now I suppose it's fine just to figure out how to create one
h

happy-kitchen-89482

12/10/2021, 1:27 AM
So the initial one I guess you could create by manually copy-pasting the requirements from your various pyproject.toml into a requirements.txt, sorting alphabetically, and removing the dups
10 Views