https://pantsbuild.org/ logo
Join Slack
Powered by
I have another dependency issue. While running tes...
# general
r
I have another dependency issue. While running tests 
pants
can't figure out dependencies generated by 
poetry_requirements()
using 
pyproject.toml
. I have listed all the relevant targets which have been generated by pants. These are just regular python packages like 
pandas
and 
numpy
. Not sure what I am doing wrong. There are BUILD files everywhere
h
Do you have the same requirements in multiple pyproject.toml files in multiple projects?
r
Yes, some python packages like pandas etc are common. But it doesn't complain about that but complains it can't find pandas at all.
h
Can you post the logs you're getting? If you have multiple targets providing the same symbols, dep inference cannot know which one you mean, but it should WARN about that.
šŸ‘ 1
cc @hundreds-father-404, who is working on support for multiple lockfiles
Presumably you have multiple pyproject.toml files for your current build tooling? Pants does not require that, you can have just one 
requirements.txt
or 
pyproject.toml
that lists the universe of allowed 3rdparty deps, and then Pants picks just the ones you actually need in any given situation.
h
r
Copy code
23:00:23.02 [ERROR] Completed: Run Pytest - src/pyfleet-vehicle-spec/test_api.py:tests failed (exit code 2).

==================================== ERRORS ====================================
_________________________ ERROR collecting test_api.py _________________________
ImportError while importing test module '/private/var/folders/9w/9_4n35r57d707zkrk86rvh9w0000gn/T/process-executionC6rjpx/src/pyfleet-vehicle-spec/test_api.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
/Users/developer/.pyenv/versions/3.7.9/lib/python3.7/importlib/__init__.py:127: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
src/pyfleet-vehicle-spec/test_api.py:5: in <module>
    import pandas as pd
E   ModuleNotFoundError: No module named 'pandas'
- generated xml file: /private/var/folders/9w/9_4n35r57d707zkrk86rvh9w0000gn/T/process-executionC6rjpx/src.pyfleet-vehicle-spec.test_api.py.tests.xml -
=========================== short test summary info ============================
ERROR src/pyfleet-vehicle-spec/test_api.py
!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!
1 error in 0.12s



š„‚ src/pyfleet-vehicle-spec/test_api.py:tests failed.
The BUILD file inside 
src/pyfleet-vehicle-spec/
Copy code
poetry_requirements()

python_tests(
    name="tests",
)
I am trying to maintain separate 
pyproject.toml
for each project inside 
src
.
h
Can you post the Pants logs from the beginning of the run until that error?
r
That was almost complete. It was missing the pants command. Here this is all with the pants command
Copy code
./pants test src/pyfleet-vehicle-spec:tests
23:08:05.88 [ERROR] Completed: Run Pytest - src/pyfleet-vehicle-spec/test_api.py:tests failed (exit code 2).

==================================== ERRORS ====================================
_________________________ ERROR collecting test_api.py _________________________
ImportError while importing test module '/private/var/folders/9w/9_4n35r57d707zkrk86rvh9w0000gn/T/process-executionf1PMNO/src/pyfleet-vehicle-spec/test_api.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
/Users/developer/.pyenv/versions/3.7.9/lib/python3.7/importlib/__init__.py:127: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
src/pyfleet-vehicle-spec/test_api.py:5: in <module>
    import pandas as pd
E   ModuleNotFoundError: No module named 'pandas'
- generated xml file: /private/var/folders/9w/9_4n35r57d707zkrk86rvh9w0000gn/T/process-executionf1PMNO/src.pyfleet-vehicle-spec.test_api.py.tests.xml -
=========================== short test summary info ============================
ERROR src/pyfleet-vehicle-spec/test_api.py
!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!
1 error in 0.12s



š„‚ src/pyfleet-vehicle-spec/test_api.py:tests failed.
h
Hmm, I was hoping to see WARNings about multiple providers of the same symbol, but this appears to be not that
āž• 1
What does this show? 
./pants dependencies src/pyfleet-vehicle-spec:tests
r
Earlier there was dependency on 
src/pyfleet-vehicle-spec/test_api.py
which was actually not a test file but just some script starting with 
test
. I have removed the 
test_api.py
Now all tests related files live inside the 
tests
directory inside 
src/pyfleet-vehicle-spec
. Now it can't figure out any dependencies when I run . It's just blank screen.
Copy code
./pants dependencies src/pyfleet-vehicle-spec:tests
Then when I run 
./pants test src/pyfleet-vehicle-spec:tests
, this is what I get
Copy code
./pants test src/pyfleet-vehicle-spec:tests  
13:17:25.61 [WARN] No applicable files or targets matched. The `test` goal works with these target types:

  * python_test

However, you only specified targets with these target types:

  * python_tests

Please specify relevant files and/or targets. Run `./pants filter --target-type=python_test ::` to find all applicable targets in your project, or run `./pants filter --target-type=python_test :: | xargs ./pants filedeps` to find all applicable files.
This 
tests
target is defined in a BUILD file under 
src/pyfleet-vehicle-spec
which looks like this
Copy code
poetry_requirements()


python_tests(
    name="tests",
)
Not sure how to define dependencies for 
tests
as target now. It's this issue which I keep running into when we put stuff inside some directory which has further nested directories which don't depend on each other. All these directories have a BUILD file generated by 
./pants tailor
This is the layout of 
tests
directory
Copy code
tests
ā”œā”€ā”€ __init__.py
ā”œā”€ā”€ app/
ā”œā”€ā”€ authentication.py
ā”œā”€ā”€ BUILD
ā”œā”€ā”€ resources/
ā””ā”€ā”€ utils/
So after adding 
sources=["tests/**/test_*.py"]
to 
python_tests
, these are the dependencies
Copy code
./pants dependencies src/pyfleet-vehicle-spec:tests 
src/pyfleet-vehicle-spec/tests/app/routes/test_vehicle_specs.py:../../../tests
src/pyfleet-vehicle-spec/tests/test_panion_ds_vehicle_specs.py:../tests
src/pyfleet-vehicle-spec/tests/utils/test_nlp.py:../../tests
src/pyfleet-vehicle-spec/tests/utils/test_regex.py:../../tests
But I still get the 
pandas
not found error
Copy code
/pants test src/pyfleet-vehicle-spec:tests        
13:55:26.15 [INFO] Completed: Run Pytest - src/pyfleet-vehicle-spec/tests/test_panion_ds_vehicle_specs.py:../tests succeeded.
13:55:26.86 [ERROR] Completed: Run Pytest - src/pyfleet-vehicle-spec/tests/__init__.py:../tests failed (exit code 5).

- generated xml file: /private/var/folders/9w/9_4n35r57d707zkrk86rvh9w0000gn/T/process-executionM2FBOj/src.pyfleet-vehicle-spec.tests.__init__.py@tests.xml -
no tests ran in 0.02s


13:55:26.86 [ERROR] Completed: Run Pytest - src/pyfleet-vehicle-spec/tests/app/routes/__init__.py:../../../tests failed (exit code 5).

- generated xml file: /private/var/folders/9w/9_4n35r57d707zkrk86rvh9w0000gn/T/process-executionEMEotU/src.pyfleet-vehicle-spec.tests.app.routes.__init__.py@@@tests.xml -
no tests ran in 0.02s


13:55:26.95 [ERROR] Completed: Run Pytest - src/pyfleet-vehicle-spec/tests/app/__init__.py:../../tests failed (exit code 5).

- generated xml file: /private/var/folders/9w/9_4n35r57d707zkrk86rvh9w0000gn/T/process-execution8FTEDZ/src.pyfleet-vehicle-spec.tests.app.__init__.py@@tests.xml -
no tests ran in 0.03s


13:55:27.02 [ERROR] Completed: Run Pytest - src/pyfleet-vehicle-spec/tests/utils/__init__.py:../../tests failed (exit code 5).

- generated xml file: /private/var/folders/9w/9_4n35r57d707zkrk86rvh9w0000gn/T/process-executionyb3PqD/src.pyfleet-vehicle-spec.tests.utils.__init__.py@@tests.xml -
no tests ran in 0.03s


13:55:27.43 [ERROR] Completed: Run Pytest - src/pyfleet-vehicle-spec/tests/utils/test_regex.py:../../tests failed (exit code 2).

==================================== ERRORS ====================================
__________________ ERROR collecting tests/utils/test_regex.py __________________
ImportError while importing test module '/private/var/folders/9w/9_4n35r57d707zkrk86rvh9w0000gn/T/process-executionBhV2TF/src/pyfleet-vehicle-spec/tests/utils/test_regex.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
/Users/developer/.pyenv/versions/3.7.9/lib/python3.7/importlib/__init__.py:127: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
src/pyfleet-vehicle-spec/tests/utils/test_regex.py:1: in <module>
    import numpy as np
E   ModuleNotFoundError: No module named 'numpy'
- generated xml file: /private/var/folders/9w/9_4n35r57d707zkrk86rvh9w0000gn/T/process-executionBhV2TF/src.pyfleet-vehicle-spec.tests.utils.test_regex.py@@tests.xml -
=========================== short test summary info ============================
ERROR src/pyfleet-vehicle-spec/tests/utils/test_regex.py
!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!
1 error in 0.30s


13:55:27.62 [ERROR] Completed: Run Pytest - src/pyfleet-vehicle-spec/tests/utils/test_nlp.py:../../tests failed (exit code 2).

==================================== ERRORS ====================================
___________________ ERROR collecting tests/utils/test_nlp.py ___________________
ImportError while importing test module '/private/var/folders/9w/9_4n35r57d707zkrk86rvh9w0000gn/T/process-executiondq4iJ5/src/pyfleet-vehicle-spec/tests/utils/test_nlp.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
/Users/developer/.pyenv/versions/3.7.9/lib/python3.7/importlib/__init__.py:127: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
src/pyfleet-vehicle-spec/tests/utils/test_nlp.py:4: in <module>
    import pandas as pd
(tox-env) āžœ  pyfleet git:(pyfleet_vehicle_spec) āœ— ./pants test src/pyfleet-vehicle-spec:tests
13:59:56.09 [INFO] Completed: Run Pytest - src/pyfleet-vehicle-spec/tests/test_panion_ds_vehicle_specs.py:../tests succeeded.
13:59:56.85 [ERROR] Completed: Run Pytest - src/pyfleet-vehicle-spec/tests/utils/test_regex.py:../../tests failed (exit code 2).

==================================== ERRORS ====================================
__________________ ERROR collecting tests/utils/test_regex.py __________________
ImportError while importing test module '/private/var/folders/9w/9_4n35r57d707zkrk86rvh9w0000gn/T/process-executionSv4gKd/src/pyfleet-vehicle-spec/tests/utils/test_regex.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
/Users/developer/.pyenv/versions/3.7.9/lib/python3.7/importlib/__init__.py:127: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
src/pyfleet-vehicle-spec/tests/utils/test_regex.py:1: in <module>
    import numpy as np
E   ModuleNotFoundError: No module named 'numpy'
- generated xml file: /private/var/folders/9w/9_4n35r57d707zkrk86rvh9w0000gn/T/process-executionSv4gKd/src.pyfleet-vehicle-spec.tests.utils.test_regex.py@@tests.xml -
=========================== short test summary info ============================
ERROR src/pyfleet-vehicle-spec/tests/utils/test_regex.py
!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!
1 error in 0.13s


13:59:57.19 [ERROR] Completed: Run Pytest - src/pyfleet-vehicle-spec/tests/utils/test_nlp.py:../../tests failed (exit code 2).

==================================== ERRORS ====================================
___________________ ERROR collecting tests/utils/test_nlp.py ___________________
ImportError while importing test module '/private/var/folders/9w/9_4n35r57d707zkrk86rvh9w0000gn/T/process-execution1qM560/src/pyfleet-vehicle-spec/tests/utils/test_nlp.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
/Users/developer/.pyenv/versions/3.7.9/lib/python3.7/importlib/__init__.py:127: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
src/pyfleet-vehicle-spec/tests/utils/test_nlp.py:4: in <module>
    import pandas as pd
E   ModuleNotFoundError: No module named 'pandas'
- generated xml file: /private/var/folders/9w/9_4n35r57d707zkrk86rvh9w0000gn/T/process-execution1qM560/src.pyfleet-vehicle-spec.tests.utils.test_nlp.py@@tests.xml -
=========================== short test summary info ============================
ERROR src/pyfleet-vehicle-spec/tests/utils/test_nlp.py
!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!
1 error in 0.37s


13:59:57.85 [ERROR] Completed: Run Pytest - src/pyfleet-vehicle-spec/tests/app/routes/test_vehicle_specs.py:../../../tests failed (exit code 2).

==================================== ERRORS ====================================
___________ ERROR collecting tests/app/routes/test_vehicle_specs.py ____________
ImportError while importing test module '/private/var/folders/9w/9_4n35r57d707zkrk86rvh9w0000gn/T/process-execution3rSptN/src/pyfleet-vehicle-spec/tests/app/routes/test_vehicle_specs.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
/Users/developer/.pyenv/versions/3.7.9/lib/python3.7/importlib/__init__.py:127: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
src/pyfleet-vehicle-spec/tests/app/routes/test_vehicle_specs.py:5: in <module>
    from pyfleet_vehicle_spec.main import authenticate, main
src/pyfleet-vehicle-spec/pyfleet_vehicle_spec/main.py:11: in <module>
    from pyfleet_vehicle_spec.app.routes.api import router as api_router
src/pyfleet-vehicle-spec/pyfleet_vehicle_spec/app/routes/api.py:2: in <module>
    from pyfleet_vehicle_spec.app.routes import configuration, healthcheck, vehicle_specs
src/pyfleet-vehicle-spec/pyfleet_vehicle_spec/app/routes/configuration.py:3: in <module>
    import pandas as pd
E   ModuleNotFoundError: No module named 'pandas'
- generated xml file: /private/var/folders/9w/9_4n35r57d707zkrk86rvh9w0000gn/T/process-execution3rSptN/src.pyfleet-vehicle-spec.tests.app.routes.test_vehicle_specs.py@@@tests.xml -
=========================== short test summary info ============================
ERROR src/pyfleet-vehicle-spec/tests/app/routes/test_vehicle_specs.py
!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!
1 error in 0.78s



āœ“ src/pyfleet-vehicle-spec/tests/test_panion_ds_vehicle_specs.py:../tests succeeded.
š„‚ src/pyfleet-vehicle-spec/tests/app/routes/test_vehicle_specs.py:../../../tests failed.
š„‚ src/pyfleet-vehicle-spec/tests/utils/test_nlp.py:../../tests failed.
š„‚ src/pyfleet-vehicle-spec/tests/utils/test_regex.py:../../tests failed.
So the issue seems to be originating from the multiple versions of the same 3rd party dependency. I had removed the project completely from the mono-repo which shared same dependencies and it worked fine. So what would be "easy" way to generate some common 3rd party dependencies files? This would mean every-time we add a new project we would have to manually take care of this. Any elegant solution?
h
Ah OK, so the issue was what I suspected - you have multiple targets providing the same dep, so Pants can't know which one you meant.
That, and also that the default sources for the 
python_tests()
target are just the tests in the same dir, so you need to set 
sources=
explicitly to cover subdirs, which you've discovered.
So what we have here is a mismatch between the legacy need to have a pyproject.toml per-project (because python tooling such as poetry has no intrinsic monorepo support) and the "Pants way", where you can have a single set of requirements for the entire repo, and pants subsets them for you.
So, important question - do you have conflicting versions of deps, or could you in theory have one consistent set of requirements for the entire repo?
This will help determine the easiest way to set things up
r
In theory we can force to keep one version of a given dependency. At the moment, we don't have any requirement where we need to maintain separate versions.
h
Great, in that case having one global requirements.txt or pyproject.toml (the former is probably easier) will work
So the question I guess is how to create one
r
Create, maintain/update
Whether it's changing version for a pre-existing or adding a new common dependency
h
Well, updating seems easy enough, you just edit the global requirements.txt
So I'm not sure I understand the question?
r
I was just adding further requirement to "how to create". But for now I suppose it's fine just to figure out how to create one
h
So the initial one I guess you could create by manually copy-pasting the requirements from your various pyproject.toml into a requirements.txt, sorting alphabetically, and removing the dups
11 Views
Open in Slack
PreviousNext
Powered by