https://pantsbuild.org/ logo
Join Slack
Powered by
I’ve got a `[source]` section like this in my `pan...
# general
c
I’ve got a 
[source]
section like this in my 
pants.toml
file:
Copy code
[source]
root_patterns = [
    "/project/app",
    "tests"
]
But my unit test code can’t find my project modules to import, things like this in the test code:
Copy code
from . import app
How do I configure pants so test code can import project modules?
w
what does your code layout look like?
source roots “cut off” a prefix of the filename, which should leave behind only the portion that you’d like to import
you can think of them as roughly equivalent to PYTHONPATH entries
c
Shortened version:
Copy code
project
   +-- app
         +-- auth
               +-- routers.py
tests
   +-- auth
         +-- routers.py
project and tests are siblings
w
got it. so i think that your source roots are probably fine, but i expect that you are getting lots of “ambiguity” warnings…?
the issue with the layout + source roots that you have is that if you go to import 
auth.routers
… there are two places to do that from
because if you chop your source roots off those paths, then you end up with 
auth/routers.py
twice
did you mean to chop less off of the 
projects/app/auth/routers.py
path, so that it was importable as 
app.auth.routers
?
c
I’m not sure I’m following. Running 
./pants test ::
gives me these errors:
Copy code
==================================== ERRORS ====================================
_________________ ERROR collecting tests/auth/test_routers.py __________________
ImportError while importing test module '/private/var/folders/5q/xbnx98wd4g12xd_60swx7yp00000gn/T/pants-sandbox-PkCA2T/tests/auth/test_routers.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
/opt/homebrew/Cellar/python@3.9/3.9.15/Frameworks/Python.framework/Versions/3.9/lib/python3.9/importlib/__init__.py:127: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
tests/auth/test_routers.py:9: in <module>
    from <http://project.app|project.app> import app
E   ModuleNotFoundError: No module named 'project'
- generated xml file: /private/var/folders/5q/xbnx98wd4g12xd_60swx7yp00000gn/T/pants-sandbox-PkCA2T/tests.auth.test_routers.py.tests.xml -
=========================== short test summary info ============================
ERROR tests/auth/test_routers.py
!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!
=============================== 1 error in 0.18s ===============================
with this line in my test file:
Copy code
from <http://project.app|project.app> import app
At the moment I’m trying to import the FastAPI app instance
w
with this line in my test file:
Copy code
from <http://project.app|project.app> import app
if you would like to be able to import as 
<http://project.app|project.app>
, then you should adjust your source roots to add one at the root of the repository: i.e, chop nothing off of those paths
which is the last example on this page: https://www.pantsbuild.org/docs/source-roots#no-source-root
so you probably want:
Copy code
[source]
root_patterns = [
    "/",
    "/tests"
]
c
Made those changes and 1 step forward, new errors:
Copy code
==================================== ERRORS ====================================
_________________ ERROR collecting tests/auth/test_routers.py __________________
ImportError while importing test module '/private/var/folders/5q/xbnx98wd4g12xd_60swx7yp00000gn/T/pants-sandbox-rQDDI8/tests/auth/test_routers.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
/opt/homebrew/Cellar/python@3.9/3.9.15/Frameworks/Python.framework/Versions/3.9/lib/python3.9/importlib/__init__.py:127: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
tests/auth/test_routers.py:9: in <module>
    from project.app.main import app
project/app/__init__.py:9: in <module>
    from .config import get_settings
project/app/config.py:17: in <module>
    from dynaconf import Dynaconf, Validator
/Users/dfarrell/.cache/pants/named_caches/pex_root/venvs/7e318eaca32c509748765c7d46333ae480dc3894/f3dca776f079a35aa240d41b31cbb24e27b21c02/lib/python3.9/site-packages/dynaconf/__init__.py:5: in <module>
    from dynaconf.contrib import DjangoDynaconf  # noqa
/Users/dfarrell/.cache/pants/named_caches/pex_root/venvs/7e318eaca32c509748765c7d46333ae480dc3894/f3dca776f079a35aa240d41b31cbb24e27b21c02/lib/python3.9/site-packages/dynaconf/contrib/__init__.py:4: in <module>
    from dynaconf.contrib.flask_dynaconf import DynaconfConfig  # noqa
/Users/dfarrell/.cache/pants/named_caches/pex_root/venvs/7e318eaca32c509748765c7d46333ae480dc3894/f3dca776f079a35aa240d41b31cbb24e27b21c02/lib/python3.9/site-packages/dynaconf/contrib/flask_dynaconf.py:17: in <module>
    import pkg_resources
E   ModuleNotFoundError: No module named 'pkg_resources'
- generated xml file: /private/var/folders/5q/xbnx98wd4g12xd_60swx7yp00000gn/T/pants-sandbox-rQDDI8/tests.auth.test_routers.py.tests.xml -
=========================== short test summary info ============================
ERROR tests/auth/test_routers.py
!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!
=============================== 1 error in 0.48s ===============================
w
that’s probably a few steps forward, yea!
so, a bunch of artifacts on 
pypi
have hidden dependencies on 
pkg_resources
which they don’t declare as requirements, because their build environment leaked it
in this case, it looks like 
dynaconf
has an undeclared dependency on 
setuptools
c
Checking that out. By the way, thanks for sticking with all my questions, I know I have a lot.
w
no worries. we have work to do to make all of this easier.
c
Ummm where exactly does 
python_requirement(…)
go, that doesn’t look like valid pants.toml syntax.
w
in a 
BUILD
file (likely already generated for you by 
tailor
)
tailor
will have generated one next to any files named 
requirements.txt
c
Gotcha
I’ve added this:
Copy code
python_requirement(
    name="setuptools",
    requirements=["setuptools==41.0.1"],
)
to the BUILD file in the directory where my test file lives. Running 
./pants test ::
still can’t find pkg_resources.
w
See the link again: you'll need to declare that 
dynaconf
in particular has a dep on 
setuptools
c
Ahhh, didn’t see that necessity….
Configured this in my pants.toml file:
Copy code
python_requirement(
    name="setuptools",
    requirements=["setuptools"],
)

python_requirement(
    name="dynaconf",
    requirements=["Dynaconf"],
    dependencies=[":setuptools"],
)
Getting a ModuleNotFoundError: No module named ‘dynaconf’ when I run the tests. I’m using Poetry, do I need to use the 
overrides
option in the 
python_requirements
section?
h
Is dynaconf also in your pyproject.toml/requirements.txt?
If so then you now have an ambiguity (two targets that provide “dynaconf”). You might see WARNings to that effect?
In which case, you can do this:
Copy code
poetry_requirements(
  ...
  overrides = {
    "dynaconf": {"dependencies":  [":setuptools"]} # Or wherever your setuptools requirement is...
  }
)
to add the dep to the existing dynaconf target, instead of having two of them
c
I do have Dynaconf in my pyproject.toml file, and I’ll make the changes you suggest. Thanks @happy-kitchen-89482!
I’ve added this:
Copy code
poetry_requirements(
    name="dynaconf",
    overrides={
        "dynaconf": {"dependencies": [":setuptools"]}
    }
)
to my BUILD file in the directory of the module I’m trying to test. Unfortunately, I got a new error:
Copy code
./pants test ::                  
08:54:07.10 [ERROR] 1 Exception encountered:

  Exception: Unmatched glob from poetry_requirements(address="tests/auth:dynaconf", description=None, module_mapping=FrozenDict({}), overrides={('dynaconf',): {'dependencies': [':setuptools']}}, source=pyproject.toml, tags=None, type_stubs_module_mapping=FrozenDict({}))'s field `source`: "tests/auth/pyproject.toml"

Do the file(s) exist? If so, check if the file(s) are in your `.gitignore` or the global `pants_ignore` option, which may result in Pants not being able to see the file(s) even though they exist on disk. Refer to <https://www.pantsbuild.org/v2.14/docs/troubleshooting#pants-cannot-find-a-file-in-your-project>.
Hi @happy-kitchen-89482, any suggestions about the above problem?
h
Is that 
poetry_requirements()
target in the same dir as the pyproject.toml containing your requirements? That error is saying that the target expects to find 
tests/auth/pyproject.toml
, so I assume you put the target in 
tests/auth/BUILD
, and that is I guess not where pyproject.toml lives?
c
Correct, I added that 
poetry_requirements()
target to the 
tests/auth/BUILD
file. I though that change was relative to the directory where the code lives that used it. From what you’re saying I should move the 
poetry_requirements
to 
BUILD
file where my pyproject.toml file lives….
Better, but 
./pants test ::
back to complaining about 
No module named 'requests'
. I do have an import requests line in the test code.
h
Yep, 
poetry_requirements
“wraps” pyproject.toml, for the purpose of generating requirements from it.
And presumably you do have an entry for 
requests
in the pyproject.toml?
Generally how all this works is, when Pants sees an import it tries to map it to something that provides the imported symbol. That can be either some other first-party file in your repo, or some thirdparty requirement. 
requests
is obviously in the latter category.
So for Pants to “know about” requests, it has to be listed in a 
requirements.txt
or a 
pyproject.toml
or something, and then a 
python_requirements
(resp. 
poetry_requirements
) target in a sibling BUILD file tells Pants to parse the 
requirements.txt
(resp. 
pyproject.toml
) and find providers of symbols there.
Does that make sense?
c
Benjy, thanks for the explanation, and yes, this does make sense. In my case, I don’t have requests in my pyproject.toml file because I don’t use it directly in my code. It’s getting pulled in as a 3rd party dependency in this way from my test code:
Copy code
from fastapi.testclient import TestClient
I’m guessing I need to add 
poetry_requirements()
something like what we did for dynaconf, but for requests?
Added this to my BUILD file in the directory where myproject.toml lives:
Copy code
poetry_requirements(
    name="fastapi", overrides={"fastapi": {"dependencies": [":requests"]}}
)
Still getting the NoModuleFound error for requests….
h
Ah, so this is a different problem. 
fastapi
should be declaring a requirement of 
requests
, and Pants will use that. So that is weird.
Ah, how are you depending on fastapi in your pyproject.toml? What does that line look like?
That overrides looks correct BTW, assuming there is a line for 
requests
in your 
pyproject.toml
?
Or maybe it should be 
"dependencies": ["#requests"]
, I am forever confused by this
However, I notice that fastapi’s own requirements are attached in its metadata to the 
all
extra, so just installing 
fastapi
will not bring 
requests
in
This is true outside of Pants!
Copy code
(venv) $ pip install fastapi
Collecting fastapi
  Downloading fastapi-0.86.0-py3-none-any.whl (55 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 55.5/55.5 KB 431.3 kB/s eta 0:00:00
Collecting pydantic!=1.7,!=1.7.1,!=1.7.2,!=1.7.3,!=1.8,!=1.8.1,<2.0.0,>=1.6.2
  Using cached pydantic-1.10.2-cp39-cp39-macosx_11_0_arm64.whl (2.6 MB)
Collecting starlette==0.20.4
  Downloading starlette-0.20.4-py3-none-any.whl (63 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 63.6/63.6 KB 1.2 MB/s eta 0:00:00
Collecting anyio<5,>=3.4.0
  Using cached anyio-3.6.2-py3-none-any.whl (80 kB)
Collecting typing-extensions>=3.10.0
  Using cached typing_extensions-4.4.0-py3-none-any.whl (26 kB)
Collecting sniffio>=1.1
  Using cached sniffio-1.3.0-py3-none-any.whl (10 kB)
Collecting idna>=2.8
  Using cached idna-3.4-py3-none-any.whl (61 kB)
Installing collected packages: typing-extensions, sniffio, idna, pydantic, anyio, starlette, fastapi
Successfully installed anyio-3.6.2 fastapi-0.86.0 idna-3.4 pydantic-1.10.2 sniffio-1.3.0 starlette-0.20.4 typing-extensions-4.4.0
WARNING: You are using pip version 22.0.4; however, version 22.3.1 is available.
You should consider upgrading via the '/private/tmp/venv/bin/python -m pip install --upgrade pip' command.
(venv) $ python
Python 3.9.13 (main, Aug  5 2022, 16:05:40) 
[Clang 13.1.6 (clang-1316.0.21.2.5)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import fastapi
>>> import requests
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'requests'
>>>
So your pyproject.toml should depend on 
fastapi[all]
rather than just 
fastapi
, to get those 3rdparty requirements in.
c
Thanks @happy-kitchen-89482, working on trying to figure out what incantation will make poetry add 
fastapi[all]
actually add it. 🙂
5 Views
Open in Slack
PreviousNext
Powered by