https://pantsbuild.org/ logo
Join Slack
Powered by
Hello, I am unable to run the tests as I have two ...
# general
g
Hello, I am unable to run the tests as I have two separate folders for the source code 
src
and 
tests
for tests. When I run the tests with 
pants test ::
I get the 
ModuleNotFoundError
as the environment does not know where the source code is. I tried to add the python_sources to the dependency but it doesn't work. Is there any document to setup testing in this scenario?
l
Shravan, 
python_sources
tells pants where all the different kinds of targets are (usually files), but an additional configuration of 
source roots
can be needed so that pants can properly relate import package/module names to the files when inferring dependencies between targets. You can try tweaking according to https://www.pantsbuild.org/docs/source-roots#configuring-source-roots-using-patterns
If that doesn't work, can you post a detailed but minimal directory structure and we can help diagnose further from there.
g
This is my folder structure:
Copy code
├── my-app
│   ├── BUILD
│   ├── pyproject.toml
│   ├── src
│   │   ├── app.py
│   │   └── __init__.py
│   └── tests
│       ├── conftest.py
│       ├── __init__.py
│       └── test_app.py
├── my-lib
│   ├── BUILD
│   ├── pyproject.toml
│   ├── src
│   │   └── lib
│   │       ├── __init__.py
│   │       └── utils
│   │           └── main.py
│   └── tests
│       ├── conftest.py
│       ├── __init__.py
│       └── test_lib.py
└── pants.toml
I have setup the source root as
Copy code
[source]
    root_patterns = [
      '/my-*',
    ]
This is because it worked with this to package the lib and app.
l
can you try using 
root_patterns = ['src/', 'test/']
. if I understand correctly this should cause all four dirs [my-app/src, my-app/test, my-lib/src, my-lib/test] to be included as source roots. Also, would you mind sharing an example import statement from one of the tests and the ModuleNotFoundError you are getting when pants tries to run the test?
if this fixes the test issue we can then go back to see if there is some remaining problem with getting the lib to get packaged in with the app.
g
It still doesn't work. It gives a module not found error. A typical import statement would be
Copy code
from lib.utils.main import foo
l
can you print your 
BUILD
files here? Also, for the individual test target can you print its dependencies? 
pants dependencies <the_test_target>
(https://www.pantsbuild.org/docs/project-introspection)
g
BUILD is like:
Copy code
resource(name="pyproject", source="pyproject.toml")
python_sources(name="lib", sources=["lib/**/*.py"])

python_distribution(
    name="my-lib",
    dependencies=[
        ":pyproject",
        ":lib"
        # Dependencies on code to be packaged into the distribution.
    ],
    provides=python_artifact(
        name="my-lib",
        version="0.0.1"
    ),
    # Example of setuptools config, other build backends may have other config.
    wheel_config_settings={"--global-option": ["--python-tag", "py39"]},
    generate_setup=True,
)
and the dependencies for the tests is like
Copy code
pants dependencies my-lib/tests
16:30:44.49 [WARN] Pants cannot infer owners for the following imports in the target my-lib/tests/test_lib.py:

  * lib.utils.main.foo (line: 1)
l
and 
src/lib/utils
has an 
__init__.py
?
(maybe you just omitted it from the tree above for brevity)
One last thing (I am kind of new here btw, so sorry if this is taking a while) - can you confirm that the source file itself shows up when you 
pants list --filter-target-type=python_source ::
and the error as printed above is with the 
root_patterns = ['src/', 'test/']
conf, correct?
I have to go afk for a bit but perhaps someone more knowledgeable can swoop in in the meanwhile
Shravan, ok, I was able to get back and partially reproduce and it seems the source pattern change should have fixed it. With this example:
Copy code
.
├── my-lib
│   ├── src
│   │   └── lib
│   │       ├── BUILD
│   │       ├── __init__.py
│   │       └── liba.py
│   └── tests
│       ├── BUILD
│       ├── __init__.py
│       ├── conftest.py
│       └── test_lib.py
└── pants.toml
where src/lib/BUILD:
Copy code
python_sources()
liba.py
Copy code
def add10(x):
    return x + 10
my-lib/tests/BUILD
Copy code
python_test_utils(name="test_utils")
python_tests()
and test_lib.py
Copy code
from lib import liba
def test_add10():
    assert 11 == liba.add10(1)
with pants.toml
Copy code
[GLOBAL]
pants_version = "2.16.0"
backend_packages.add = ["pants.backend.python"]

[source]
root_patterns = ["src", "tests"]
then 
pants test ::
runs fine and passes, but if we switch to
Copy code
root_patterns = ['/my-*',]
we get:
Copy code
E   ImportError: cannot import name 'liba' from 'lib' (unknown location)
when running the same test, which is very similar to what you are experiencing. Here the BUILD files are the simple ones generated by 
pants tailor ::
. I tried at first to do a single BUILD in 
my-lib
but struggled to get it going.
h
@gentle-florist-96289 Can you post a github repo containing the example that isn't working? That would make it easier to spot the configuration issue
🙏 1
But for now, 
from lib.utils.main import foo
implies that the parent of 
lib
should be a source root. Is it?
g
Hey @happy-kitchen-89482 @late-advantage-75311,Sorry for the delayed response. I have created a sample git repo here https://github.com/leshravnya/pants-test. Please let me know if it helps. I tried your solution @late-advantage-75311. However, it doesnt work for me.
l
Shravan, I'll take a look at it in a couple of hours
g
Sure Gautham, Thanks.
l
pulled and reproduced, taking a crack
@gentle-florist-96289 here are the fixes: https://github.com/leshravnya/pants-test/pull/1/files
You were missing a BUILD file to target some of the sources, and then there were some little errors that were probably from the translation to the example repo.
I debugged it by first doing 
pants dependencies my-app/tests/test_app.py
and noticing that the main file was not showing up as a dep. It was because it was not included as a target via BUILD file.
g
I got rid of the glob patterns to select the python sources in the lib and created 
BUILD
files in each directory. This works fine for running the tests. However, the packaging doesn't work anymore.
Thanks @late-advantage-75311 for the PR that helped me get rid of the glob patterns.
I have pushed the changes that work for the test cases in the git repo.
Open in Slack
PreviousNext
Powered by