https://pantsbuild.org/ logo
Join Slack
Powered by
I am having a possibly very silly/stupid issue get...
# general
s
I am having a possibly very silly/stupid issue getting started using 
python_tests
, if someone could help me get unblocked. I have this very simple setup
Copy code
#BUILD
python_sources(
    name="src",
    resolve="cuspy",
    sources=["src/**/*.py"],
    dependencies=[
        ":reqs",
    ],
)

python_requirements(
    name="reqs",
    resolve="cuspy",
    source="pyproject.toml",
)

python_tests(
    name="tests",
    sources=[
        "tests/test_*.py",
    ],
    resolve="cuspy",
    dependencies=[
        ":src",
    ],
)
super basic test
Copy code
def test_env_file():
    from cuspy_datahold.config import config

    assert config.s3.access_key is not None
and yet, when I run a test, I get
Copy code
def test_env_file():
>       from cuspy_datahold.config import config
E       ModuleNotFoundError: No module named 'cuspy_datahold'

libraries/cuspy_datahold/tests/test_config.py:2: ModuleNotFoundError
- generated xml file: /home/coder/.cachelibraries.cuspy_datahold.tests.test_config.py@tests.xml -
Copy code
[pytest]
install_from_resolve = "cuspy"
Copy code
pants test libraries/cuspy_datahold/tests/test_config.py 
16:53:23.81 [INFO] Initializing scheduler...
16:53:26.18 [INFO] Scheduler initialized.
16:53:28.26 [INFO] Canceled: Building 23 requirements for requirements.pex from the third-party/python/cuspy.lock resolve: aioboto3, click, elasticsearch[async], flytekit, microsoft-python-type-stubs@ git+<https://github.com/micro>... (258 characters truncated)
16:53:32.02 [INFO] Completed: Building pytest.pex
16:53:32.02 [INFO] Completed: Building 23 requirements for requirements.pex from the third-party/python/cuspy.lock resolve: aioboto3, click, elasticsearch[async], flytekit, microsoft-python-type-stubs@ git+<https://github.com/micro>... (258 characters truncated)
16:53:33.05 [INFO] Completed: Building local_dists.pex
16:53:35.47 [INFO] Completed: Building pytest_runner.pex
16:53:35.93 [ERROR] Completed: Run Pytest - libraries/cuspy_datahold/tests/test_config.py:../tests - failed (exit code 1).
============================= test session starts ==============================
platform linux -- Python 3.12.3, pytest-8.3.5, pluggy-1.6.0
rootdir: /home/coder/.cachelibraries/cuspy_datahold
configfile: pytest.ini
plugins: cov-6.1.1, asyncio-0.26.0
asyncio: mode=Mode.AUTO, asyncio_default_fixture_loop_scope=function, asyncio_default_test_loop_scope=function
collected 1 item

libraries/cuspy_datahold/tests/test_config.py F
WARNING: Failed to generate report: No data to report.

                                                                         [100%]

=================================== FAILURES ===================================
________________________________ test_env_file _________________________________

    def test_env_file():
>       from cuspy_datahold.config import config
E       ModuleNotFoundError: No module named 'cuspy_datahold'

libraries/cuspy_datahold/tests/test_config.py:2: ModuleNotFoundError
- generated xml file: /home/coder/.cachelibraries.cuspy_datahold.tests.test_config.py@tests.xml -
================================ tests coverage ================================
_______________ coverage: platform linux, python 3.12.3-final-0 ________________

=========================== short test summary info ============================
FAILED libraries/cuspy_datahold/tests/test_config.py::test_env_file - ModuleNotFoundError: No module named 'cuspy_datahold'
============================== 1 failed in 0.10s ===============================

/home/coder/.cache/pants/named_caches/pex_root/venvs/1/s/6d70758f/venv/lib/python3.12/site-packages/coverage/inorout.py:509: CoverageWarning: Module cuspy_datahold was never imported. (module-not-imported)
  self.warn(f"Module {pkg} was never imported.", slug="module-not-imported")
/home/coder/.cache/pants/named_caches/pex_root/venvs/1/s/6d70758f/venv/lib/python3.12/site-packages/coverage/control.py:915: CoverageWarning: No data was collected. (no-data-collected)
  self._warn("No data was collected.", slug="no-data-collected")
/home/coder/.cache/pants/named_caches/pex_root/venvs/1/s/6d70758f/venv/lib/python3.12/site-packages/pytest_cov/plugin.py:336: CovReportWarning: Failed to generate report: No data to report.

  warnings.warn(CovReportWarning(message), stacklevel=1)



āœ• libraries/cuspy_datahold/tests/test_config.py:../tests failed in 0.43s.
f
a
can python_requirements even use pyproject.toml? EDIT: yes it can
f
What is your full 
pants.toml
?
Also one of the paths is 
libraries/cuspy_datahold/tests/test_config.py
, you would need to set 
src/libraries
(assuming that is the parent path) as a source root.
s
Copy code
[GLOBAL]
pants_version = "2.25.1"
colors = true

backend_packages = [
  "pants.backend.shell",
  "pants.backend.python",
  "pants.backend.python.typecheck.mypy",
  "pants.backend.docker",
  "pants.backend.experimental.python",
  "pants.backend.experimental.python.lint.ruff.check",
  "pants.backend.experimental.python.lint.ruff.format",
]


[python]
interpreter_constraints = ["CPython>=3.10,<3.13"]
enable_resolves = true
default_resolve = "default"

[pytest]
install_from_resolve = "cuspy"

[python.resolves]
default = "third-party/python/default.lock"
cuspy = "third-party/python/cuspy.lock" 

[python-infer]
imports = false
Copy code
pants roots | grep datahold
libraries/cuspy_datahold/src
f
I'd also check: ā€¢ Is dependency inference connecting the tests with the sources? Run 
pants dependencies TARGET
to see. ā€¢ Are all of the sources present in the execution sandbox? Run your test with 
--keep-sandboxes=on_failure
and then inspect the preserved sandbox. The path will be printed out in the log.
šŸ‘€ 1
And what are the full relative paths of the source files at issue here (from the build root)? (Output of something like 
find . -type f
would be useful.)
re source roots, maybe try setting the following in 
pants.toml
?
Copy code
[source]
root_patterns = ["src", "libraries"]
although then a source file would be imported as 
cuspy_datahold.src.foo
and not 
cuspy_datahold.foo
s
does the root needs to be setup for both library/cusp_datahold/src and library/cusp_datahold/tests/src
f
yes
s
ok let me play with that, already very helpful
f
and let's say you have a 
cuspy_datahold.foo
module, then its file would live at 
library/cusp_datahold/src/cusp_datahold/foo.py
note the inclusion of the parent module names under the source root directory
essentially treat the source roots as if they were entries in the 
PYTHONPATH
(that is an oversimpification but it is a somewhat apt comarison)
s
and how would you structure your tests?
because I know that the source part seems to work, i just cannot get it to work from tests
i.e. I can build pex, they work, run code. so it's not an issue with the root setup of the main src, but something to do with the python_tests setup
f
Then I'd use 
--keep-sandoxes=on_failure
to inspect the actual execution sandbox used for the failing test,.
s
I found how to solve, although not super happy about it. This is the current structure I have
Copy code
libraries/cuspy_datahold/
ā”œā”€ā”€ BUILD
ā”œā”€ā”€ pyproject.toml
ā”œā”€ā”€ src
ā”‚   ā””ā”€ā”€ cuspy_datahold
ā”‚       ā”œā”€ā”€ examples
ā”‚       ā”‚   ā”œā”€ā”€ cow
ā”‚       ā”‚   ā”‚   ā””ā”€ā”€ __init__.py
ā”‚       ā”‚   ā”œā”€ā”€ flyte
ā”‚       ā”‚   ā”‚   ā””ā”€ā”€ my_project
ā”‚       ā”‚   ā”‚       ā””ā”€ā”€ workflows
ā”‚       ā”‚   ā”‚           ā””ā”€ā”€ __init__.py
ā”‚       ā”‚   ā”œā”€ā”€ omat24
ā”‚       ā”‚   ā”‚   ā””ā”€ā”€ __init__.py
ā”‚       ā”‚   ā””ā”€ā”€ pymatgen
ā”‚       ā”‚       ā””ā”€ā”€ v0
ā”‚       ā”‚           ā””ā”€ā”€ __init__.py
ā”‚       ā”œā”€ā”€ __init__.py
ā”‚       ā””ā”€ā”€ models
ā”‚           ā””ā”€ā”€ __init__.py
ā””ā”€ā”€ tests
    ā”œā”€ā”€ cuspy_datahold
    ā”‚   ā””ā”€ā”€ __init__.py
    ā””ā”€ā”€ __init__.py
Notice the 
BUILD
file at the root of this library, and the nested 
src/cuspy_datahold
code source. my BUILD file includes the following
Copy code
python_sources(
    name="src",
    sources=["src/**/*.py"],
    resolve="cuspy",
    dependencies=[
        ":reqs",
    ],
)
and so the tests use it this way
Copy code
python_tests(
    name="tests",
    sources=["tests/**/test_*.py"],
    resolve="cuspy",
    dependencies=[
        ":src",
        ":reqs"
    ],
)
However, with that, I get
Copy code
ModuleNotFoundError: No module named 'cuspy_datahold'
I guess because with this configuration, the code becomes visible with
Copy code
src.cuspy_datahold
? This is weird because my pants roots actually shows
Copy code
pants roots | grep datahold
libraries/cuspy_datahold/src
So I thought... this means it's going to be picked up as 
cuspy_datahold
The solution, which I don't like, is to create a BUILD file inside 
libraries/cuspy_datahold/src
with the following content:
Copy code
python_sources(
    sources=["**/*.py"],
    resolve="cuspy",
)
and then the tests in ``libraries/cuspy_datahold/BUILD`` can be configured as such
Copy code
python_tests(
    name="tests",
    sources=["tests/**/test_*.py"],
    resolve="cuspy",
    dependencies=[
        "./src",    # uses the BUILD in the src subdirectory
        ":reqs"
    ],
)
this is a bit annoying, but I think it's more to do with our structure than with pants
8 Views
Open in Slack
PreviousNext
Powered by