Hey, guys! I'm starting a Python monorepo for some libs and tools. For now, I have the following structure:

├── constraints.txt
├── pants
├── pants.toml
├── pyproject.toml
├── README.md
├── requirements.txt
└── src
    ├── bees
    │   ├── BUILD
    │   ├── cli
    │   │   ├── base.py
    │   │   ├── BUILD
    │   │   └── __init__.py
    │   └── __init__.py
    └── tools
        └── rnotes
            ├── BUILD
            └── main.py

Where

bees

is the namespace module to store libs and

tools

is where CLIs go. The problem is this way I have to set both

bees

and

tools

as source roots, but I did't want to have the

tools

folder to be discoverable as a dependency source. Setting the

bees

folder only results in the following error:

09:46:00.33 [ERROR] Exception caught: (pants.engine.internals.scheduler.ExecutionError)
  File "/home/hiago/.cache/pants/setup/bootstrap-Linux-x86_64/2.3.0rc2_py38/lib/python3.8/site-packages/pants/bin/local_pants_runner.py", line 246, in run
    engine_result = self._perform_run(goals)
  File "/home/hiago/.cache/pants/setup/bootstrap-Linux-x86_64/2.3.0rc2_py38/lib/python3.8/site-packages/pants/bin/local_pants_runner.py", line 167, in _perform_run
    return self._perform_run_body(goals, poll=False)
  File "/home/hiago/.cache/pants/setup/bootstrap-Linux-x86_64/2.3.0rc2_py38/lib/python3.8/site-packages/pants/bin/local_pants_runner.py", line 184, in _perform_run_body
    return self.graph_session.run_goal_rules(
  File "/home/hiago/.cache/pants/setup/bootstrap-Linux-x86_64/2.3.0rc2_py38/lib/python3.8/site-packages/pants/init/engine_initializer.py", line 130, in run_goal_rules
    exit_code = self.scheduler_session.run_goal_rule(
  File "/home/hiago/.cache/pants/setup/bootstrap-Linux-x86_64/2.3.0rc2_py38/lib/python3.8/site-packages/pants/engine/internals/scheduler.py", line 568, in run_goal_rule
    self._raise_on_error([t for _, t in throws])
  File "/home/hiago/.cache/pants/setup/bootstrap-Linux-x86_64/2.3.0rc2_py38/lib/python3.8/site-packages/pants/engine/internals/scheduler.py", line 536, in _raise_on_error
    raise ExecutionError(

Exception message: 1 Exception encountered:

  NoSourceRootError: No source root found for `src/tools/rnotes`. See <https://www.pantsbuild.org/v2.3/docs/source-roots> for how to define source roots.

So I thought removing rnotes's

python_library

target would be a good idea, but it messes things up as well:

Traceback (most recent call last):
  File "/usr/lib/python3.8/runpy.py", line 130, in _get_module_details
    spec = importlib.util.find_spec(mod_name)
  File "/usr/lib/python3.8/importlib/util.py", line 94, in find_spec
    parent = __import__(parent_name, fromlist=['__path__'])
ModuleNotFoundError: No module named 'rnotes'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/hiago/Workspace/golden-TOOLS-library/.pants.d/tmptg24ref7/bin.pex/.bootstrap/pex/pex.py", line 488, in execute
  File "/home/hiago/Workspace/golden-TOOLS-library/.pants.d/tmptg24ref7/bin.pex/.bootstrap/pex/pex.py", line 405, in _wrap_coverage
  File "/home/hiago/Workspace/golden-TOOLS-library/.pants.d/tmptg24ref7/bin.pex/.bootstrap/pex/pex.py", line 436, in _wrap_profiling
  File "/home/hiago/Workspace/golden-TOOLS-library/.pants.d/tmptg24ref7/bin.pex/.bootstrap/pex/pex.py", line 544, in _execute
  File "/home/hiago/Workspace/golden-TOOLS-library/.pants.d/tmptg24ref7/bin.pex/.bootstrap/pex/pex.py", line 661, in execute_entry
  File "/home/hiago/Workspace/golden-TOOLS-library/.pants.d/tmptg24ref7/bin.pex/.bootstrap/pex/pex.py", line 670, in execute_module
  File "/usr/lib/python3.8/runpy.py", line 203, in run_module
    mod_name, mod_spec, code = _get_module_details(mod_name)
  File "/usr/lib/python3.8/runpy.py", line 136, in _get_module_details
    raise error(msg.format(mod_name, type(ex).__name__, ex)) from ex
ImportError: Error while finding module specification for 'rnotes.main' (ModuleNotFoundError: No module named 'rnotes')

Is there a better way to structure this? Or I'll have to live with burden of watching if people are importing from the

tools

folder?

Well, if socializing / enforcing the rule that tools must have a main.py and libs should never, you could set:

[source]
marker_filenames = ["main.py"]

You can do a quick check this does what you want with

./pants --source-marker-filenames=main.py roots

. Clearly it would be more direct if Pants supported visibility rules directly. That's probably a generally useful feature but we don't have it today.

@enough-analyst-54434, actually I just want tools not to be found by other projects inside

tools

or libs inside the

bees

folder

This would sort of do that. all main.py would live in a source root that is their parent directory. The implication is they all have the same module name: `main`; so

import main

from over in a lib will be ambiguous the minute you have >1 tool and pants should blow up.

Enforcing dependency rules is a good feature to have - it is however hard to do in a generic way, since every repo has its own rules, and it's hard to capture all those different cases in a declarative format.

@enough-analyst-54434 setting the source roots to include all marker_filenames parent dirs blows pants up no matter if i import an enf module or don't. see:

Traceback (most recent call last):
  File "/home/hiago/Workspace/golden-TOOLS-library/.pants.d/tmpxzzo1nkc/bin.pex/.bootstrap/pex/pex.py", line 488, in execute
  File "/home/hiago/Workspace/golden-TOOLS-library/.pants.d/tmpxzzo1nkc/bin.pex/.bootstrap/pex/pex.py", line 405, in _wrap_coverage
  File "/home/hiago/Workspace/golden-TOOLS-library/.pants.d/tmpxzzo1nkc/bin.pex/.bootstrap/pex/pex.py", line 436, in _wrap_profiling
  File "/home/hiago/Workspace/golden-TOOLS-library/.pants.d/tmpxzzo1nkc/bin.pex/.bootstrap/pex/pex.py", line 544, in _execute
  File "/home/hiago/Workspace/golden-TOOLS-library/.pants.d/tmpxzzo1nkc/bin.pex/.bootstrap/pex/pex.py", line 661, in execute_entry
  File "/home/hiago/Workspace/golden-TOOLS-library/.pants.d/tmpxzzo1nkc/bin.pex/.bootstrap/pex/pex.py", line 670, in execute_module
  File "/usr/lib/python3.8/runpy.py", line 203, in run_module
    mod_name, mod_spec, code = _get_module_details(mod_name)
  File "/usr/lib/python3.8/runpy.py", line 138, in _get_module_details
    raise error("No module named %s" % mod_name)
ImportError: No module named main

Excellent. There's also

filedeps

. You should be able to cobble something together.

@happy-kitchen-89482 i'm still new to Pants, but i'll be glad to help find a way to implement this feature

Thanks, happy to give some guidance! Might make sense to get a little further along with adoption first, but I don't think this would be terribly hard to do overall.