https://pantsbuild.org/ logo
#general
Title
# general
m

miniature-lion-44217

04/13/2021, 12:51 PM
Hey, guys! I'm starting a Python monorepo for some libs and tools. For now, I have the following structure:
Copy code
├── constraints.txt
├── pants
├── pants.toml
├── pyproject.toml
├── README.md
├── requirements.txt
└── src
    ├── bees
    │   ├── BUILD
    │   ├── cli
    │   │   ├── base.py
    │   │   ├── BUILD
    │   │   └── __init__.py
    │   └── __init__.py
    └── tools
        └── rnotes
            ├── BUILD
            └── main.py
Where
bees
is the namespace module to store libs and
tools
is where CLIs go. The problem is this way I have to set both
bees
and
tools
as source roots, but I did't want to have the
tools
folder to be discoverable as a dependency source. Setting the
bees
folder only results in the following error:
Copy code
09:46:00.33 [ERROR] Exception caught: (pants.engine.internals.scheduler.ExecutionError)
  File "/home/hiago/.cache/pants/setup/bootstrap-Linux-x86_64/2.3.0rc2_py38/lib/python3.8/site-packages/pants/bin/local_pants_runner.py", line 246, in run
    engine_result = self._perform_run(goals)
  File "/home/hiago/.cache/pants/setup/bootstrap-Linux-x86_64/2.3.0rc2_py38/lib/python3.8/site-packages/pants/bin/local_pants_runner.py", line 167, in _perform_run
    return self._perform_run_body(goals, poll=False)
  File "/home/hiago/.cache/pants/setup/bootstrap-Linux-x86_64/2.3.0rc2_py38/lib/python3.8/site-packages/pants/bin/local_pants_runner.py", line 184, in _perform_run_body
    return self.graph_session.run_goal_rules(
  File "/home/hiago/.cache/pants/setup/bootstrap-Linux-x86_64/2.3.0rc2_py38/lib/python3.8/site-packages/pants/init/engine_initializer.py", line 130, in run_goal_rules
    exit_code = self.scheduler_session.run_goal_rule(
  File "/home/hiago/.cache/pants/setup/bootstrap-Linux-x86_64/2.3.0rc2_py38/lib/python3.8/site-packages/pants/engine/internals/scheduler.py", line 568, in run_goal_rule
    self._raise_on_error([t for _, t in throws])
  File "/home/hiago/.cache/pants/setup/bootstrap-Linux-x86_64/2.3.0rc2_py38/lib/python3.8/site-packages/pants/engine/internals/scheduler.py", line 536, in _raise_on_error
    raise ExecutionError(

Exception message: 1 Exception encountered:

  NoSourceRootError: No source root found for `src/tools/rnotes`. See <https://www.pantsbuild.org/v2.3/docs/source-roots> for how to define source roots.
So I thought removing rnotes's
python_library
target would be a good idea, but it messes things up as well:
Copy code
Traceback (most recent call last):
  File "/usr/lib/python3.8/runpy.py", line 130, in _get_module_details
    spec = importlib.util.find_spec(mod_name)
  File "/usr/lib/python3.8/importlib/util.py", line 94, in find_spec
    parent = __import__(parent_name, fromlist=['__path__'])
ModuleNotFoundError: No module named 'rnotes'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/hiago/Workspace/golden-TOOLS-library/.pants.d/tmptg24ref7/bin.pex/.bootstrap/pex/pex.py", line 488, in execute
  File "/home/hiago/Workspace/golden-TOOLS-library/.pants.d/tmptg24ref7/bin.pex/.bootstrap/pex/pex.py", line 405, in _wrap_coverage
  File "/home/hiago/Workspace/golden-TOOLS-library/.pants.d/tmptg24ref7/bin.pex/.bootstrap/pex/pex.py", line 436, in _wrap_profiling
  File "/home/hiago/Workspace/golden-TOOLS-library/.pants.d/tmptg24ref7/bin.pex/.bootstrap/pex/pex.py", line 544, in _execute
  File "/home/hiago/Workspace/golden-TOOLS-library/.pants.d/tmptg24ref7/bin.pex/.bootstrap/pex/pex.py", line 661, in execute_entry
  File "/home/hiago/Workspace/golden-TOOLS-library/.pants.d/tmptg24ref7/bin.pex/.bootstrap/pex/pex.py", line 670, in execute_module
  File "/usr/lib/python3.8/runpy.py", line 203, in run_module
    mod_name, mod_spec, code = _get_module_details(mod_name)
  File "/usr/lib/python3.8/runpy.py", line 136, in _get_module_details
    raise error(msg.format(mod_name, type(ex).__name__, ex)) from ex
ImportError: Error while finding module specification for 'rnotes.main' (ModuleNotFoundError: No module named 'rnotes')
Is there a better way to structure this? Or I'll have to live with burden of watching if people are importing from the
tools
folder?
e

enough-analyst-54434

04/13/2021, 1:32 PM
Well, if socializing / enforcing the rule that tools must have a main.py and libs should never, you could set:
Copy code
[source]
marker_filenames = ["main.py"]
You can do a quick check this does what you want with
./pants --source-marker-filenames=main.py roots
. Clearly it would be more direct if Pants supported visibility rules directly. That's probably a generally useful feature but we don't have it today.
m

miniature-lion-44217

04/13/2021, 1:44 PM
@enough-analyst-54434, actually I just want tools not to be found by other projects inside
tools
or libs inside the
bees
folder
e

enough-analyst-54434

04/13/2021, 1:54 PM
This would sort of do that. all main.py would live in a source root that is their parent directory. The implication is they all have the same module name: `main`; so
import main
from over in a lib will be ambiguous the minute you have >1 tool and pants should blow up.
Not pretty, but it should be a hacky way to enforce all this.
The right answer is clearly for Pants to support this. We could spec out a feature in an issue for that. I'm hoping there might be a way to use the current features to unblock you though.
I'm assuming here you've ruled out using
./pants dependees
or `./pants dependencies`goals in git hooks / CI to enforce this.
If you are willing to have an out of band goal do the enforcement though, you should be able to set something up with these two existing goals and some scripting or else write a custom goal.
h

happy-kitchen-89482

04/13/2021, 4:09 PM
Enforcing dependency rules is a good feature to have - it is however hard to do in a generic way, since every repo has its own rules, and it's hard to capture all those different cases in a declarative format.
We might be able to provide some basic stuff, with easy hooks to add custom rules.
m

miniature-lion-44217

04/13/2021, 11:31 PM
@enough-analyst-54434 setting the source roots to include all marker_filenames parent dirs blows pants up no matter if i import an enf module or don't. see:
Copy code
Traceback (most recent call last):
  File "/home/hiago/Workspace/golden-TOOLS-library/.pants.d/tmpxzzo1nkc/bin.pex/.bootstrap/pex/pex.py", line 488, in execute
  File "/home/hiago/Workspace/golden-TOOLS-library/.pants.d/tmpxzzo1nkc/bin.pex/.bootstrap/pex/pex.py", line 405, in _wrap_coverage
  File "/home/hiago/Workspace/golden-TOOLS-library/.pants.d/tmpxzzo1nkc/bin.pex/.bootstrap/pex/pex.py", line 436, in _wrap_profiling
  File "/home/hiago/Workspace/golden-TOOLS-library/.pants.d/tmpxzzo1nkc/bin.pex/.bootstrap/pex/pex.py", line 544, in _execute
  File "/home/hiago/Workspace/golden-TOOLS-library/.pants.d/tmpxzzo1nkc/bin.pex/.bootstrap/pex/pex.py", line 661, in execute_entry
  File "/home/hiago/Workspace/golden-TOOLS-library/.pants.d/tmpxzzo1nkc/bin.pex/.bootstrap/pex/pex.py", line 670, in execute_module
  File "/usr/lib/python3.8/runpy.py", line 203, in run_module
    mod_name, mod_spec, code = _get_module_details(mod_name)
  File "/usr/lib/python3.8/runpy.py", line 138, in _get_module_details
    raise error("No module named %s" % mod_name)
ImportError: No module named main
and also, i hadn't thought of using those goals to check whether a rule is being broken or not. gonna try that.
e

enough-analyst-54434

04/13/2021, 11:32 PM
Excellent. There's also
filedeps
. You should be able to cobble something together.
💯 1
m

miniature-lion-44217

04/13/2021, 11:33 PM
@happy-kitchen-89482 i'm still new to Pants, but i'll be glad to help find a way to implement this feature
👏 1
h

happy-kitchen-89482

04/14/2021, 12:02 AM
Thanks, happy to give some guidance! Might make sense to get a little further along with adoption first, but I don't think this would be terribly hard to do overall.
2 Views