Hi again, I’ve a question regarding the repo structure considering importing and linting: my repo has mainly two folders “apps” that contain subfolders of all apps let call them “a”, “b and “c”. Additionally, a libs folder containing “z”, “y”, “x”. in my

pants.toml

i’ve defined roots

[source]
root_patterns = [
    "/apps/*",
    "/libs"
]

I can run my apps with:

./pants run apps/a

However, all imports cannot be found using linting (IDE is not happy either). I have include in dependencies

python_library(
    dependencies=[
        "3rdparty/python:flask",,
        "libs/z",
        "libs/y",
        "libs/x"
        ],
)

I import as:

from z import something

and

from sublib import somthingelse

where the sublib is defined in a

BUILD

file using

python_library()

Everything runs great but I would like full support. When I look at the documentation I can see an example of multiple top level project https://www.pantsbuild.org/docs/source-roots#multiple-top-level-projects. Is there a way to make project subtop level and still have linters finding imports and stuff? BR Alex

So

./pants run X

works and

./pants lint X

fails? Can you provide full

./pants lint

output?

Hey Alex, I'm trying to understand what your imports look like. Are you doing:

from x.sublib1 import Bar
from y.sublib2 import Qux

Or, are you doing

from sublib1 import Bar

Where

sublib1

is a subdirectory of one of

{x, y, z}

, and which parent folder it uses is determined by where the folder is defined? That is, you might have

x/sublib

and

y/sublib

, and which to use depends on the location you're importing from? (I don't expect that to work, but trying to understand your setup)

@enough-analyst-54434 well both run. However, I get a lot of import errors in

lint

OK - its odd that run works but lint does not. This either means the code you're running has alot of unused imports that are broken or there is a very non-obvious difference between lint and run goals. @hundreds-father-404 will know about the latter I suspect.

No, I'm not sure why

run

would work but not

lint

.

run

is usually more stringent My main confusion is how these imports of

subdir

are structured (☝️) and what is meant by

Is there a way to may project subtop level

@hundreds-father-404 I have: . |_apps | |-a | |-b | |-c | |-libs |-z |-y |-x And I import

import x.sublib

Okay, great, so your import statements appear to be well-formed then. Thanks.

but I get PyLint error in lint

By chance, do you have

__init__.py

files in your repo? Where are those? Pylint and MyPy do not work with implicit namespace packages (PEP 420) very well, whereas other workflows like

run

support it

If project

a.py

in project folder

a

from handler.handle import func

handler

is a subfolder in

a

that has a

handle.py

with a function named

func

I get:

E0611: No name 'handlers' in module 'handlers' (no-name-in-module)
E0401: Unable to import 'handlers.handle' (import-error)

oh, then this is not what I thought you were doing

so your import statements appear to be well-formed then

If you have `libs/a/subdir/file.py`—and source root is `libs`—then your import should always be

import a.subdir.file

or

from a.subdir.file import func

. It shouldn't be

subdir.file

- having that top-level

a

is really crucial. Note that in a monorepo, you can import from anywhere in the repo. You need that top-level

a

part for every tool (not only Pants) to know where

subdir.file

is coming from.

How does this all work with run but not lint though? That still seems clear as mud.

Only hypothesis I can think of is that

run

isn't touching the problematic imports, whereas they are likely running

./pants lint ::

and running over everything. Run would fail were it to touch those same imports Oh, well and PEP 420. Run should work with that now, but iirc Pylint still doesn't like it.

Can't be latter if you're looking to explain the handler example.

If I would like IDE support for import should i then have all project folders in root along with a lib? Or can one make it work with source roots?

Source roots work with IDEs, but I suspect you're not applying them correctly per https://pantsbuild.slack.com/archives/C046T6T9U/p1610639148025100?thread_ts=1610631585.016600&cid=C046T6T9U For PyCharm, I also have to teach it what are my source roots by right-clicking on the source root folders and saying "Mark Directory As -> Source Root", which I do on

src/python

, for example. You'd do it on

libs/

in your repo

interesting I was using vs code. I’ll give that a go. Can you select the ./pants venv as interpreter?

See https://www.pantsbuild.org/docs/python-third-party-dependencies#tip-set-up-a-virtual-environment-optional for a script to generate a venv to do that

Thanks, that was neat 🙂

What does

./pants roots

say?

apps/semantic_tagging_manager_service apps/semantic_tagging_service apps/translation_service libs

Try adding

apps/translation_service/__init__.py

. It can be empty. It sounds like right now you only have

apps/translation_service/handlers/__init__.py

iiuc

nope, I already have one there as well 😄

Ahhhhh. Try renaming

handlers/handlers.py

to

handlers/handlers_foo.py

, and the import

handlers.handlers_foo

still 😅

apps/translation_service/app.py:3:0: E0611: No name 'handlers_foo' in module 'handlers' (no-name-in-module)
apps/translation_service/app.py:3:0: E0401: Unable to import 'handlers.handlers_foo' (import-error)

Okay, and yeah I realize now that was a red herring. What does

./pants dependencies apps/translation_service/app.py

say? I'm wondering if perhaps Pants is for some reason not picking up the dependency (although that's weird that

run

would work.)

3rdparty/python:Flask-API 3rdparty/python:boto3 3rdparty/python:flask apps/translation_service/handlers/handlers_foo.py libs/util/__init__.py

One simple test we can do: ensure the file actually shows up in the chroot: https://www.pantsbuild.org/docs/rules-api-tips#debugging-look-inside-the-chroot Use

--no-process-execution-cleanup-local-dirs

, then look if all the files you expect are there. This will help us to confirm whether it's an issue with Pants's not properly including the files vs. with Pylint not liking your project setup

Do you mean like finding the folder in pants.d/tmp…?

./pants run

will run under

<build-root>/pants.d

so that you can do things like write files to your build root. But most processes like

test

and

lint

instead run in a temporary directory, often in

/tmp/

. That option will tell you where the tmpdir is and avoid cleanup of it so that you can inspect the "chroot"

I do not know what inspect the chroot mean, sorry 🙂

That's okay, I didn't know at first either. A chroot == temporary directory. Inspecting the chroot means running something like

ls /tmp/path/to/dir

and seeing what files are in it. Run

./pants --no-process-execution-cleanup-local-dirs lint ::

, then look for the log message saying what folder was used

Thanks how do i know which of the many folders to look for? I’ve searches for handlers, but found none.

Ah, so there are going to be multiple different log messages. That first one is a process to find something installed on your machine and irrelevant. You want the want near the end, near where it says the Pylint output. You should see inside the folder most of your source code

Got it 😄

With that, I expect you'd need to run

PYTHONPATH=apps/translation-service pylint apps/translation-service/app.py

so that Python/Pylint know to ignore the prefix

that worked 😄

Excellent. So now the goal is to get Pants to emulate that same behavior, and figure out why it's not. Given that you said

./pants roots

already includes

apps/translation-service

, Pants should be setting PYTHONPATH to that value automatically. You've already confirmed the right files are showing up in the chroot, so it's not an issue with missing dependencies. One silly thing to check, do you accidentally have an

__init__..py

in the

apps/

folder?

No that I did not 😄

Yep, that would do it! Yay