https://pantsbuild.org/ logo
#general
Title
# general
e

echoing-nightfall-72095

01/14/2021, 1:39 PM
Hi again, I’ve a question regarding the repo structure considering importing and linting: my repo has mainly two folders “apps” that contain subfolders of all apps let call them “a”, “b and “c”. Additionally, a libs folder containing “z”, “y”, “x”. in my
pants.toml
i’ve defined roots
Copy code
[source]
root_patterns = [
    "/apps/*",
    "/libs"
]
I can run my apps with:
./pants run apps/a
However, all imports cannot be found using linting (IDE is not happy either). I have include in dependencies
Copy code
python_library(
    dependencies=[
        "3rdparty/python:flask",,
        "libs/z",
        "libs/y",
        "libs/x"
        ],
)
I import as:
Copy code
from z import something
and
Copy code
from sublib import somthingelse
where the sublib is defined in a
BUILD
file using
python_library()
Everything runs great but I would like full support. When I look at the documentation I can see an example of multiple top level project https://www.pantsbuild.org/docs/source-roots#multiple-top-level-projects. Is there a way to make project subtop level and still have linters finding imports and stuff? BR Alex
e

enough-analyst-54434

01/14/2021, 2:59 PM
So
./pants run X
works and
./pants lint X
fails? Can you provide full
./pants lint
output?
h

hundreds-father-404

01/14/2021, 3:00 PM
Hey Alex, I'm trying to understand what your imports look like. Are you doing:
Copy code
from x.sublib1 import Bar
from y.sublib2 import Qux
Or, are you doing
Copy code
from sublib1 import Bar
Where
sublib1
is a subdirectory of one of
{x, y, z}
, and which parent folder it uses is determined by where the folder is defined? That is, you might have
x/sublib
and
y/sublib
, and which to use depends on the location you're importing from? (I don't expect that to work, but trying to understand your setup)
e

echoing-nightfall-72095

01/14/2021, 3:13 PM
@enough-analyst-54434 well both run. However, I get a lot of import errors in
lint
e

enough-analyst-54434

01/14/2021, 3:16 PM
OK - its odd that run works but lint does not. This either means the code you're running has alot of unused imports that are broken or there is a very non-obvious difference between lint and run goals. @hundreds-father-404 will know about the latter I suspect.
h

hundreds-father-404

01/14/2021, 3:18 PM
No, I'm not sure why
run
would work but not
lint
.
run
is usually more stringent My main confusion is how these imports of
subdir
are structured (☝️) and what is meant by
Is there a way to may project subtop level
e

echoing-nightfall-72095

01/14/2021, 3:18 PM
@hundreds-father-404 I have: . |_apps | |-a | |-b | |-c | |-libs |-z |-y |-x And I import
Copy code
import x.sublib
@hundreds-father-404 and @enough-analyst-54434 I might not be making myself clear. But run and lint runs as expected, but I get PyLint error in lint. Most errors regards imports of modules that can be imported in pants run.
h

hundreds-father-404

01/14/2021, 3:20 PM
Okay, great, so your import statements appear to be well-formed then. Thanks.
but I get PyLint error in lint
By chance, do you have
__init__.py
files in your repo? Where are those? Pylint and MyPy do not work with implicit namespace packages (PEP 420) very well, whereas other workflows like
run
support it
e

echoing-nightfall-72095

01/14/2021, 3:39 PM
If project
a.py
in project folder
a
Copy code
from handler.handle import func
handler
is a subfolder in
a
that has a
handle.py
with a function named
func
I get:
Copy code
E0611: No name 'handlers' in module 'handlers' (no-name-in-module)
E0401: Unable to import 'handlers.handle' (import-error)
h

hundreds-father-404

01/14/2021, 3:45 PM
oh, then this is not what I thought you were doing
so your import statements appear to be well-formed then
If you have `libs/a/subdir/file.py`—and source root is `libs`—then your import should always be
import a.subdir.file
or
from a.subdir.file import func
. It shouldn't be
subdir.file
- having that top-level
a
is really crucial. Note that in a monorepo, you can import from anywhere in the repo. You need that top-level
a
part for every tool (not only Pants) to know where
subdir.file
is coming from.
Otherwise, if you don't want that top-level
a
folder in your imports, you'd need to change your source roots to be
libs/a
,
libs/b
. And then your imports should always be
import subdir.file
without ever having
a
be used But you have to guarantee that you don't have something like
libs/a/subdir
and
libs/b/subdir
at the same time - those would both strip down to
subdir
and Python would be confused which to use. This is why the namespaces like
a
and
b
are helpful
e

enough-analyst-54434

01/14/2021, 3:51 PM
How does this all work with run but not lint though? That still seems clear as mud.
👍 1
h

hundreds-father-404

01/14/2021, 3:54 PM
Only hypothesis I can think of is that
run
isn't touching the problematic imports, whereas they are likely running
./pants lint ::
and running over everything. Run would fail were it to touch those same imports Oh, well and PEP 420. Run should work with that now, but iirc Pylint still doesn't like it.
e

enough-analyst-54434

01/14/2021, 3:56 PM
Can't be latter if you're looking to explain the handler example.
e

echoing-nightfall-72095

01/14/2021, 7:13 PM
If I would like IDE support for import should i then have all project folders in root along with a lib? Or can one make it work with source roots?
h

hundreds-father-404

01/14/2021, 7:15 PM
Source roots work with IDEs, but I suspect you're not applying them correctly per https://pantsbuild.slack.com/archives/C046T6T9U/p1610639148025100?thread_ts=1610631585.016600&cid=C046T6T9U For PyCharm, I also have to teach it what are my source roots by right-clicking on the source root folders and saying "Mark Directory As -> Source Root", which I do on
src/python
, for example. You'd do it on
libs/
in your repo
e

echoing-nightfall-72095

01/14/2021, 7:19 PM
interesting I was using vs code. I’ll give that a go. Can you select the ./pants venv as interpreter?
h

hundreds-father-404

01/14/2021, 7:20 PM
e

echoing-nightfall-72095

01/14/2021, 7:48 PM
Thanks, that was neat 🙂
❤️ 1
I keep having this issue though in
apps/translation_service/app.py
I try to
from handlers.handlers import transalte_text
in
apps/translation_service/handler
I have three files:
handlers.py
,
__init__.py
and
BUILD
with a
python_library()
declaration. When running
./pants lint ::
I keep getting this PyLint error:
Copy code
apps/translation_service/app.py:3:0: E0611: No name 'handlers' in module 'handlers' (no-name-in-module)
apps/translation_service/app.py:3:0: E0401: Unable to import 'handlers.handlers' (import-error)
If i move the
handlers.py
into
apps/translation_service
and
from handlers import translate_text
then I get:
Copy code
apps/translation_service/app.py:3:0: E0611: No name 'handle_translate_tags' in module 'handlers' (no-name-in-module)
apps/translation_service/app.py:3:0: E0611: No name 'handle_translate_text' in module 'handlers' (no-name-in-module)
h

hundreds-father-404

01/14/2021, 7:57 PM
What does
./pants roots
say?
e

echoing-nightfall-72095

01/14/2021, 7:57 PM
apps/semantic_tagging_manager_service apps/semantic_tagging_service apps/translation_service libs
👍 1
h

hundreds-father-404

01/14/2021, 7:59 PM
Try adding
apps/translation_service/__init__.py
. It can be empty. It sounds like right now you only have
apps/translation_service/handlers/__init__.py
iiuc
e

echoing-nightfall-72095

01/14/2021, 8:00 PM
nope, I already have one there as well 😄
h

hundreds-father-404

01/14/2021, 8:01 PM
Ahhhhh. Try renaming
handlers/handlers.py
to
handlers/handlers_foo.py
, and the import
handlers.handlers_foo
e

echoing-nightfall-72095

01/14/2021, 8:04 PM
still 😅
Copy code
apps/translation_service/app.py:3:0: E0611: No name 'handlers_foo' in module 'handlers' (no-name-in-module)
apps/translation_service/app.py:3:0: E0401: Unable to import 'handlers.handlers_foo' (import-error)
h

hundreds-father-404

01/14/2021, 8:06 PM
Okay, and yeah I realize now that was a red herring. What does
./pants dependencies apps/translation_service/app.py
say? I'm wondering if perhaps Pants is for some reason not picking up the dependency (although that's weird that
run
would work.)
e

echoing-nightfall-72095

01/14/2021, 8:06 PM
3rdparty/python:Flask-API 3rdparty/python:boto3 3rdparty/python:flask apps/translation_service/handlers/handlers_foo.py libs/util/__init__.py
I really want to thank you for all your support. It is nice to feel that I’m not left alone in trying to set all of this up. All start can be frustrating, right 😄
It is weird in my other app
apps/semantic_tagging_service/app.py
,
from apis/v1/api import BLUEPRINT
where
apps/semantic_tagging_service/apis/v1/api.py
works fine 😄
h

hundreds-father-404

01/14/2021, 8:14 PM
One simple test we can do: ensure the file actually shows up in the chroot: https://www.pantsbuild.org/docs/rules-api-tips#debugging-look-inside-the-chroot Use
--no-process-execution-cleanup-local-dirs
, then look if all the files you expect are there. This will help us to confirm whether it's an issue with Pants's not properly including the files vs. with Pylint not liking your project setup
e

echoing-nightfall-72095

01/14/2021, 8:18 PM
Do you mean like finding the folder in pants.d/tmp…?
h

hundreds-father-404

01/14/2021, 8:19 PM
./pants run
will run under
<build-root>/pants.d
so that you can do things like write files to your build root. But most processes like
test
and
lint
instead run in a temporary directory, often in
/tmp/
. That option will tell you where the tmpdir is and avoid cleanup of it so that you can inspect the "chroot"
e

echoing-nightfall-72095

01/14/2021, 8:24 PM
I do not know what inspect the chroot mean, sorry 🙂
h

hundreds-father-404

01/14/2021, 8:25 PM
That's okay, I didn't know at first either. A chroot == temporary directory. Inspecting the chroot means running something like
ls /tmp/path/to/dir
and seeing what files are in it. Run
./pants --no-process-execution-cleanup-local-dirs lint ::
, then look for the log message saying what folder was used
e

echoing-nightfall-72095

01/14/2021, 8:30 PM
Thanks how do i know which of the many folders to look for? I’ve searches for handlers, but found none.
Sleecting just an arbitrary reveals two files _`_run_._sh` and
____findbinary.sh
h

hundreds-father-404

01/14/2021, 8:34 PM
Ah, so there are going to be multiple different log messages. That first one is a process to find something installed on your machine and irrelevant. You want the want near the end, near where it says the Pylint output. You should see inside the folder most of your source code
e

echoing-nightfall-72095

01/14/2021, 8:37 PM
Got it 😄
Could it have something to do with this pants.toml?
Copy code
[python-setup]
interpreter_constraints = ["CPython==3.8.*"]
All files are there if that is what I should look for 🙂
I can run
pylint apps/translation-service/app.py
and get
apps/translation_service/app.py:3:0: E0401: Unable to import 'handlers.handlers_foo' (import-error)
h

hundreds-father-404

01/14/2021, 8:54 PM
With that, I expect you'd need to run
PYTHONPATH=apps/translation-service pylint apps/translation-service/app.py
so that Python/Pylint know to ignore the prefix
e

echoing-nightfall-72095

01/14/2021, 9:00 PM
that worked 😄
the error is not there anymore 😄
h

hundreds-father-404

01/14/2021, 9:02 PM
Excellent. So now the goal is to get Pants to emulate that same behavior, and figure out why it's not. Given that you said
./pants roots
already includes
apps/translation-service
, Pants should be setting PYTHONPATH to that value automatically. You've already confirmed the right files are showing up in the chroot, so it's not an issue with missing dependencies. One silly thing to check, do you accidentally have an
__init__..py
in the
apps/
folder?
e

echoing-nightfall-72095

01/14/2021, 9:03 PM
No that I did not 😄
I found the issue. I believe having two resources with identical filenames caused it. It have now moved to unique python package names.
h

hundreds-father-404

01/15/2021, 8:37 AM
Yep, that would do it! Yay