https://pantsbuild.org/ logo
Join Slack
Powered by
What is the magic incantation for nested namespace...
# general
b
What is the magic incantation for nested namespace dirs?
w
in my experience, the only way to get namespace packages working (anywhere) is to ensure that you have absolutely no 
__init__.py
files… are there any in this directory structure?
pants will not create any for you, but if they exist, it will pull them in
b
pants seems fine with the TOP of my tree being-init-less
repo/ns/core -> and I package ns.core
ns has no init file - works fine
w
are there 
__init__.py
files in parent directories?
b
ns/core/leader_election/impl/zookeeper
ns && impl have no init files, the others have them
just occurred to me that htis is insane
pants is really just generating a setup.py file and calling python ./setup.py $@ on it
w
in a much more structured fashion, yea.
mixing 
__init__.py
and no-init py though will cause trouble with or without a build tool though, afaik
đź‘Ť 1
b
yea, I was asked to look into how to do nested pkgs
ns.core & ns.core.leader_election.zookeeper
which live in the same tree as described above
w
would suggest either fully using 
__init__.py
files, or fully not using them.
you only need namespace packages when the same directory structure exists in multiple places (i.e. under multiple source roots in pants terminology)… and it’s a good idea to avoid that layout anyway
đź‘Ť 1
b
people like to have that 
impl
dir for "traditional" reasons, but it doen't really play well with the tooling
to me they're really "dir tree syntactic sugar"
w
the pep is a mess =(
đź’Ż 1
b
but could you give an example of what you mean by "same dir structure exists in multiple places" ?
heh
w
well, the only useful example in the pep happens to be for “same dir structure in multiple places”:
Copy code
project1
        parent
            child
                one.py
    project2
        parent
            child
                two.py
but… that would not be a recommended layout in a monorepo anyway.
(the “same dir structure” is 
parent.child
, and the “multiple places” is project1/2)
b
I have
Copy code
ns
   project 1
      files...
      ns2
          project2
                files...
w
i don’t think that you do
b
and to make it more fun - the pkg built from project2 requires project1 pkg as a 3rdparty dep
w
i don’t think that’s how the example applies in your situation.
if you have 
ns.core
and 
ns.core.leader_election.zookeeper
b
ns.core==project1 ns.core.leader_election.zookeeper==project2
w
right, but that doesn’t reference the “same dir structure” aspect.
b
ah
no, that's a fair comment
w
when it’s nested like that, you have a unique directory per python package
they’re not duplicated
(that’s good)
b
well, we also have ns.core.leader_election.cassandra
so there is duplication above cassandra and zookeeper
ns/core/leader_election/impl
(admittedlly, the impl dir is unecessary, but some folks like those things)
w
so, a better way to explain PEP-420 perhaps is: “if you have multiple PYTHONPATH entries that contain the same packages (where 
ns.core
is a package)”
source roots in pants become PYTHONPATH entries
you would ~never want to nest multiple PYTHONPATH entries (and thus source roots) inside of one another
but you don’t need to.
this structure has one source root (at the bottom of the repository), and lots of packages, and no need for duplication:
Copy code
ns
    core
        <code>
        leader_election
            <code>
            cassandra
                <code>
            zookeeper
                <code>
b
right
w
and that is certainly what we recommend.
b
but I'm being asked to stick an init-less 
impl
dir under leader_election
w
it’s fine to have multiple source roots… no problem. but don’t put overlapping packages in them if you can avoid it.
b
which I'm starting to suspect is insane
w
@billowy-motherboard-58443: the challenge is mixing 
__init__.py
and namespace packages. you cannot have both ime.
all init, or none, afaik.
b
well, can have one at the top (
ns
) just fine, but seems like "don't do that in the middle of the tree"
w
off to grab some food. if you’d like some more comprehensive advice on how to lay things out, a ticket that explains the big picture and what you’re trying to accomplish would be very helpful
b
nod
thanks
w
https://pantsbuild.slack.com/archives/C046T6T9U/p1601664535282600?thread_ts=1601661709.263500&amp;cid=C046T6T9U again, i do not think so. i think that when python encounters one in your parents, it disables namespace packages
đź‘Ť 1
(back to lunch)
h
Yeah, that sounds correct. I’ve found PEP 420 to be a mess, personally. It’s easy to accidentally break it by someone unintentionally checking in an 
__init__.py
file. People often end up using it without realizing they’re using it Several tools don’t work properly with it, either. For example, we haven’t figured out how to get MyPy to play nicely with PEP 420, even independently of Pants. Generally, I’d recommend for folks to use empty 
__init__.py
files in each directory. It’s boiler-platey, but simple. And you can use the 
./pants validate
goal if you want to enforce that nothing is ever included in those files. (It does regex checks for your files)
e
One thing to note about the "mess" of PEP 420 is that it solves another mess that Pants encourages users to get into otherwise - old style namespace packages. If you use a monorepo layout and distribute mutiple packages you need to use namespace packages. So in that case, without PEP 420, you need to know the arcanery of a non-empty 1-line ~magic namespace package creating / joining init.py
đź‘Ť 1
Open in Slack
PreviousNext
Powered by