https://pantsbuild.org/ logo
#general
Title
# general
s

stale-nightfall-29801

11/30/2021, 4:22 PM
I'm slowly but surely converting our codebase to use Pants... And I've just hit the following issue: my
pex_binary
depends on a
python_library
and when it's packed up the pex contains
mylib/[python_file_i_need.py | __init__.py]
Which seems fine, I've seen it do this in other places in my codebase and work... BUT in the
__init__.py
other parts of
mylib
are imported so I'm getting:
ModuleNotFoundError: No module named ...
errors. Do I need to explicitly depend on the rest of the library (or at least the part pulled into init_.py ?
e

enough-analyst-54434

11/30/2021, 4:25 PM
To tighten this up: are you saying you have the same package in multiple locations on the filesystem, not all with matching
__init__.py
contents?
s

stale-nightfall-29801

11/30/2021, 4:26 PM
nope sorry. I've other pex_binary's that have library dependencies where the init.py doesn't try and hoist things by importing them.
e

enough-analyst-54434

11/30/2021, 4:28 PM
Do you have --inits configured for inference?: https://www.pantsbuild.org/docs/reference-python-infer
s

stale-nightfall-29801

11/30/2021, 4:32 PM
Ah no... offt I can see why that's not on be default. It really blew out the size of the binary Thanks 🙂
e

enough-analyst-54434

11/30/2021, 4:34 PM
You're welcome. I'm not sure why it's not on by default, but it can't be due to size. Surely you absolutely need those deps if they're imported. The extra size is non-optional for a functioning app.
s

stale-nightfall-29801

11/30/2021, 4:36 PM
Ah In my case: My Pex needs:
mylib/python_file_i_need.py
and only that file But
mylib/files_i_dont_need.py
(which init.py import) requires lots of other parts of my monorepo and other libraries... so the pex goes from 10mb to 40mb bundling in dependencies I don't need
e

enough-analyst-54434

11/30/2021, 4:40 PM
Aha. Yeah, not much we can do there. You'd need to change the code structure. Pants making a decision like "They're using a non-empty init.py but we can ignore that and insert an empty or use an implicit ns package" is problematic since the unit.py code could be there for relied upon side effects while importing.
w

witty-crayon-22786

11/30/2021, 5:41 PM
yea, this would be good to enable by default. enough has changed, that i’m not sure that the original justification for leaving it disabled is still valid (we went back and forth a few times on it) cc @happy-kitchen-89482, @hundreds-father-404
…i think it had to do with needing to have BUILD files for empty
__init__.py
files…? but
tailor
makes that a non issue i think.
e

enough-analyst-54434

11/30/2021, 5:42 PM
With it turned on you might be able to avoid your bloat problem with judicious manual dependency excludes (
dependencies=["!___init___.py"]
). Thats a lazy unvetted idea though.
h

happy-kitchen-89482

11/30/2021, 5:44 PM
Yeah, I think it should be on by default now, since tailor will do the right thing.
w

witty-crayon-22786

11/30/2021, 5:44 PM
it might have even been before we had* file-level dependencies…
h

happy-kitchen-89482

11/30/2021, 6:41 PM
Definitely possible
h

hundreds-father-404

12/01/2021, 4:24 PM
that i’m not sure that the original justification for leaving it disabled is still valid
The motivation was because before we had the
overrides
mechanism, you would have to create a separate
python_library
target specifically for the
__init__.py
to avoid metadata for sibling files being set on the
__init__.py
. For example, assume only
utils.py
actually needs NumPy and it can't be inferred, so you set this
Copy code
python_library(dependencies=["//:numpy"])
Now, every single file in that folder and all subdirectories will transitively depend on NumPy because the
__init__.py
picked it up and you automatically depend on ancestor
__init__.py
files This was particularly egregious before dependency inference, when you declared everything explicitly in BUILD files. We reverted
__init__.py
being on by default for Pants 2.0 because it was hurting caching etc too much with v1 users migrating -- We don't live in that world anymore. Most people use dep inference now, and
overrides
gives you a workaround to the above problem We should change the default. Or possibly least try to auto-detect the behavior, i.e. only infer if there is content in the file?
👍 2
w

witty-crayon-22786

12/01/2021, 5:40 PM
Thanks for the reminder!