Hi all, we have been setting up a mono repo (all p...
# general
f
Hi all, we have been setting up a mono repo (all python) with Pants and are having some trouble figuring out how we ought to get Pants to construct the parts of the code we'd like to distribute. (Help needed!) I've put the full details as an in thread reply.
Below is an sketch of the current folder structure,
Copy code
repo
| pants.toml
` src
  | mathlib
    | folder1
    | folder2
    ` BUILD
  ` apps
    | lib
    | app1.py  # depends on functions in ./lib and src/mathlib
    | app2.py  # shares some dependencies in ./lib with app1
    ` BUILD
What I had intended to do was: 1. Distribute mathlib as a python package that folks at the company can script with as they would something like numpy 2. Distribute app1.py and app2.py in some way they can be easily run without scripting Since the company operates on Windows I'm using the python_distribution() target rather than PEX. What I had expected to do (perhaps naively) was to tell Pants to package the folder mathlib and its subdirectories via the BUILD file,
src/mathlib/BUILD
:
Copy code
python_distribution(
    name="mathlib_dist",
    dependencies=[ ??? ],  # I couldn't figure out how to include all the code, including subdirectories
    provides=python_artifact(
        name="mathlib",
        version="0.1.0",
        description="Various functions to process data.",
        author="Kaden",
        classifiers=[
            "Programming Language :: Python :: 3.8",
        ],
    ),
    wheel_config_settings={"--global-option": ["--python-tag", "py38"]},
)
Then to tell Pants to package each of app1.py and app2.py separately in `src/apps/BUILD`:
Copy code
python_sources()

python_distribution(
    name="app1_dist",
    entry_points={"gui_scripts": {"app1": "apps.app1:main"}},
    provides=python_artifact(
        name="app1",
        version="0.1.0",
        description="A GUI to do something with data.",
        author="Kaden",
        classifiers=[
            "Programming Language :: Python :: 3.8",
        ],
    ),
    wheel_config_settings={"--global-option": ["--python-tag", "py38"]},
)

python_distribution(
    name="app2_dist",
    entry_points={"gui_scripts": {"app2": "apps.app2:main"}},
    provides=python_artifact(
        name="app2",
        version="0.1.0",
        description="A GUI to do something else with data.",
        author="Kaden",
        classifiers=[
            "Programming Language :: Python :: 3.8",
        ],
    ),
    wheel_config_settings={"--global-option": ["--python-tag", "py38"]},
)
This hasn't quite worked. What I have now are four problems. 1. Pants wants to only include files in mathlib with dependees at the level that I specify in
src/mathlib/BUILD
. For example,
dependencies=[ "./folder1:folder1" ]
discards code that isn't depended on by source code in
folder1
. If it is in a subdirectory like
folder1/folder_a
and is not imported by some source code in
folder1
, it's not included. a. This makes sense for an application - don't include vestigial code. But here the directories serve to organize a collection of useful functions. How do I ensure that all code within mathlib is packaged up and exported together? Ideally I don't have to explicitly write out each file or directory that I want included. 2. When testing out the packaged mathlib I've found that some import behavior differs from what I experience when using the source code directly. Normally, I can import a directory and then access its contents (for example,
from mathlib import folder1
then
folder1.file.function()
). However, I'm finding that when I do this after installing the packaged wheel I get
AttributeError: module 'folder1' has no attribute 'file'
. a. What am I doing wrong? 3. Pants sensibly doesn't want two distributions to own the same file - so I get a multiple owners error on files in
src/apps/lib
that both app1.py and app2.py depend on. a. Do I need to package up and distribute
src/apps/lib
as well? Is there a better way to organize and package up the apps I'd like to distribute? 4. As far as I understand, the entry_point argument tells Pants (who tells setuptools) to generate an executable called app1 that will run app1.main(). When I install the package to a venv in Windows using pip, a file called app1.exe is placed in
.venv/Scripts
. Running this file does nothing. a. Is this due to an issue with how I've built the file? I sense that there are some basic ideas about Pants and/or packaging that I'm missing. Any help would be great. Thanks in advance for your time! Side note: In reviewing some past questions we asked I noticed that I didn't pick up on some good folder structure advice, I may have to do some further restructuring after I better understand above problems.
b
For 2.a, do you have
__init__.py
files that do something like
from . import file
? Are they in the final wheel? (it's just a zip file and so can be introspected)
f
Thanks for the help! I did not. I was under the impression Python 3 did not require
__init__.py
files for importing. Including one and importing (
from . import file
) seems to work, but I don't understand why. It seems that I have to add an
__init__.py
file in each folder and import every sibling submodule all the way down? Also note I was a little incomplete in my description of 2 above. I could run
import mathlib.folder1.file
. But I could not use
import mathlib
then
mathlib.folder1.file
. I've amended the description above.
b
It seems that I have to add an
__init__.py
file in each folder and import every sibling submodule all the way down?
Yeah, nested modules are only loaded lazily, and an
__init__.py
is one way to force them. I think
from . import *
can work although may not be explicit enough for you. (I can't help with your other questions, sorry)
f
Ahh I see. Thank you very much, one down!
In case someone with similar problems stumbles upon this, all the problems listed above were eventually solved. 2. As noted above: an
__init__.py
file importing all sibling modules that I wanted accessible at each level did the trick. Note that I needed to explicitly import each module,
from . import my_module
instead of
from . import *
. 1. When I had fixed #2 as above, Pants included everything I had explicitly imported 3. I ended up just packaging all of apps and defining two entry points. For us the difference is immaterial (I think). 4. There was a bug in the code of app1.py 🙂. So it wasn't due to how I set up Pants.
👍 1