What's the correct syntax for using a local packag...
# general
b
What's the correct syntax for using a local package that has a
setup.py
? The use case here is moving code to a monorepo, and trying to use code that was previously installed with a
pip install
. From some old Github conversations, I believe this should be possible, but I couldn't find documentation for how to set things up. Details in 🧵.
My monorepo directory structure looks like this:
Copy code
src
|- common
|  |- packageA
|     |- BUILD
|     |- __init__.py
|     |- packageA_class.py
|     |- ...
|  |- packageB
|     |- packageB
|        |- BUILD
|        |- __init__.py
|        |- packageB_class.py
|        |- ...
|     |- setup.py
|     |- BUILD
|     |- ...
In this case,
packageB
is coming from another repo. In a non-monorepo world, I would simply
pip install packageB
, and use
import packageB
in packageA's code. However, once I moved it to my monorepo, I can't get the dependency coded correctly in
packageA
. Here's my `src/common/packageB/BUILD`:
Copy code
python_sources()

python_distribution(
    name="packageB_dist",
    provides=setup_py(
        setup_script="setup.py",
        name="packageB",
        version="0.2.0",
    ),
)
And `src/common/packageA/BUILD`:
Copy code
python_sources()

pex_binary(
    name="packageA_bin",
    entry_point="packageA_class.py",
    dependencies=["src/common/packageB:packageB_dist"]
)
My expectation here is that when I try to build
packageA
, it will use `packageB`'s
setup.py
to install it, and then I can do a simple
import common.packageB
, but that gives me a
ModuleNotFoundError
. What am I missing here to get this use case working? Running
./pants dependencies src/common/packageA
correctly indicates the problem:
Copy code
13:23:53.21 [WARN] Pants cannot infer owners for the following imports in the target src/common/packageA/packageA_class.py:

  * common.packageB (line: 1)

If you do not expect an import to be inferrable, add `# pants: no-infer-dep` to the import line. Otherwise, see <https://www.pantsbuild.org/v2.14/docs/troubleshooting#import-errors-and-missing-dependencies> for common problems.
src/common/packageA/packageA_class.py
src/common/packageB:packageB_dist
So it sees the packageB distribution, but cannot resolve its import.
h
Hi! Sorry for the trouble. Just to clarify the context: In a monorepo you shouldn’t usually need to build packageB via a setup.py in order to consume it in packageA. Indeed that is one of the advantages of a monorepo! It is possible to do so in Pants (for example if packageB contains native code that has to be built via a setup.py), but it’s not idiomatic. So before troubleshooting this I just wanted to double-check that it’s necessary? (Note that you can still build and publish a dist from packageB for other repos to consume, while consuming it directly from source inside the monorepo)
All that said, Pants does support building “local dists” and consuming them. Try moving that dependency on
src/common/packageB:packageB_dist
to A’s
python_sources
target. It’s the code in that target that is actually consuming packageB via an import.
đź‘€ 1
The thing I’m not sure about is that Pants should be able to infer a source-level dep because of that
common.packageB
import, so not clear why that is failing.
Oh, what does
./pants roots
show?
b
Going in reverse order:
./pants roots
correctly shows
src
as a root, I'm using the
import common.whatever
elsewhere succesfully
h
It does look like you have either a source roots issue or an import path issue:
src/common/packageB
has no
__init__.py
, so I wouldn’t expect you to be able to import from it
So that explains why dep inference fails
âś… 1
b
The main reason I kept the
setup.py
for now is laziness - I expected it to keep everything nice, without me having to mess with the BUILD too much. But there's nothing really special in there, I believe I can get away without it
h
I think if you move packageB up a level and remove that dep on the python_distribution, things will just work?
Without invoking the setup.py
(and remove the setup.py if you move things up a level)
Or you can add
src/common/packageB
as a source root
b
By "up a level" you mean move the current
common/packageB/packageB/
(where all the code lives) to just
common/packageB/
?
h
exactly, I think that’s what you had in mind by importing
common.packageB
?
The alternative, if you do want to stick with running the
setup.py
is to add
src/common/packageB
as a source root
But note that then the import will be
import packageB
, not
import common.packageB
but presumably
import packageB
was how packageA was importing it before the monorepo?
b
Yes exactly, as if I were `pip install`ing it. I think you are correct - I tried
import common.packageB.packageB
and got past that first import, but bombed deeper in the code where there are old `import packageB`s that I need to fix. I think I have a few options to try, let me play around and let you know.
Thanks for your help!
❤️ 1
h
You’re welcome!
b
I'm so excited - now my code failed in a completely different place for an unrelated reason! Thank you once again for your help!
❤️ 1
🤣 1
h
haha, nice