https://pantsbuild.org/ logo
Powered by
a

aloof-appointment-30987

11/23/2022, 5:58 PM
I'm struggling with something that seems like it should be a common objective. I have a monorepo where shared libraries are in a 
libraries
directory. I'd like to be able to reference these relatively so that apps and libraries are using the latest source code rather than being referenced via a version-pinned requirement reference. In poetry, I can use path references in pyproject.toml.
Copy code
[tool.poetry.dependencies]
quoteoftheday = {path = "../quoteoftheday"}
I'd like to configure 
path
BUILD files to reflect this. I find my Google searches pointing me to 
python_library
in the v1 documentation quite often. Can someone point me toward a solid resource to understand 1. How to properly configure a library distribution 2. How to reference that as a dependency from another project in the same monorepo
w

wide-midnight-78598

11/23/2022, 6:00 PM
Something like this?
Copy code
python_sources(
    name="libhelloworld",
    sources=["**/*.py"],
    dependencies=[
        "examples/python/core:libcore",
    ],
)
a

aloof-appointment-30987

11/23/2022, 6:01 PM
Yes then what does the BUILD look like at examples/python/core?
p

polite-garden-50641

11/23/2022, 6:05 PM
If you are using v2 all you probably need to to properly configure your source roots.
then run 
./pants tailor
and pants dependency inference should be able to do the rest for you.
👍 1
w

wide-midnight-78598

11/23/2022, 6:07 PM
My example came from here: https://github.com/sureshjoshi/pants-plugins A bit more hand-written, since I've been moving and nesting, and haven't re-tailored anything yet
In my example, it's something like this - blissfully unaware of anything else,
Copy code
python_sources(
    name="libcore",
    sources=["**/*.py"],
)
a

aloof-appointment-30987

11/23/2022, 6:11 PM
Thank you. Let me play with that.
👍 1
w

wide-midnight-78598

11/23/2022, 6:15 PM
Check out the source roots in 
pants.toml
- also note that I might do things a little differently to the community, as I don't usually use 
tailor
🙏 1
h

happy-kitchen-89482

11/23/2022, 8:43 PM
There is a paradigm shift that needs to occur when switching from poetry/pip-based setups to Pants. Pants figures out the deps (both in-repo and external) for you, you just need to tell it what the universe of allowed external deps is, and it uses dep inference to go the rest of the way.
So in your “2. How to reference that as a dependency from another project in the same monorepo” - the answer is “you don’t need to!“ 🙂
w

wide-midnight-78598

11/23/2022, 8:50 PM
Ah, good point Benjy! In my example, I think I wanted to make it implicit, but for some reason the dep inference wasn't picking it up correctly, and I didn't feel like debugging while I was moving and re-naming files. Eventually I would like to remove that (though, for example code, I'll probably leave it there commented out - to show what the inference picks up)
a

aloof-appointment-30987

11/23/2022, 10:35 PM
Thanks for the additional info. I was able to figure out some of the pathing through trial and error. Referencing modules from the monorepo root was the first success. I followed @polite-garden-50641’s advice and configured source roots, imitating @wide-midnight-78598’s config and pattern using 
marker_filenames = ["BUILD.pants"]
It was not clear that this essentially sets up a global search path that all dependees use to resolve import statements. Is that correct? I found I can now remove 
dependencies=
from the 
python_sources
config and inference works. I can then import relative to the sub directory(s) included by the library BUILD.pants as that marks the project root as a source root. This is wicked cool! I have a couple of questions: • Let's say someone really likes to have their source files in a _project_name/src/project_name_ (or maybe we're migrating an existing repo to pants). How do we configure the BUILD file or otherwise instruct pants to omit the src directory? The goal being to be able to import starting with the contents of src. • What is a good documentation / resource to dig deep into some of these concepts? Assume I am relatively new to Python pathing and dependency resolution in general (assumed correctly 😉) Thanks so much, BTW!
Yowsers, I may have figured this out... Please let me know if I am following convention. My goal is to take the path of least resistance. My goal is to be able to support a monorepo tree such as:
Copy code
├── apps
│   └── python
│       ├── example_app
│       └── monorepo_util
├── libraries
│   └── python
│       ├── another_library
│       ├── content_module
│       └── quoteoftheday
├── pants
└── pants.toml
I'd like to support imports relative to each library project source root. 
from content_module import content
IOW, the imports should not have to change if one decides to publish the libs to a package repo and source them using pip. Given the _content_module_ project sub directory tree:
Copy code
├── BUILD
├── README.md
├── src
│   ├── BUILD.pants
│   └── content_module
│       ├── __init__.py
│       └── content.py
└── tests
    ├── BUILD
    ├── __init__.py
    └── test_content_module.py
where 
./BUILD
contains:
Copy code
python_sources(
    dependencies="./src:content-module"
)
and 
./src/BUILD.pants
contains:
Copy code
python_sources(
    name="content-module",
    sources=["**/*.py"],
)
A dependee (located 
,,/../apps/python/example_app/app.py
relative to the above library project) is able to import as:
Copy code
from content_module import content
Am I configuring these things appropriately?
Note that 
Build.pants
is configured in 
./pants.toml
as
Copy code
[source]
marker_filenames = ["BUILD.pants"]
h

happy-kitchen-89482

11/25/2022, 2:01 PM
Re “It was not clear that this essentially sets up a global search path that all dependees use to resolve import statements. Is that correct? ” That is correct! Source roots have other uses, but that is a huge one.
So source roots are absolutely critical to your first question there: If you have 
project1/src/foo/bar.py
and you want your code to 
from foo import bar
then 
project1/src/
is a source root (i.e., must be on the sys.path of the python interpreter doing the importing) and so Pants must be told of this…
You are configuring this correctly for this, it looks like
However I don’t think you need that 
./BUILD
? I don’t see any sources in that dir
Note that if you go the marker filenames route for source root config then you have to have a BUILD.pants in every source root, and not in any subdirs of those source roots, which is fine as long as you stick to that. I don’t think I’ve seen BUILD files used as marker files, because often people do need BUILD files in subdirs as well. Typically the marker files are useful when you’re bringing together a bunch of existing projects, and so 
setup.py
or 
pyproject.toml
are useful as marker files.
This is the essential reading on source roots: https://www.pantsbuild.org/docs/source-roots
w

wide-midnight-78598

11/25/2022, 2:09 PM
don’t think I’ve seen BUILD files used as marker files
Except from this guy 👍 👍 I use them as project delineators - as mentioned above, I do some things a little differently
a

aloof-appointment-30987

11/25/2022, 2:15 PM
Thank you! That all makes sense. The root 
BUILD
creates a dist but I think you are correct in that I could move that into ./src. Using BUILD.pants as the source root marker appealed to me because it’s explicit and gets around the src sub directory issue.
h

happy-kitchen-89482

11/25/2022, 2:21 PM
If it works it works! Just remember that if you ever want a BUILD file in a subdir of a source root (e.g., to override some settings on the files in that subdir), then don’t call it BUILD.pants
Or, just keep those overrides and tweaks in the existing BUILD.pants at the root
is probably better and less confusing in your case
6 Views
Powered by