https://pantsbuild.org/ logo
Join Slack
Powered by
Hey I’m trying to set up Pants with an example pro...
# general
b
Hey I’m trying to set up Pants with an example project but I’m my project structure is wrong (probably). What I got going is under the root of the git repo there are three folders where each folder is a poetry project. Each poetry project has got it’s own name, dependencies, etc. I tried to follow the instructions in the docs but when I reached the end pants is not able to understand the depenencies. How can I make this structure work with pants?
Here’s a tree of the directory
Copy code
monorepo
├── application
│   ├── BUILD
│   ├── README.md
│   ├── application
│   │   ├── BUILD
│   │   ├── __init__.py
│   │   └── main.py
│   ├── poetry.lock
│   ├── pyproject.toml
│   └── tests
│       ├── BUILD
│       ├── __init__.py
│       └── test_application.py
├── library
│   ├── BUILD
│   ├── README.md
│   ├── library
│   │   ├── BUILD
│   │   ├── __init__.py
│   │   └── main.py
│   ├── poetry.lock
│   ├── pyproject.toml
│   └── tests
│       ├── BUILD
│       ├── __init__.py
│       └── test_library.py
├── pants
├── pants.toml
└── service
    ├── BUILD
    ├── README.md
    ├── poetry.lock
    ├── pyproject.toml
    ├── service
    │   ├── BUILD
    │   ├── __init__.py
    │   └── main.py
    └── tests
        ├── BUILD
        ├── __init__.py
        └── test_service.py
and here’s the pants.toml
Copy code
[GLOBAL]
pants_version = "2.14.0"

backend_packages = [
    "pants.backend.python",
    "pants.backend.docker",
    "pants.backend.python.lint.bandit",
    "pants.backend.python.lint.black",
    "pants.backend.python.lint.flake8",
    "pants.backend.python.lint.isort",
    "pants.backend.python.typecheck.mypy",
]

[python-bootstrap]
search_path = ["/opt/homebrew/bin/"]

[source]
root_patterns = [
    "/application",
    "/library",
    "/service",
]

[anonymous-telemetry]
enabled = true
repo_id = "3063a0b6-cb3b-40a4-8221-2641e0ae0df4"
p
how do import statements in your code looks like?
b
Application
Copy code
from pyspark.sql import DataFrame as SparkDataFrame, SparkSession
import pandas as pd
from library.main import Run
Service
Copy code
from fastapi import FastAPI
from library.main import Run, get_run
from uuid import UUID
Library
Copy code
from pydantic import BaseModel
from pathlib import Path
from uuid import UUID
p
what is the output of running 
./pants dependencies <file path>
on the service file you mentioned above (the one with 
from library.main import Run, get_run
) ?
b
For service
Copy code
❯ ./pants dependencies service/service/main.py
09:28:10.21 [INFO] Initializing scheduler...
09:28:10.34 [INFO] Scheduler initialized.
library/library/main.py
service/service/__init__.py
service:poetry#fastapi
For application
Copy code
❯ ./pants dependencies application/application/main.py
application/application/__init__.py
For library
Copy code
❯ ./pants dependencies library/library/main.py
09:28:41.34 [WARN] The target library/library/main.py imports `pydantic.BaseModel`, but Pants cannot safely infer a dependency because more than one target owns this module, so it is ambiguous which to use: ['library:poetry#pydantic', 'service:poetry#pydantic'].

Please explicitly include the dependency you want in the `dependencies` field of library/library/main.py, or ignore the ones you do not want by prefixing with `!` or `!!` so that one or no targets are left.

Alternatively, you can remove the ambiguity by deleting/changing some of the targets so that only 1 target owns this module. Refer to <https://www.pantsbuild.org/v2.14/docs/troubleshooting#import-errors-and-missing-dependencies>.
09:28:41.35 [WARN] Pants cannot infer owners for the following imports in the target library/library/main.py:

  * pydantic.BaseModel (line: 1)

If you do not expect an import to be inferrable, add `# pants: no-infer-dep` to the import line. Otherwise, see <https://www.pantsbuild.org/v2.14/docs/troubleshooting#import-errors-and-missing-dependencies> for common problems.
library/library/__init__.py
p
do u have something like this: https://github.com/pantsbuild/example-python/blob/main/BUILD#L7 for your requirements.txt file ? and have you seen https://www.pantsbuild.org/docs/python-third-party-dependencies ?
b
There is no requirements.txt file. 
application/BUILD
is
Copy code
poetry_requirements(
    name="poetry",
)
application/application/BUILD
is
Copy code
python_sources()
The pattern is the same for 
library
and 
service
I have read the docs you posted. I find it really difficult to understand what’s going on, though, because pants doesn’t fit into like the poetry, pdm, pipenv model. And then it “works with poetry”. I can’t conceptualize where the tool fits conceptually so stuff like “Teaching Pants your “universe”(s) of dependencies” doesn’t make sense to me. Doesn’t it just read a requirements.txt or a poetry file for that?
p
ok, so running 
./pants dependencies libary:poetry
(basically on the 
poetry_requirements
that has the pydantic req should show u if pants sees that dep or not.
b
Copy code
❯ ./pants dependencies libary:poetry
09:42:00.17 [ERROR] 1 Exception encountered:

  ResolveError: The file or directory 'libary' does not exist on disk in the workspace, so the address 'libary:poetry' from CLI arguments cannot be resolved.
p
use 
./pants list ::
to get a list of targets for the project, this will tell you how pants see that specific target.
b
Wait 
library
was misspelled above
Copy code
❯ ./pants dependencies library:poetry
library/pyproject.toml:poetry
library:poetry#bandit
library:poetry#black
library:poetry#flake8
library:poetry#ipython
library:poetry#isort
library:poetry#mypy
library:poetry#pydantic
library:poetry#pytest
Copy code
❯ ./pants list ::
application:poetry
application:poetry#pandas
application:poetry#pyspark
application:poetry#pytest
application/pyproject.toml:poetry
application/application:application
application/application/__init__.py
application/application/main.py
application/tests:tests
application/tests/test_application.py
library:poetry
library:poetry#bandit
library:poetry#black
library:poetry#flake8
library:poetry#ipython
library:poetry#isort
library:poetry#mypy
library:poetry#pydantic
library:poetry#pytest
library/pyproject.toml:poetry
library/library:library
library/library/__init__.py
library/library/main.py
library/tests:tests
library/tests/test_library.py
service:poetry
service:poetry#bandit
service:poetry#black
service:poetry#fastapi
service:poetry#flake8
service:poetry#ipython
service:poetry#isort
service:poetry#json-logging
service:poetry#mypy
service:poetry#pydantic
service:poetry#pytest
service/pyproject.toml:poetry
service/service:service
service/service/__init__.py
service/service/main.py
service/tests:tests
service/tests/test_service.py
p
ah.. nvm... I see the issue now:
Copy code
09:28:41.34 [WARN] The target library/library/main.py imports `pydantic.BaseModel`, but Pants cannot safely infer a dependency because more than one target owns this module, so it is ambiguous which to use: ['library:poetry#pydantic', 'service:poetry#pydantic'].
so you have two requirements files (poetry files) that specify 
pydantic
pants doesn't know from which one of those it should get 
pydantic
b
One for the library the other for the service?
p
yes. try commenting out the 
poetry_requirements
target in one of those and try again.
b
Wait wait wait… So even though it read poetry tomls it is at the same level — it replaces — poetry, pdm, etc? Furthermore, it requires that all dependencies are managed at the top level (ie no dependencies should be managed on a per project basis)?
But when I was reading these docs it made it seem like it still uses the pyproject.toml on a per project basis (it said it reads the build section)?
But then it’s talking about setting up some pants specific non-pyproject.toml stuff?
h
Hi! Sorry for the confusion. I think my replies here might be relevant: https://pantsbuild.slack.com/archives/C046T6T9U/p1668107496568069
When we say Pants is compatible with Poetry, what we mean by that is that (using a 
poetry_requirements()
target in a BUILD file) we can read poetry-style requirements from a pyproject.toml.
If you 
pydantic
in multiple of those, then you’re introducing multiple possible sources that could provide 
pydantic
, so Pants doesn’t know which one you mean.
You can resolve this by adding an explicit dependency. But it’s more idiomatic to have a single top-level set of requirements that Pants then selects from as needed (as I explained in that other thread)
I really need to write this up in a post…
TBH we could also add disambiguation logic that says “if there are multiple providers of a module, take the closest one in the filesystem sense”
that would make this work more smoothly in your case
b
Yeah. I haven’t finished that thread you sent but it’s exactly what I needed (I would like it in documentation form).
h
It’s on my todo list to write up, for sure
it is the #1 thing that is confusing to newcomers to Pants, and that is our responsibility to clarify
So it’s not you, it’s us…
b
BTW thank you so much for being available on slack. This is incredible. I’m going to take a few days to read this and think about things again then get back. But, for now, it seems like the best (simplest) thing to do (in light of your “current python packaging is not friendly to monorepos” comment in the thread) is to just drop the entire poetry system and move to pants whole hog, all at once. One commit type thing. It’s not easy to manage some sort of half poetry half pants system as that’s just mixing oil and water. Is this right?
h
That sounds right to me. Sometimes you can’t cross a chasm in two jumps…
You can move incrementally, but it might be more effort than it’s worth
And generally clinging to a non-monorepo-friendly idiom when you’re trying to switch to a monorepo is not ideal
6 Views
Open in Slack
PreviousNext
Powered by