https://pantsbuild.org/ logo
Join Slack
Powered by
Hi all. I’m using pants (poorly, unfortunately) an...
# general
c
Hi all. I’m using pants (poorly, unfortunately) and I’d like to improve what I’m doing. I’ve got it installed and running and have been using the 
./pants fmt ::
with success. I also use 
./pants lint ::
, but with mixed results. It helps me find various code issues, but I get this: “E0401: Unable to import ‘<some required module>’ (import-error)” for all my required modules. My project is setup with a remote Python interpreter (Python 3.10.8) in a Docker container. I’m also using Poetry to manage the dependencies. I’m not sure what to do to resolve this, other than tell pylint to stop reporting on E0401 errors.
e
Do your 
./pants roots
source roots match what you'd expect? https://www.pantsbuild.org/docs/source-roots Pants needs to be able to differentiate Python package directories from the directories they are nested under. Basically you need to tell Pants your 1st party 
PYTHONPATH
.
It could infer this from Poetry project metadata but does not currently. You'll add that support though once you become a happy customer 😉
c
@enough-analyst-54434, thanks for the link, but pants continues to make me feel really dumb. 😞 My project looks pretty much like this:
Copy code
my_repo_name
    └── project
        ├── __init__.py
        ├── app # bulk of code in dir
            ├── __init__.py
        ├── pyproject.toml
        └── Dockerfile
I’ve tried all kinds of variations recommended in the link you sent and I’m still getting “Unable to import”, along with a ton of “ambiguity” messages prior to pylint’s output.
Update, using this:
Copy code
[source]
root_patterns = [
    "project/app",
    "tests",
]
in my pants.toml file removes all the ambiguity messages, but pylint is still generating “Unable to import” messages.
e
Ok, so give an example of a valid import. Should 
<http://project.app|project.app>
be importable? Your 
__init__.py
in that tree imply it.
Or is the 
project/__init__.py
file spurious?
Your source roots claim it is spurious since the root is beneath it. So they contradict right now.
If at all possible linking to an open source repo is always a super quick way to get to the bottom of things. Sometimes someone on your end doesn't know the relevant info to convey and someone on my end isn't asking the right 20 questions.
Also, error messages are useful! Include them in general. They often give important details to someone with the right eye.
c
@enough-analyst-54434 I very much appreciate your attention to this. 🙂 Here are some sample errors:
Copy code
project/app/endpoints/schedules/v1/crud.py:7:0: E0401: Unable to import 'fastapi' (import-error)
project/app/endpoints/schedules/v1/crud.py:8:0: E0401: Unable to import 'sqlmodel' (import-error)
project/app/endpoints/schedules/v1/crud.py:9:0: E0401: Unable to import 'sqlmodel.ext.asyncio.session' (import-error)
pylint is having no problem finding/importing the modules I’ve built for the project. It’s only choking on 3rd party modules, like fastapi above.
e
Ok. So the contradiction I mentioned above is something to keep an eye on, but great - just 3rdparty import errors. How have you told Pants where to find your 3rdparty requirements? Can you share 
pants.toml
in it's entirety? Do you have a 
poetry_requirements
target in a BUILD file somewhere and have you used 
./pants tailor
or not?
c
Here is my pants.toml file:
Copy code
[GLOBAL]
pants_version = "2.14.0"

backend_packages = [
#    "pants.backend.docker",
#    "pants.backend.docker.lint.hadolint",
    "pants.backend.python",
    "pants.backend.build_files.fmt.black",
    "pants.backend.python.lint.black",
    "pants.backend.python.lint.isort",
    "pants.backend.python.lint.pylint",
    "pants.backend.python.typecheck.mypy",
]


[python]
tailor_pex_binary_targets = false
interpreter_constraints = ["CPython==3.9.*"]

[source]
root_patterns = [
    "project/app",
    "tests",
]

[anonymous-telemetry]
enabled = false

[pylint]
config = "config/.pylintrc"

[test]
output = "all"

[pytest]
extra_requirements.add = []
I don’t have a 
poetry_requirements
in there, but I have run 
./pants tailor
pretty regularly
e
Aha, that's killiing you then, Pants has 0 clues right now about info in your pyproject.toml right now.
Again, when you are down the road and a happy customer, clearly Pants could infer this and save someone else the pain and you'll add support for auto-inference of this setup stuff you missed. Between tailor plugins and plugins in general, it's not too bad to add support for things like this.
I guess a good general strategy is to recognize that Pants is a general build tool. It's true Pants v2 shipped Python support 1st and that has had the most polish, but in general, if you hit a problem its good to assume you have info Pants does not. I think the magic of dependency inference leads people to assume Pants can infer everything about Python, Java, Go, ... etc projects. Ideally it could, but often there are just time and effort holes waiting to be filled to auto-infer more and more. More rarely the inference is too expensive to implement performantly.
c
I kind of figured I was not telling pants what it needed to know. My problem is consistently not understanding what the documentation is trying to tell me and what I’m supposed to do about it. As in this case, I’m reading the link you sent (and have read it before), but I’m just not grokking what specifically I’m supposed to do/add in the 
pants.toml
file.
e
Literally plop this BUILD next to pyproject.toml:
Copy code
poetry_requirements()
That is all.
c
Really? Does it actually say that somewhere in the documentation? Maybe I need to read it again to see it….
e
It's super-subtle but look at the example box in the 1st link - it has two tabs
That is a bit hidden.
image.png
In general the tabbed examples can lead to this confusion. Cute but probably a net negative.
👍 1
c
I’ve been down this road previously, like a month ago when I was first trying out pants. Here is my current BUILD file next to 
pyproject.toml
Copy code
poetry_requirements(name="poetry")

poetry_requirements(
    name="dynaconf", overrides={"dynaconf": {"dependencies": [":setuptools"]}}
)

poetry_requirements(
    name="fastapi", overrides={"fastapi": {"dependencies": [":requests"]}}
)
When I run 
./pants lint ::
it generates the “unable to import errors” If I edit the BUILD to this:
Copy code
poetry_requirements(name="poetry")
And run 
./pants lint ::
the imports work and I’m down to issues in my code. 😉
The removed lines were something I was trying to resolve about Dynaconf not having information about its own dependencies.
e
So, you're not quite getting overrides - the idea is they allow you to decalre everything in 1 target.
So you'd add all the overrides to a single 
poetry_requirements
target.
Not use 3 
poetry_requirements
targets
c
Ahhhh
e
You've mainly got it it seems to me and were just missing a few small things. Again though - watch those source roots! They look strange! Or the places you have some 
__init__.py
look strange.
c
Will do, thanks! I did try to combine things to one 
poetry_requirements(…
Copy code
poetry_requirements(
    name="poetry",
    name="dynaconf", overrides={"dynaconf": {"dependencies": [":setuptools"]}},
    name="fastapi", overrides={"fastapi": {"dependencies": [":requests"]}}
)
but it didn’t like this because of the duplicate “name”. Can you give me a hint how I should approach this?
e
no, 1 target, 1 name, 1 overrides field, 1 map with many keys
c
Can you say D’OH!! I know I can, and often do… 😉
e
Copy code
poetry_requirements(
    name="poetry",
    overrides={
        "dynaconf": {"dependencies": [":setuptools"]},
        "fastapi": {"dependencies": [":requests"]}
    }
)
c
Just tried that as you were typing it, got this:
Copy code
./pants lint ::  
12:47:44.45 [INFO] Completed: Format with isort - isort made no changes.
12:47:44.45 [WARN] Completed: Format with Black - black made changes.
  project/app/endpoints/schedules/v1/crud.py
  project/app/endpoints/schedules/v1/routes.py
  project/app/endpoints/users/v1/crud.py
  project/app/endpoints/users/v1/routes.py
  project/app/models.py
  project/app/server.py
12:47:44.47 [WARN] Pants cannot infer owners for the following imports in the target project/app/endpoints/services/v1/transactions/routes.py:

  * project.app.common.tools.get_common_parameters (line: 7)
  * project.app.endpoints.auth.routes.authenticate (line: 8)

If you do not expect an import to be inferrable, add `# pants: no-infer-dep` to the import line. Otherwise, see <https://www.pantsbuild.org/v2.14/docs/troubleshooting#import-errors-and-missing-dependencies> for common problems.
12:47:44.48 [ERROR] 1 Exception encountered:

  ResolveError: The address project:requests from the `dependencies` field of the target project:poetry#fastapi does not exist.

The target name ':requests' is not defined in the directory project. Did you mean one of these target names?

  * :poetry
e
What is the path of the BUILD file containing the poetry_requirements target in your repo?
project/BUILD
?
Do you have code that imports 
requests
?
c
project/BUILD
That’s the absolute path
e
Yeah, just paths from the build root (repo root) are relevant
✅ 1
So the implication is you have no requests dependency defined in pyproject.toml
Is that true?
c
I don’t have anything that imports requests, the `fastapi`module does apparently
e
Yeah that requirement needs to go in pyproject.toml
You're saying with `:requests`it's in there.
c
Actually, I do have a 
requests
line in my pyproject.toml file. I must have done that when I was trying to figure this out a while back.
e
There is another way too:
Copy code
...
python_requirement(name="requests", requirements=["requests==1.2.3"])
So same BUILD, but external to pyproject.toml
Really? Can you share pyproject.toml then?
Oh, oh.
So
c
Sure thing:
Copy code
[tool.poetry]
name = "recurring-payment"
version = "0.1.0"
description = "Service to provide a REST API to create, manage and access recurring payments to bridgepay"
license = "MIT"
authors = ["dfarrell <dfarrell@rectanglehealth.com>"]
readme = "README.md"
homepage = "<https://github.com/rectanglehealth/recurring-payment>"
repository = "<https://github.com/rectanglehealth/recurring-payment>"
packages = [{include = "recurring_payment"}]

[tool.poetry.dependencies]
python = "^3.10.8"
structlog = "^22.1.0"
uvicorn = "^0.19.0"
python-dotenv = "^0.21.0"
boto3 = "^1.20.0"
python-multipart = "^0.0.5"
pyjwt = {extras = ["crypto"], version = "^2.6.0"}
python-jose = {extras = ["cryptography"], version = "^3.3.0"}
toml = "^0.10.2"
dynaconf = "^3.1.11"
fastapi = "^0.86.0"
requests = "^2.28.1"
rocketry = "^2.4.1"
sqlmodel = "^0.0.8"
asyncmy = "^0.2.5"

[tool.poetry.group.dev.dependencies]
pytest-fastapi = "^0.1.0"
pytest-cov = "^4.0.0"
pytest = "^7.2.0"

[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"

[tool.poetry.scripts]
recurring-payment = "app:main"

[tool.isort]
profile = "black"
line_length = 100
skip_glob = "tests"

[tool.black]
line-length = 100
extend-exclude = "tests"

[tool.pylint."MASTER"]
extension-pkg-whitelist=[
    "pydantic",
    "pydantic.typing"
]

[tool.pytest.ini_options]
addopts = [
    "--import-mode=importlib",
]
e
It's 
:poetry#requests
not 
:requests
To refer to a dep in the local 
:poetry
target, you address with 
#
So ~any target with a pluralized target name (poetry_requirements) is a "target generator" and to refer to the individual targets it generates behind the scenes you need the 2-level dereference of 
:<target generator>#<subtarget>
Another definitely confusing thing.
🤷‍♂️ 1
c
Here’s my updated 
BUILD
file:
Copy code
poetry_requirements(
    name="poetry",
    overrides={
        "fastapi": {"dependencies": [":poetry#requests"]},
    }
)
Which looks like it works when I run 
./pants lint ::
e
Great.
If you had happened to 
./pants list project ::
the address syntax would've come to the fore.
c
Thanks John. I very much appreciate your help with this!! I’m going to keep hammering on with pants till I get more independent with it. 🙂
e
You should give that a spin. It's a good gut-check.
c
Just did…
And I see what you mean
e
There are other useful introspection goals like 
dependencies
and 
filedeps
.
c
I’ll be working on getting smarter with them.
e
I was a Pants early adopter and found those invaluable to add and use in the way back.
I still get confused and still use them. Just had a heap of trouble setting up Pants in a new project that admittedly asks strange things of Pants.
c
It’s been a “treasure hunt” for me certainly, must have been the wild west in the way back machine
e
Oh I assure you.
c
LOL!
13 Views
Open in Slack
PreviousNext
Powered by