https://pantsbuild.org/ logo
#general
Title
# general
c

chilly-tailor-75063

12/09/2022, 3:02 PM
Hi all. I’m using pants (poorly, unfortunately) and I’d like to improve what I’m doing. I’ve got it installed and running and have been using the
./pants fmt ::
with success. I also use
./pants lint ::
, but with mixed results. It helps me find various code issues, but I get this: “E0401: Unable to import ‘<some required module>’ (import-error)” for all my required modules. My project is setup with a remote Python interpreter (Python 3.10.8) in a Docker container. I’m also using Poetry to manage the dependencies. I’m not sure what to do to resolve this, other than tell pylint to stop reporting on E0401 errors.
e

enough-analyst-54434

12/09/2022, 3:16 PM
Do your
./pants roots
source roots match what you'd expect? https://www.pantsbuild.org/docs/source-roots Pants needs to be able to differentiate Python package directories from the directories they are nested under. Basically you need to tell Pants your 1st party
PYTHONPATH
.
It could infer this from Poetry project metadata but does not currently. You'll add that support though once you become a happy customer 😉
c

chilly-tailor-75063

12/09/2022, 4:31 PM
@enough-analyst-54434, thanks for the link, but pants continues to make me feel really dumb. 😞 My project looks pretty much like this:
Copy code
my_repo_name
    └── project
        ├── __init__.py
        ├── app # bulk of code in dir
            ├── __init__.py
        ├── pyproject.toml
        └── Dockerfile
I’ve tried all kinds of variations recommended in the link you sent and I’m still getting “Unable to import”, along with a ton of “ambiguity” messages prior to pylint’s output.
Update, using this:
Copy code
[source]
root_patterns = [
    "project/app",
    "tests",
]
in my pants.toml file removes all the ambiguity messages, but pylint is still generating “Unable to import” messages.
e

enough-analyst-54434

12/09/2022, 4:43 PM
Ok, so give an example of a valid import. Should
<http://project.app|project.app>
be importable? Your
__init__.py
in that tree imply it.
Or is the
project/__init__.py
file spurious?
Your source roots claim it is spurious since the root is beneath it. So they contradict right now.
If at all possible linking to an open source repo is always a super quick way to get to the bottom of things. Sometimes someone on your end doesn't know the relevant info to convey and someone on my end isn't asking the right 20 questions.
Also, error messages are useful! Include them in general. They often give important details to someone with the right eye.
c

chilly-tailor-75063

12/09/2022, 5:18 PM
@enough-analyst-54434 I very much appreciate your attention to this. 🙂 Here are some sample errors:
Copy code
project/app/endpoints/schedules/v1/crud.py:7:0: E0401: Unable to import 'fastapi' (import-error)
project/app/endpoints/schedules/v1/crud.py:8:0: E0401: Unable to import 'sqlmodel' (import-error)
project/app/endpoints/schedules/v1/crud.py:9:0: E0401: Unable to import 'sqlmodel.ext.asyncio.session' (import-error)
pylint is having no problem finding/importing the modules I’ve built for the project. It’s only choking on 3rd party modules, like fastapi above.
e

enough-analyst-54434

12/09/2022, 5:20 PM
Ok. So the contradiction I mentioned above is something to keep an eye on, but great - just 3rdparty import errors. How have you told Pants where to find your 3rdparty requirements? Can you share
pants.toml
in it's entirety? Do you have a
poetry_requirements
target in a BUILD file somewhere and have you used
./pants tailor
or not?
c

chilly-tailor-75063

12/09/2022, 5:22 PM
Here is my pants.toml file:
Copy code
[GLOBAL]
pants_version = "2.14.0"

backend_packages = [
#    "pants.backend.docker",
#    "pants.backend.docker.lint.hadolint",
    "pants.backend.python",
    "pants.backend.build_files.fmt.black",
    "pants.backend.python.lint.black",
    "pants.backend.python.lint.isort",
    "pants.backend.python.lint.pylint",
    "pants.backend.python.typecheck.mypy",
]


[python]
tailor_pex_binary_targets = false
interpreter_constraints = ["CPython==3.9.*"]

[source]
root_patterns = [
    "project/app",
    "tests",
]

[anonymous-telemetry]
enabled = false

[pylint]
config = "config/.pylintrc"

[test]
output = "all"

[pytest]
extra_requirements.add = []
I don’t have a
poetry_requirements
in there, but I have run
./pants tailor
pretty regularly
e

enough-analyst-54434

12/09/2022, 5:22 PM
Aha, that's killiing you then, Pants has 0 clues right now about info in your pyproject.toml right now.
Again, when you are down the road and a happy customer, clearly Pants could infer this and save someone else the pain and you'll add support for auto-inference of this setup stuff you missed. Between tailor plugins and plugins in general, it's not too bad to add support for things like this.
I guess a good general strategy is to recognize that Pants is a general build tool. It's true Pants v2 shipped Python support 1st and that has had the most polish, but in general, if you hit a problem its good to assume you have info Pants does not. I think the magic of dependency inference leads people to assume Pants can infer everything about Python, Java, Go, ... etc projects. Ideally it could, but often there are just time and effort holes waiting to be filled to auto-infer more and more. More rarely the inference is too expensive to implement performantly.
c

chilly-tailor-75063

12/09/2022, 5:30 PM
I kind of figured I was not telling pants what it needed to know. My problem is consistently not understanding what the documentation is trying to tell me and what I’m supposed to do about it. As in this case, I’m reading the link you sent (and have read it before), but I’m just not grokking what specifically I’m supposed to do/add in the
pants.toml
file.
e

enough-analyst-54434

12/09/2022, 5:30 PM
Literally plop this BUILD next to pyproject.toml:
Copy code
poetry_requirements()
That is all.
c

chilly-tailor-75063

12/09/2022, 5:32 PM
Really? Does it actually say that somewhere in the documentation? Maybe I need to read it again to see it….
e

enough-analyst-54434

12/09/2022, 5:32 PM
It's super-subtle but look at the example box in the 1st link - it has two tabs
That is a bit hidden.
image.png
In general the tabbed examples can lead to this confusion. Cute but probably a net negative.
👍 1
c

chilly-tailor-75063

12/09/2022, 5:39 PM
I’ve been down this road previously, like a month ago when I was first trying out pants. Here is my current BUILD file next to
pyproject.toml
Copy code
poetry_requirements(name="poetry")

poetry_requirements(
    name="dynaconf", overrides={"dynaconf": {"dependencies": [":setuptools"]}}
)

poetry_requirements(
    name="fastapi", overrides={"fastapi": {"dependencies": [":requests"]}}
)
When I run
./pants lint ::
it generates the “unable to import errors” If I edit the BUILD to this:
Copy code
poetry_requirements(name="poetry")
And run
./pants lint ::
the imports work and I’m down to issues in my code. 😉
The removed lines were something I was trying to resolve about Dynaconf not having information about its own dependencies.
e

enough-analyst-54434

12/09/2022, 5:40 PM
So, you're not quite getting overrides - the idea is they allow you to decalre everything in 1 target.
So you'd add all the overrides to a single
poetry_requirements
target.
Not use 3
poetry_requirements
targets
c

chilly-tailor-75063

12/09/2022, 5:41 PM
Ahhhh
e

enough-analyst-54434

12/09/2022, 5:42 PM
You've mainly got it it seems to me and were just missing a few small things. Again though - watch those source roots! They look strange! Or the places you have some
__init__.py
look strange.
c

chilly-tailor-75063

12/09/2022, 5:45 PM
Will do, thanks! I did try to combine things to one
poetry_requirements(…
Copy code
poetry_requirements(
    name="poetry",
    name="dynaconf", overrides={"dynaconf": {"dependencies": [":setuptools"]}},
    name="fastapi", overrides={"fastapi": {"dependencies": [":requests"]}}
)
but it didn’t like this because of the duplicate “name”. Can you give me a hint how I should approach this?
e

enough-analyst-54434

12/09/2022, 5:45 PM
no, 1 target, 1 name, 1 overrides field, 1 map with many keys
c

chilly-tailor-75063

12/09/2022, 5:46 PM
Can you say D’OH!! I know I can, and often do… 😉
e

enough-analyst-54434

12/09/2022, 5:46 PM
Copy code
poetry_requirements(
    name="poetry",
    overrides={
        "dynaconf": {"dependencies": [":setuptools"]},
        "fastapi": {"dependencies": [":requests"]}
    }
)
c

chilly-tailor-75063

12/09/2022, 5:50 PM
Just tried that as you were typing it, got this:
Copy code
./pants lint ::  
12:47:44.45 [INFO] Completed: Format with isort - isort made no changes.
12:47:44.45 [WARN] Completed: Format with Black - black made changes.
  project/app/endpoints/schedules/v1/crud.py
  project/app/endpoints/schedules/v1/routes.py
  project/app/endpoints/users/v1/crud.py
  project/app/endpoints/users/v1/routes.py
  project/app/models.py
  project/app/server.py
12:47:44.47 [WARN] Pants cannot infer owners for the following imports in the target project/app/endpoints/services/v1/transactions/routes.py:

  * project.app.common.tools.get_common_parameters (line: 7)
  * project.app.endpoints.auth.routes.authenticate (line: 8)

If you do not expect an import to be inferrable, add `# pants: no-infer-dep` to the import line. Otherwise, see <https://www.pantsbuild.org/v2.14/docs/troubleshooting#import-errors-and-missing-dependencies> for common problems.
12:47:44.48 [ERROR] 1 Exception encountered:

  ResolveError: The address project:requests from the `dependencies` field of the target project:poetry#fastapi does not exist.

The target name ':requests' is not defined in the directory project. Did you mean one of these target names?

  * :poetry
e

enough-analyst-54434

12/09/2022, 5:51 PM
What is the path of the BUILD file containing the poetry_requirements target in your repo?
project/BUILD
?
Do you have code that imports
requests
?
c

chilly-tailor-75063

12/09/2022, 5:52 PM
project/BUILD
That’s the absolute path
e

enough-analyst-54434

12/09/2022, 5:52 PM
Yeah, just paths from the build root (repo root) are relevant
1
So the implication is you have no requests dependency defined in pyproject.toml
Is that true?
c

chilly-tailor-75063

12/09/2022, 5:54 PM
I don’t have anything that imports requests, the `fastapi`module does apparently
e

enough-analyst-54434

12/09/2022, 5:54 PM
Yeah that requirement needs to go in pyproject.toml
You're saying with `:requests`it's in there.
c

chilly-tailor-75063

12/09/2022, 5:55 PM
Actually, I do have a
requests
line in my pyproject.toml file. I must have done that when I was trying to figure this out a while back.
e

enough-analyst-54434

12/09/2022, 5:55 PM
There is another way too:
Copy code
...
python_requirement(name="requests", requirements=["requests==1.2.3"])
So same BUILD, but external to pyproject.toml
Really? Can you share pyproject.toml then?
Oh, oh.
So
c

chilly-tailor-75063

12/09/2022, 5:56 PM
Sure thing:
Copy code
[tool.poetry]
name = "recurring-payment"
version = "0.1.0"
description = "Service to provide a REST API to create, manage and access recurring payments to bridgepay"
license = "MIT"
authors = ["dfarrell <dfarrell@rectanglehealth.com>"]
readme = "README.md"
homepage = "<https://github.com/rectanglehealth/recurring-payment>"
repository = "<https://github.com/rectanglehealth/recurring-payment>"
packages = [{include = "recurring_payment"}]

[tool.poetry.dependencies]
python = "^3.10.8"
structlog = "^22.1.0"
uvicorn = "^0.19.0"
python-dotenv = "^0.21.0"
boto3 = "^1.20.0"
python-multipart = "^0.0.5"
pyjwt = {extras = ["crypto"], version = "^2.6.0"}
python-jose = {extras = ["cryptography"], version = "^3.3.0"}
toml = "^0.10.2"
dynaconf = "^3.1.11"
fastapi = "^0.86.0"
requests = "^2.28.1"
rocketry = "^2.4.1"
sqlmodel = "^0.0.8"
asyncmy = "^0.2.5"

[tool.poetry.group.dev.dependencies]
pytest-fastapi = "^0.1.0"
pytest-cov = "^4.0.0"
pytest = "^7.2.0"

[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"

[tool.poetry.scripts]
recurring-payment = "app:main"

[tool.isort]
profile = "black"
line_length = 100
skip_glob = "tests"

[tool.black]
line-length = 100
extend-exclude = "tests"

[tool.pylint."MASTER"]
extension-pkg-whitelist=[
    "pydantic",
    "pydantic.typing"
]

[tool.pytest.ini_options]
addopts = [
    "--import-mode=importlib",
]
e

enough-analyst-54434

12/09/2022, 5:57 PM
It's
:poetry#requests
not
:requests
To refer to a dep in the local
:poetry
target, you address with
#
So ~any target with a pluralized target name (poetry_requirements) is a "target generator" and to refer to the individual targets it generates behind the scenes you need the 2-level dereference of
:<target generator>#<subtarget>
Another definitely confusing thing.
🤷‍♂️ 1
c

chilly-tailor-75063

12/09/2022, 6:02 PM
Here’s my updated
BUILD
file:
Copy code
poetry_requirements(
    name="poetry",
    overrides={
        "fastapi": {"dependencies": [":poetry#requests"]},
    }
)
Which looks like it works when I run
./pants lint ::
e

enough-analyst-54434

12/09/2022, 6:04 PM
Great.
If you had happened to
./pants list project ::
the address syntax would've come to the fore.
c

chilly-tailor-75063

12/09/2022, 6:05 PM
Thanks John. I very much appreciate your help with this!! I’m going to keep hammering on with pants till I get more independent with it. 🙂
e

enough-analyst-54434

12/09/2022, 6:05 PM
You should give that a spin. It's a good gut-check.
c

chilly-tailor-75063

12/09/2022, 6:05 PM
Just did…
And I see what you mean
e

enough-analyst-54434

12/09/2022, 6:07 PM
There are other useful introspection goals like
dependencies
and
filedeps
.
c

chilly-tailor-75063

12/09/2022, 6:07 PM
I’ll be working on getting smarter with them.
e

enough-analyst-54434

12/09/2022, 6:08 PM
I was a Pants early adopter and found those invaluable to add and use in the way back.
I still get confused and still use them. Just had a heap of trouble setting up Pants in a new project that admittedly asks strange things of Pants.
c

chilly-tailor-75063

12/09/2022, 6:09 PM
It’s been a “treasure hunt” for me certainly, must have been the wild west in the way back machine
e

enough-analyst-54434

12/09/2022, 6:09 PM
Oh I assure you.
c

chilly-tailor-75063

12/09/2022, 6:09 PM
LOL!
13 Views