https://pantsbuild.org/ logo
Join Slack
Powered by
Not sure what I'm doing wrong--I thought my poetry...
# general
p
Not sure what I'm doing wrong--I thought my poetry dep in 
python-app
should pull in 
python-lib
in pants but it's not
Copy code
3rdparty
  python
    default.lock
python-lib
  BUILD
  pyproject.toml
  poetry.lock
  python_lib
    BUILD
    __init__.py
    file.py
    test_file.py
python-app
  BUILD
  pants.lock
  pyproject.toml
  poetry.lock
  python_app
    BUILD
    file.py
    test_file.py
python-lib/BUILD
Copy code
poetry_requirements(
    name="poetry",
    module_mapping={"boto3-stubs": ["mypy_boto3_dms"]},
)

python_distribution(
    name="python_lib",
    provides=python_artifact(
        name="python-lib",
        version="0.2.0",
    ),
    sdist=False,
    interpreter_constraints=[">=3.9,<4.0"],
    dependencies=[
        "python-lib/python_lib:python_lib"
    ],
)
python-app/BUILD
Copy code
__defaults__(all=dict(resolve="python-app"))

poetry_requirements(
    name="poetry",
    module_mapping={
        "psycopg2-binary": ["psycopg2"],
    },
)

python_aws_lambda_function(
    name="lambda",
    runtime="python3.11",
    # Pants will convert this to `project.lambda_example:example_handler`.
    handler="python_app/handler.py:lambda_handler",
    dependencies=[
        "python-app/python_app:python_app",
    ],
)
pants.toml
Copy code
[GLOBAL]
pants_version = "2.19.1"

backend_packages = [
    "pants.backend.awslambda.python",
    "pants.backend.docker",
    "pants.backend.docker.lint.hadolint",
    "pants.backend.python",
    "pants.backend.python.lint.black",
]

pants_ignore.add = ["venv"]

[docker]
# Requires "HOME" in build_args for macOS <https://github.com/pantsbuild/pants/issues/20605#issuecomment-2020309567>
use_buildx = true

# Pants doesn't currently supply git metadata (there's an open GH issue somewhere...), however,
# it can utilize environment variables. It also has an "init" script hook (.pants.bootstrap)
# where arbitrary environment variables can be set.
# See <https://pantsbuild.slack.com/archives/C046T6T9U/p1676292672358099>
build_args = ["GIT_COMMIT", "HOME"]

[hadolint]
config_discovery = true

[python]
enable_resolves = true
interpreter_constraints = ['>=3.11']

[python-repos]
find_links = ["%(pants_distdir)s"]
# This makes pants put relative paths in pants.lock instead of absolute
path_mappings = ["WHEELS_DIR|%(pants_distdir)s"]

[python.resolves]
python-default = "3rdparty/python/default.lock"
python-app = "python-app/pants.lock"

[source]
marker_filenames = ["pyproject.toml"]
I was under the impression I could add 
python-app2
later with a different resolve and also reference 
python-lib
under the assumption python-app and python-app2 run independently and may have overlapping or conflicting dependencies
If I try to run
Copy code
pants generate-lockfiles --resolve=python-app
I get 
No matching distribution found for python-lib
unless I run 
pants package python-lib:
first (this makes sense because the find_links config picks up the wheel), but I thought pants should handle resolving dependencies for me?
If I do 
pants package python-lib: ; pants generate-lockfiles --resolve=python-app
this works fine, but if I run 
pants test ::
on a fresh clone of the repo, it fails in 
python-app
No such file or directory: '/home/user/repo/dist/python_lib-0.1.0-py3-none-any.whl
(since the artifact/wheel dep isn't built first/automatically)
c
Resolves represent separate bubbles of possible dependencies. If python-lib and python-app live in separate resolves, Pants believes they must remain separate. Most commonly, the reason to use multiple resolves is because of different and incompatible dependency versions or python versions (interpreter constraints). Ex python 2 vs 3, or django 4 vs 5. If your packages simply have different requirements but are not incompatible (more precisely, a consistent resolution exists), you can put them in the same resolve. If one of your packages depends on the other, there necessarily exists some dependency versions that satisfy all their requirements, so you can put them in the same resolve. Pants can then make the dependency link and have python-app automatically pull in code from python-lib.
The way you use find_links is not the intended use of that param. It's intended to be forwarded to pip to use to fetch wheels or sdists, usually as part of self-hosted artifact stores.
One complication with putting all your python packages in the same resolve is when you have a lib that is compatible with multiple apps individually, but some apps are not compatible with each other. For example, a lib might support pythons 2 and 3, but a python 2 app cannot exist in the same resolve as a python 3 app. In that case, you will want to use multiple resolves. The documentation for lockfiles has more details about this, specifically the section on multiple lockfiles. The doc on multiple versions of the same dependency might also help.
p
Yeah, there's other things that I could go out of my way and try to make compatible but I was trying to avoid. Namely niquests (urllib3_future) and various versions of boto3 used in Docker containers and AWS Lambdas. Ideally this library can be shared across these independent Docker apps since it only has a single dep and a small amount of code
It seemed the easiest way was to treat the 1st party shared lib as a 3rd party dep and host a local repo with find_links (which also makes building Docker containers super easy) but it doesn't seem pants has a way to depend on a packaged artifact (only the source..?)
c
what a coincidence, I raised the issue or "urllib3-future" (dependency of "niquests") incompatibility with "docker" just last week and the maintainer fixed it within a day and released a new version (2.7.913). Let me see if I understand your situation correctly: When you say "various versions of boto3", I think that means that you have several application projects, some of which use different versions of boto3. But you have some common dependencies (3rd party and 1st party). When Pants is trying to solve for the dependency versions, it can't find a resolution that satisfies all the constraints (since there are conflicting versions of boto3). Pants would therefore require us to separate they into their own resolves, so each one can be solved. To share our common code, we'd then parametrise it for all of these resolves. Pants dependency inference will infer dependencies of 1st-party 
python_sources
, not packages. You can manually specify dependencies on the 
python_distribution
itself, although again they'd have to be in the same resolve (so, probably using parametrisation).
also for building docker containers, Pants can also do some dependency inference on anything pants can package, explained in the doc on adding dependencies. Does that help?
p
Yes, correct and that's what I've found
The Docker container sub projects seem to work with packaging a 1st party dep but Lambda projects don't
Ultimately the goal of this particular repo is lots of small utilities/scripts with a little bit of overlapping code but we didn't want to setup a ton of git repos and CI and 
pants
is working really well for Docker/Lambda packaging--just not sure the optimal approach given the majority of the code is independent
Put another way, the repo-wide version locking (with single resolve) isn't very helpful given most of the code isn't related and packaged separately
psycopg2
is also somewhat annoying given 
psycopg2
source dist vs 
psycopg2-binary
bin dist although that and urllib3-future are arguably problems with Python's lack of "provides" functionality with packages (so I don't expect pants to magically solve that for me)
Also maybe there's a better way with pex binaries? Haven't looked too deeply into that route
c
We've also got a repo that's a pile of small scripts, for managing infra. We put everything in the same resolve (after harmonising versions of dependencies). Pex binaries help, since they will only bundle the dependencies that they depend on. So even though the scripts aren't related, I don't find we pay a penalty for being in the same resolve when we put them in docker containers or ship them to VMs. (I'm not sure what the AWS Lambda experience is like. I thought we had dep inference that could pull in 
python_sources
, is that not working?)
12 Views
Open in Slack
PreviousNext
Powered by