I understood the warning to mean: “hey, you’re importing something i don’t see in the graph”

yea... that sounds like an actual bug in pants mypy goal implementation.

I’m not sure if it is a bug or not… genuinely (I’m a newbie)

It seems like the type checker is working as intended

I don’t think my test target has any kind of explicit dependency on pytest

And the type checker is just surfacing that reality.

But I’m not trying to argue any particular point here, I’m just trying to figure out how this works!

FWIW adding the explicit dependency on

pytest

in requirements.txt fixed the type checker warning.

So

pants

scans the Python sources, if it finds an

import xyz

statement, it creates a child node on the dependency graph with the label

xyz

and then it looks in its

python_requirements

to see if there are modules that map to

xyz

? Is that correct?

So stepping back, you can have a

requirements-dev.txt

if you want (just wrap it in a python_requirements() target), but you don't need to - generally it's idiomatic to have a single requirements.txt with everything in it, since Pants will subset that for you. At test time it'll only take the dependencies needed for that test, at pex-building time it'll only bundle what is actually used into the pex, and so on.

And the way it does this is through dependency inference (plus any explicit deps)

dependency inference works exactly as you suggest: it maps

import xyz

to

whatever provides xyz

. That can either be an internal .py file in your repo, or an external requirement.

So if you put

pytest

in your

requirements.txt

, pants will notice that it now has something that provides the

pytest

module, and do the mapping.

To add to the confusion, there is the pytest lockfile... That represents "pytest as a tool". There is also "pytest as an importable library", which is what I described above.

I am slowly working towards unifying the "tool lockfile" and "user lockfile" concepts, so that you don't have to specify pytest in two places.

But for now, alas, you do, once as a tool, once as a library

Does that all make sense?

And when you say "adding the explicit dependency on pytest in requirements.txt fixed the type checker warning" do you mean just adding pytest to requirements.txt, or also adding an explicit dep on it from the python_tests target? The latter should not be necessary

Does that all make sense?

It really does make sense. I’m coming from recent context in Bazel so these concepts map pretty well.

when you say “adding the explicit dependency on pytest in requirements.txt fixed the type checker warning” do you mean just adding pytest to requirements.txt, or also adding an explicit dep on it from the python_tests target?

It was just adding

pytest

to the requirements.txt file, once that resolved the issue I got an inkling of how the dependency inferencing must be working.

I didn’t add anything to the

python_tests

target.

This was a super helpful explanation @happy-kitchen-89482 and @polite-garden-50641. Thank you both for taking the time to walk me through it.

Cool, glad this makes some sort of sense

Hey! I'm trying to upgrade to 2.15 and our custom plugin is failing because of a rule graph error. The tips for debugging this I should activate some rules in

register.py

, but I'm trying to find out whether doing it in there is the same as just adding it to

pants.toml

(to rule that out). The docs aren't very clear (or, well, at all, there are very few mentions of activation in the docs).

Hi I have a pants docker question. I would like to be able to decide at run time (when I run ./pants publish ::) which docker registry to push to. My use case is that on CI I want it to push to prod registry, and on dev machines to push to some staging registry. Is this something that is supported? It seems like the registry is statically defined in pants.toml and I don’t see any mention in the docs on how to switch default registry via env var at run time.

Hello! Have a question. I have different dependency versions and two lockfiles for it jvm-default and sdk-V2 and during the check I get bunch of errors like:

services/integration/src/main/scala/com/integration/repository/MxRepo.scala:14: error: not found: object mongo4cats
import mongo4cats.client.MongoClient

Dependency BUILD:

GROUP_ID = "io.github.kirill5k"
VERSION = "0.6.4"

jvm_artifact(
    name="core",
    group=GROUP_ID,
    artifact="mongo4cats-core_2.13",
    version=VERSION,
    resolve="sdk-V2",
    packages=[
        "mongo4cats.**",
        "mongo4cats.client.**",
        "mongo4cats.database.**",
        "mongo4cats.operations.**",
    ],
)

toml for this service:

[groups.integration]
addresses = ["services/integration::"]
resolve = "jvm:sdk-V2"
display_name = "integration"

pants.toml, where I read this config:

[experimental-bsp]
groups_config_files = ["build.d/bsp/services.toml"]

I tried to show dependencies manually in the package BUILD file and the result is:

NoCompatibleResolve: The selected targets did not have a resolve in common:

jvm-default:
here all the dependencies resolved for jvm-default and paths to integration microservice, that should be resolved by sdk-V2

sdk-V2:
  * 3rdparty/jvm/mongo4cats:core

Targets which will be merged onto the same classpath must share a resolve (from the [resolve]

Do you have any suggestions about it? Thanks in advance!

Does Pants provide OOTB support for declaring dependencies recursively for a directory? 🧵

Hi, am new to Pants and I encountered this error whenever I download files: Exception: Error downloading file: error sending request for url (https://github.com/boyter/scc/releases/download/v3.0.0/scc-3.0.0-x86_64-apple-darwin.zip): error trying to connect: invalid peer certificate contents: invalid peer certificate: UnknownIssuer Does anyone have any tips for this? Thanks

Hey there! I have progressing really nicely to get all my CI/CD merged with Pants but one project is bugging me hard. I have a Django Project and I have been following this https://blog.pantsbuild.org/optimizing-python-docker-deploys-using-pants/ to start creating the BUILD file that will generate the Docker image at the end. Currently I am having this error while running

package

#7 [deps 3/3] RUN PEX_TOOLS=1 /usr/local/bin/python3.10 /binary.pex venv --scope=deps --compile /bin/app
#7 sha256:3328d42cc75c5094a8bfd08b0070ed2a8d406a2bba037baaba700942750bf8d2
#7 4.316 /root/.pex/unzipped_pexes/58bfb582e660413de5eeb8bb20f70d49654571c8/.bootstrap/pex/dist_metadata.py:397: PEXWarning: Ignoring 1 `Requires` field in /root/.pex/unzipped_pexes/58bfb582e660413de5eeb8bb20f70d49654571c8/.deps/django_colorfield-0.8.0-py3-none-any.whl metadata:
#7 4.316 1.) Requires: django (>=2.2)
#7 4.316
#7 4.316 You may have issues using the 'django-colorfield' distribution as a result.
#7 4.316 More information on this workaround can be found here:
#7 4.316   <https://github.com/pantsbuild/pex/issues/1201#issuecomment-791715585>
#7 4.316
#7 4.316   pex_warnings.warn(
#7 4.338 /root/.pex/unzipped_pexes/58bfb582e660413de5eeb8bb20f70d49654571c8/.bootstrap/pex/dist_metadata.py:397: PEXWarning: Ignoring 1 `Requires` field in /root/.pex/unzipped_pexes/58bfb582e660413de5eeb8bb20f70d49654571c8/.deps/django_widget_tweaks-1.4.12-py3-none-any.whl metadata:
#7 4.338 1.) Requires: django (>=2.2)
#7 4.338
#7 4.338 You may have issues using the 'django-widget-tweaks' distribution as a result.
#7 4.338 More information on this workaround can be found here:
#7 4.338   <https://github.com/pantsbuild/pex/issues/1201#issuecomment-791715585>
#7 4.338
#7 4.338   pex_warnings.warn(
#7 6.736 Encountered collision building venv at /bin/app from /binary.pex:
#7 6.736 1. /bin/app/lib/python3.10/site-packages/tests/__init__.py was provided by:
#7 6.736        sha1:da39a3ee5e6b4b0d3255bfef95601890afd80709 -> /root/.pex/installed_wheels/5b3bc328e79c857f3d9a59b674e6770097cd73bb8696625b754c3b077c5e2fd3/django_rest_passwordreset-1.3.0-py3-none-any.whl/tests/__init__.py
#7 6.736        sha1:adc83b19e793491b1c6ea0fd8b46cd9f32e592fc -> /root/.pex/installed_wheels/591847b93161828c4e1b0197e3f531e82689d43624e2ff5142ea938514584112/python_quickbooks-0.9.2-py3-none-any.whl/tests/__init__.py
#7 ERROR: executor failed running [/bin/sh -c PEX_TOOLS=1 /usr/local/bin/python3.10 /binary.pex venv --scope=deps --compile /bin/app]: exit code: 1
------
 > [deps 3/3] RUN PEX_TOOLS=1 /usr/local/bin/python3.10 /binary.pex venv --scope=deps --compile /bin/app:
------
executor failed running [/bin/sh -c PEX_TOOLS=1 /usr/local/bin/python3.10 /binary.pex venv --scope=deps --compile /bin/app]: exit code: 1

and this is my BUILD file

python_sources(
    dependencies=[":static-files"],
)

resources(name="static-files", sources=["**/*.html", "**/*.css", "**/*.js", "**/*.png"])

poetry_requirements(
    name="poetry",
    module_mapping={
        "django-allauth": ["allauth"],
        "django-decorator-include": ["decorator_include"],
        "django-multifactor": ["multifactor"],
        "python-quickbooks": ["quickbooks", "intuitlib"],
        "djangorestframework-simplejwt": ["rest_framework_simplejwt"],
        "django-recaptcha": ["captcha"],
        "django-invitations": ["invitations"],
    },
)

pex_binary(
    name="binary-deps",
    entry_point="gunicorn",
    dependencies=[":poetry", "src/services/dashboard/dashboard/wsgi.py"],
    layout="packed",
    include_sources=False,
    include_tools=True,
)

pex_binary(
    name="binary-srcs",
    entry_point="gunicorn",
    dependencies=[":poetry", "src/services/dashboard/dashboard/wsgi.py"],
    layout="packed",
    include_requirements=False,
    include_tools=True,
)

pex_binary(
    name="binary-manage",
    entry_point="manage.py",
    dependencies=[":poetry", "src/services/dashboard/manage.py"],
    layout="packed",
    execution_mode="venv",
)

docker_image(
    name="img-deps",
    repository="dashboard",
    registries=["build"],
    image_tags=["deps"],
    skip_push=True,
    instructions=[
        "FROM python:3.10-slim",
        "COPY src.services.dashboard/binary-deps.pex /binary-deps.pex",
        "RUN PEX_TOOLS=1 /usr/local/bin/python3.10 /binary-deps.pex venv --scope=deps --compile /opt/app",
    ],
)

docker_image(
    name="img-srcs",
    repository="dashboard",
    registries=["build"],
    image_tags=["srcs"],
    skip_push=True,
    instructions=[
        "FROM python:3.10-slim",
        "COPY src.services.dashboard/binary-srcs.pex /binary-srcs.pex",
        "RUN PEX_TOOLS=1 /usr/local/bin/python3.10 /binary-srcs.pex venv --scope=srcs --compile /opt/app",
    ],
)

docker_image(
    name="img",
    dependencies=[":img-srcs", ":img-deps"],
    registries=["<http://docker.io|docker.io>"],
    repository="repo/dashboard",
    image_tags=["{build_args.RELEASE_VERSION}"],
    instructions=[
        "FROM python:3.10-slim",
        "RUN apt-get update && apt-get install iputils-ping -y",
        'ENTRYPOINT ["/opt/app/pex"]',
        "WORKDIR /opt/app",
        "EXPOSE 80",
        "COPY --from=build/dashboard:deps /opt/app /opt/app",
        "COPY --from=build/dashboard:srcs /opt/app /opt/app",
        "COPY src.services.dashboard/binary-manage.pex /opt/app/manage",
        'CMD ["--worker-class=gevent","--worker-connections=1000", "--workers=1", "-b", "0.0.0.0:80", "dashboard.wsgi:application"]',
    ],
)

I understand that 2 library is trying to write the same thing in the PEX compile, but I am totally lost on how to solve this issue, I have tried with a simple version and with many more configuration. Any idea on how to solve this?

Has anyone found or built a tool for automated editing of pants BUILD files? I have tried bazel's buildozer but it can't seem to work for me.