We have three projects A, B and C. Project C is a common util project which is used by projects A and B. We are using

pex_binaries

and

python_distribution

targets in all projects using a macro. Before adding

python_distribution

,

pants repl

used to work fine. After adding

python_distribution

, we are seeing a resolve error

pex.environment.ResolveError: Failed to resolve requirements from PEX environment @ /path/to/.cache/pants/named_caches/pex_root/unzipped_pexes/d5bb451b67b4592376d4f6e8dde708facaf22sf7.
Needed cp39-cp39-manylinux_2_31_x86_64 compatible dependencies for:
 1: project-c==0.0.1
    Required by:
      project-a 0.0.1
    But this pex had no ProjectName(raw='project-c', normalized='project-c') distributions.
 2: numpy>=1.24.0
    Required by:
      project-a 0.0.1
    But this pex had no ProjectName(raw='numpy', normalized='numpy') distributions.

A couple of questions: 1. When only using

pex_binaries

, the repl command works fine but when adding

python_distribution

it breaks. Why is this happening? 2. numpy is a direct dependency added in the

dependencies

field in

python_distribution

and

project-c

should be an inferred dependency from the code. Why is it not resolving these dependencies? I verified they were added successfully to the wheel file. Thanks Edit: we are using pants 2.16.0rc4

Why do you need a pex and a distribution? Those are in at least conceptual conflict: a pex contains everything it neeeds to run, including all transitive deps. A distribution contains just its own code, and its metadata lists the other distributions it requires.

Benjy, is the Pants thinking that a

python_distribution

is only needed when it's an external artifact (e.g. publishing wheels), rather than for consumption within the repo by `pex_binary`ies etc.?

Something vaguely like that 🙂 I'd like to see the specific example to be sure

Hi @happy-kitchen-89482, Andrew and I work together. We need the

python_distribution

defined so we can build wheels for pushing to our artifact store. We use

pex

for things like that whole "run our code on a remote host" thing I outlined here.

It is, and it does.

That's good to know. We are on Pants 2.16.0rc6 and seeing this behavior. Here's a trimmed down version of our tree structure for this repo ATM:

.
├── locks
│   ├── real_stuff.lock.jsonc (numpy 1.25)
│   ├── pants-plugins.lock.jsonc
│   ├── resolve_example_1.lock.jsonc (numpy 1.23)
│   └── resolve_example_2.lock.jsonc (numpy 1.24)
├── pants-plugins
│   ├── BUILD
│   ├── plugin_1
│   │   ├── rules.py
│   │   ├── register.py
│   ├── plugin_2
│   │   ├── register.py
│   │   └── rules.py
│   ├── macros.py
├── pants.toml
├── projects
│   ├── examples
│   │   ├── example_project_1
│   │   │   ├── BUILD (resolve=resolve_example_1)
│   │   │   ├── example_project_1
│   │   │   │   └── f1.py (imports example_util_library.sample)
│   │   │   ├── requirements.txt (numpy < 1.24)
│   │   ├── example_project_2
│   │   │   ├── BUILD (resolve=resolve_example_2)
│   │   │   ├── example_project_2
│   │   │   │   └── f2.py (imports example_util_library.sample)
│   │   │   ├── requirements.txt (numpy >= 1.24)
│   │   └── example_util_library
│   │       ├── BUILD (resolves=[resolve_example_1, resolve_example_2])
│   │       ├── example_util_library
│   │       │   └── sample.py (imports numpy)
│   │       ├── requirements.txt (numpy < 1.25)
│   └── real_stuff
│       ├── groceries
│       │   └── shopping
│       │       ├── BUILD (resolve=real_stuff)
│       │       ├── requirements.txt
│       │       ├── shopping
│       │       │   ├── buy.py (references grocery_utils.make_a_list)
│       │   └── grocery_utils (resolve=real_stuff)
│       │       ├── BUILD
│       │       ├── requirements.txt (numpy)
│       │       ├── grocery_utils
│       │       │   ├── make_a_list.py (imports numpy)
└── pyproject.toml

Things seem to be working fine for the two "example" resolves, but maybe that's just because all three BUILD files specify something about

numpy

? For the

real_stuff

resolve,

shopping

references

grocery_utils

, but doesn't know about numpy. At runtime though, when invoking

make_a_list

, things fail because the numpy dependency isn't inferred. For now, we've worked around this by adding an explicit entry in our

python_sources

dependencies list to point to the

python_requirements

in the same

BUILD

file. This allows us to keep moving forward, but isn't ideal as every source file in a

python_sources

now depends on every line in a

requirements.txt

which will lead to more cache-misses even when that's not actually necessary

What are your source roots?

Something like this:

.
pants-plugins
projects/examples/example_project_1
projects/examples/example_project_2
projects/examples/example_util_library
projects/real_stuff/groceries/shopping
projects/real_stuff/groceries/grocery_utils