Hey all, my journey in the field of multiple resolves is continuing, and I'm investigating some solutions that might match all the use cases. I'm reporting one in particular and I wonder if you would have any recommendations for this. Basically, we have a monorepo that needs to support PyTorch in both CPU and GPU versions. The idea was to create two resolves: •

python-default

for the CPU version •

gpu

for the GPU We created two files

requirements.txt

and

requirements-gpu.txt

files that are equal, except for the version used for torch. In the case of

python-default

, it has the suffix

+cpu

. The resolves are indicated in `pants-utils/3dparty/python/BUILD`:

python_requirements(
    source="requirements.txt",
    resolve="python-default",
)

python_requirements(
    name="gpu",
    source="requirements-gpu.txt",
    resolve="gpu",
)

Basically, I'm having all the projects target the two resolves by defining the following in `/BUILD`:

__defaults__(
    {
        (python_source, python_sources): dict(resolve=parametrize("python-default", "gpu")),
    }
)

Therefore, I can specify the resolve I want to use for specific targets, like for

package

of wheels. It seems to work, but I face some problems when a developer specifies something like the following in their

BUILD

files:

python_sources(
    dependencies=["//pants-utils/3rdparty/python#pyarrow"],
)

In this case, PyArrow was manually inserted, since it could not automatically be inferred otherwise. Therefore, I have the questions: 1. Does it make sense what we are doing in the first place? 2. Is there a way to make

python_sources

to dynamically select the third-party library in a specific resolve? 3. I have the impression that using

__defaults__

is applied globally, but then overridden by who then use

___defaults___

for

python_sources

in subfolders. Is there a way just to extend the defaults?

1. Yes 2. I tend to avoid explicit dependencies as far as possible. However, the example you show should not cause issues regarding the resolve/defaults. What kind of issues are you experiencing with it? (also, if pyarrow is a dependency only indirectly required by some other 3rd party library, a better place is to add the dependency on that library directly as an override on the

python_requirements

target) 3. The

__defaults__

apply for the subtree where it is defined, so when placed in the root BUILD file, it applies globally, but if you provide it further down, it is limited in scope. You can override defaults at deeper levels as many times as you wish. You can add to defaults by using

__defaults__({ ... }, extend=True)

It will merge default values for new fields, but doesn't attempt to merge default values themselves. I.e. you can add default

tags

for instance, but you can't add another tag to a previously defined list of default tags.

Thank you @curved-television-6568! The example I reported is regarding a project

A

calling a library

B

which is also developed in the monorepo, and declaring the dependency form

pyarrow

with

python_sources

. I assume it's done since it's a library not imported, then Pants couldn't be able to infer it otherwise. Being the

pyarrow

declared in that way (it's also part of the resolves), I get the following error when building a PEX for

A

with the resolve `gpu`:

NoCompatibleResolveException: The target projects/a:test-gpu uses the `resolve` `gpu`, but some of its dependencies are not compatible with that resolve:

  * pants-utils/3rdparty/python#pyarrow (python-default)

Ah, ok. Still, the dependency on the 3rd party lib ought to be inferred if there is any source that imports it.. but circling back to the issue with the explicit dependency, as the

python_sources

target is parametrized for two different resolves, that explicit dep will be used for both of them which is problematic as it points to only a version of the lib for python-default not gpu. I don't know how to solve this from the top of my head, would need to run some experimentation..

Apparently this one case of `pyarrow`can be solved easily. However, I cannot exclude that there might be cases in which it is needed to explicitly define a third-party dependency. I believed there is no such a thing like parametrize for dependencies, considering the resolve.

pants 2.19.0 introduces the concept of parametrizing multiple fields together, so you could do:

python_sources(
   name="src",
   **parametrize("default", resolve="python-default", dependencies=["some/3rdparty/for-default"]),
   **parametrize("gpu", resolve="gpu", dependencies=["some/other/gpu-dependency"]),
)

nice .. Can that override the name of the target also ?

# target to add the self-signed-cert into all base images

docker_image(
  name="python_image_self_signed",
  **parametrize("py310", name="python310-self-signed", extra_build_args=["BASE_IMAGE=python:3-10-slim"],
  **parametrize("py311", name="python310-self-signed", extra_build_args=["BASE_IMAGE=python:3-11-slim"],
)

Does it work if you leave off name from the non-parametrized value?