Hey all I am having some problems packaging a library that depends on a shared library that uses a parametrize resolve like so I have a commons library like so

python_sources(resolve=parametrize("python-default", "new-deps"))
python_distribution(
  name="commons",
  dependencies=["src/python/commons/libs/commons"],
  provides=python_artifact(
    name="commons",
    version="1.0",
  ),
)

And I also have a library using the library:

libA

above like so

python_sources()
python_distribution(
  name="libA",
  dependencies=["src/python/commons/libs/commons"],
  provides=python_artifact(
    name="libA",
    version="1.0",
  ),
)

when I run package on libA

pants package src/python/qe/libs//libA:libA

. I get the following error:

ValueError: The explicit dependency `src/python/commons/libs/commons:commons` of the target at `src/python/qe/libs/libA:libA` does not provide enough address parameters to identify which parametrization of the dependency target should be used.
….

what I would like to do is use the version of

commons

that uses the

python-default

resolve, any idea how I can do that?? thanks All

I think it should be something like

src/python/commons/libs/commons:commons@your-resolve

try running

pants list ::

it will show you the addresses

gotcha let me try that again I have a feeling I tried something like that before.

If I understood your problem correctly, right now there is no good way to do parametrize on multiple fields. You can only create a full matrix that will generate 4 targets in your case Instead you can manually create 2 targets, for example, commons-v1 and commons-v2, and use the correct version in dependencies

Actually it seems like I paramtrize python_soruces but not

python_distributions

for example in order for other libraries to consume the

new-deps

version of commons, I would need to something like this?

python_sources(resolve=parametrize("python-default", "new-deps"))
python_distribution(
  name="commons",
  dependencies=["src/python/commons/libs/commons@resolve=python-default"],
  provides=python_artifact(
    name="commons",
    version="1.0",
  ),
)
python_distribution(
  name="commons-new-deps",
  dependencies=["src/python/commons/libs/commons@resolve=new-deps"],
  provides=python_artifact(
    name="commons",
    version="1.0",
  ),
)

perhaps I am missing somethig

yes, you can even do

python_sources(resolve=parametrize("python-default", "new-deps"))

kwargs = dict(
  provides=python_artifact(
    name="commons",
    version="1.0",
  ),
)

python_distribution(
  name="commons",
  dependencies=["src/python/commons/libs/commons@resolve=python-default"],
  **kwargs,
)
python_distribution(
  name="commons-new-deps",
  dependencies=["src/python/commons/libs/commons@resolve=new-deps"],
  **kwargs,
)

ah that's an interesting idea, this is an internal library shared amongst other applications and other libraries, so I don't necessarily need a wheel but do I need the consumers of the library to be pip installable, not sure how modeling a shared library as a pex binary would work in that case, but perhaps I am missunderstanding the suggestion?

yeah, if you need the code to be pip installable, then you need python_distribution unfortunately unless you want them all to use the monorepo and pants 😉

😆 I wish, that's actually a bit of a challenge I am facing, pants is great at managing/modeling dep between modules, but at the end of the day, the packaged libraries need to all be pip installable, which creates certain limitations.

Do you need all three commons, libA and libB pip installable?

nop i just need libA, and libB to be installable, commons is just a share code between them

nop i just need libA, and libB to be installable

yeah, if you need both to be installable in a single venv then you probably need 3 python_distribution-s

the interesting thing is that libA and libB won't co-exist, that is, they wont be installed in the same env

if they won't co-exist then you can just create 2 - one for libA and one for libB

right, and commons? does it need to be a

python_distribu..

?

I think it will work if you don't explicitly set dependencies on common. If it doesn't work you can put the python_distribution-s in the root BUILD, it should work this way

btw, regarding grouping parametrizations will be supported in pants 2.19.0: https://github.com/pantsbuild/pants/pull/20065 😉

exactly, just ran into that @curved-television-6568

yea, build files are plain python, only you can’t use import

nice!! love that!

if you need code sharing, there’s macros that will be shared across all build files

code sharing amongst build files, gotcha!

no worries, I gotcha, hey another quick q can you access data from the build files in the BUILD files, for example all the resolves, so I don't have to harcode them eg

resolves  = ['python-default', 'new-dep']

vs

resolves = get_resolves() # magcally returns all the resolves?

where would

get_resolves

be defined?

gotcha, ok so the question is, is there a default macro provided by pants that returns all the resolves

kind of like a stdlib at the same level of

parametrize

yea, there’s a few high level methods like that.. like

build_file_dir()

gotcha!

I would probably hard code the resolve in one place (in a macro) and use that in all BUILD files, that way you at least limit the change to a single place should that list need to be adjusted.

yeap that makes sense to me

hah, I don’t think we have a list of all available BUILD file methods.. they’re spread out and presented in the context of what they do

Generally if libA and libB must be distributions, and they depend on some common code, then you probably do want to publish that common code as a third dist and have libA and libB consume it as a requirement. Otherwise the common code will be packaged into libA and libB, and it's generally a bad idea to have the same code appear in two different dists.

thanks for the input @happy-kitchen-89482, you know I have been wondering how to acomplish packing common code in libraries that are guaranteed not to be installed on the same env, currently the

python_distribution

macro? (primitive? don't know how you all refer to those functions) enforces single ownership so

commons

(libA, libB, etc), thus I have to model

commons

as a distribution so that it can be used by other libs, eg libA, libB, etc. One idea I explored was to not use

python_distribtuion

and just use a custom

setup.py

but honestly haven't gone that far down the road, I would like to avoid recalculating dependencies that pants already know how to create, I guess I want something like this.

# The below showcases a non-existent "overrides" argument to specify how to package a given dependency.

python_distribution(
  name="libA",
  dependencies=["src/python/commons/libs/commons"],
  overrides= {"source": "src/python/commons/libs/commons"} # which tells pants to include the source of the dep as opposed to model it as installable dependency
  provides=python_artifact(
    name="libA",
    version="1.0",
  ),
)

Yes 90% of the time this is not what you want to do, as you point out, but there are ocassions in which you might need to do something like the above.

(

python_distribution

is a “target type”) 😉

Yes, that ticket encompasses a more general way of overriding what Pants generates. It'll take some work though.

a nice there is a ticket for that? would you mind pointing me to it?