https://pantsbuild.org/ logo
#plugins
Title
# plugins
p

powerful-eye-58407

05/04/2023, 1:24 PM
Hi, I'm looking for some tips on how to create a plugin that solves the following problem: I want to create a python wheel using the following target:
Copy code
python_distribution(
    name="common_package",
    dependencies=<DEPENDENCIES_LIST>,
    provides=python_artifact(
        name="common_package",
        version="1.0.0",
        author="Pantsbuild",
        python_requires=">=3.8.0",
    ),
)
This is a "library" wheel, being a collection of various packages. This means there is no single entry point. I still want to use pants dependency inference for 3rd party packages. I could have just listed all subdirectories of the sources root as dependencies and have them all included in the wheel (exactly as I want) - but the problem is that there's over 100 of them, and they keep changing.... So, I thought I can create a plugin to tweak the python distribution target so that the dependencies list is automatically generated using some sort of mechanism described in the doc, for example by just using
AllTargets
. The problem is that I'm not sure how PythonDistribution target is used, it doesn't seem as easy as tweaking SetupKwargs for python artifacts, as described here 🙂 I'd appreciate any tips on how to get started on the plugin (or maybe if there's a simpler way that I might have missed). Thanks!
I was thinking that maybe creating a rule that somehow modifies
InferPythonDistributionDependencies
request is a good way to start https://github.com/pantsbuild/pants/blob/main/src/python/pants/backend/python/target_types_rules.py#L446
so that I do not have to copy-paste the logic of of
infer_python_distribution_dependencies
c

curved-television-6568

05/04/2023, 2:02 PM
I have two approaches I would consider here, I’ll present both as the plugin version would just be a streamlined version of the first. 1. the brute force way: use some bespoke script to generate a
target(name="library_deps", dependencies=[<DEP LIST>])
snippet that you put in a BUILD file and then have a dep from the python_distribution to that:
dependencies=["libs:library_deps"]
2. as a plugin, setup a custom target type that picks up your targets as deps according to the logic in the custom script and use that custom target in the BULD file for the dist to depend upon
p

powerful-eye-58407

05/04/2023, 2:21 PM
Thanks! You are right, that's sounds simpler than hacking python_distribution
ok, so I think I have the first one more or less ready as I already have a script that gives me that list, but I will investigate the option nr 2 a bit more 🙂
👍 1
OK, so to clarify the 2nd approach, I'd have to create BUILD file like below:
Copy code
python_distribution(
    name="common_package",
    dependencies=[":library_deps"]
    provides=...
)

library_root(  # this is a custom LibraryRoot target
    root_dir="directory_name_goes_here",
    name="library_deps",
    # dependencies=... << this I want my plugin to automatically populate somehow
)
Then I need a rule to transform a target with root_dir field so that it becomes depended on all targets inside root_dir that make sense, right? I'm not sure what rule signature do I need here, as I'm not sure what pants does to get dependencies for a target. Based on this I'd guess it would be sth like:
Copy code
@rule 
async def populate_deps(target: LibraryRoot, request: DependenciesRequest) -> Targets:
    # extract library root dir from target
    # create the list of targets and return them
Does this make sense? I have not tested if that's going to be called by the engine yet 🙂
c

curved-television-6568

05/04/2023, 3:04 PM
yea, that’s pretty close. I’ll dig up some pointers/examples for dep inference from pants repo that might be helpful to look at for inspiration
p

powerful-eye-58407

05/04/2023, 3:05 PM
that would be great, thank you! Docs are really good but I feel like I'm trying to achieve something non-standard here :)
c

curved-television-6568

05/04/2023, 3:32 PM
I like to use the docker backend as example because I know it the best. https://github.com/pantsbuild/pants/blob/main/src/python/pants/backend/docker/util_rules/dependencies.py In your case you’d have a field set that has the
root_dir
field as required rather than some dependencies field. then implement the rule returning
InferredDependencies
as appropriate
p

powerful-eye-58407

05/04/2023, 3:46 PM
thank you, that's exactly what I need, at least my rule is triggered properly now, I can start implementing the logic
Copy code
class LibraryRootsField(StringSequenceField):
    alias = "library_roots"
    help = "Paths to directory containing source code to be packaged"


class PythonLibraryConfiguration(Target):
    alias = "python_library_configuration"
    core_fields = (*COMMON_TARGET_FIELDS, LibraryRootsField)


@dataclass(frozen=True)
class PythonLibraryDependenciesInferenceFieldSet(FieldSet):
    required_fields = (LibraryRootsField,)
    library_roots: LibraryRootsField

class InferPythonLibraryDependencies(InferDependenciesRequest):
    infer_from = PythonLibraryDependenciesInferenceFieldSet

@rule
def infer_python_library_dependencies(
    request: InferPythonLibraryDependencies,
) -> InferredDependencies:
    <http://logger.info|logger.info>("infer_python_library_dependencies called")
    # TODO logic goes here...
    return InferredDependencies(...)
for reference, I ended up with something as above
👍 1
thank you @curved-television-6568, it would take me much more time to figure this out by myself 🙂
c

curved-television-6568

05/04/2023, 3:50 PM
fyi, you’d likely want to lookup the owning targets for the source files you find, as that is what should go into the output result of that rule.
👍 1
oh, nvm the last comment, if you use specs, you get targets already..
h

happy-kitchen-89482

05/04/2023, 6:13 PM
Possibly even simpler - could you replace your fine-grained per-directory python_sources with one at the source root, with
sources=["**/*.py"]
(maybe excluding tests if you have tests in the same dirs as sources) and then have a single dep on that target?
p

powerful-eye-58407

05/04/2023, 7:11 PM
Possibly even simpler - could you replace your fine-grained per-directory python_sources with one at the source root, with
sources=["**/*.py"]
(maybe excluding tests if you have tests in the same dirs as sources) and then have a single dep on that target?
That would work without plugins, that's true. I already have the structure with one BUILD file per directory and some of them required extra changes to infer 3rd party dependencies, that's why I was wondering how to do it in a different way, without removing all existing BUILD files in subdirs.