Hi, I'm looking for some tips on how to create a ...
# pluginsp
powerful-eye-58407
05/04/2023, 1:24 PMHi, I'm looking for some tips on how to create a plugin that solves the following problem:
I want to create a python wheel using the following target:
Copy code
python_distribution(
name="common_package",
dependencies=<DEPENDENCIES_LIST>,
provides=python_artifact(
name="common_package",
version="1.0.0",
author="Pantsbuild",
python_requires=">=3.8.0",
),
)AllTargetspowerful-eye-58407
05/04/2023, 1:27 PMI was thinking that maybe creating a rule that somehow modifies
InferPythonDistributionDependenciespowerful-eye-58407
05/04/2023, 1:28 PMso that I do not have to copy-paste the logic of of
infer_python_distribution_dependenciesc
curved-television-6568
05/04/2023, 2:02 PMI have two approaches I would consider here, I’ll present both as the plugin version would just be a streamlined version of the first.
1. the brute force way: use some bespoke script to generate a
target(name="library_deps", dependencies=[<DEP LIST>])dependencies=["libs:library_deps"]p
powerful-eye-58407
05/04/2023, 2:21 PMThanks! You are right, that's sounds simpler than hacking python_distribution
powerful-eye-58407
05/04/2023, 2:22 PMok, so I think I have the first one more or less ready as I already have a script that gives me that list, but I will investigate the option nr 2 a bit more 🙂
👍 1
powerful-eye-58407
05/04/2023, 3:00 PMOK, so to clarify the 2nd approach, I'd have to create BUILD file like below:
Copy code
python_distribution(
name="common_package",
dependencies=[":library_deps"]
provides=...
)
library_root( # this is a custom LibraryRoot target
root_dir="directory_name_goes_here",
name="library_deps",
# dependencies=... << this I want my plugin to automatically populate somehow
) Copy code
@rule
async def populate_deps(target: LibraryRoot, request: DependenciesRequest) -> Targets:
# extract library root dir from target
# create the list of targets and return thempowerful-eye-58407
05/04/2023, 3:02 PMDoes this make sense? I have not tested if that's going to be called by the engine yet 🙂
c
curved-television-6568
05/04/2023, 3:04 PMyea, that’s pretty close. I’ll dig up some pointers/examples for dep inference from pants repo that might be helpful to look at for inspiration
p
powerful-eye-58407
05/04/2023, 3:05 PMthat would be great, thank you! Docs are really good but I feel like I'm trying to achieve something non-standard here :)
c
curved-television-6568
05/04/2023, 3:32 PMI like to use the docker backend as example because I know it the best.
https://github.com/pantsbuild/pants/blob/main/src/python/pants/backend/docker/util_rules/dependencies.py
In your case you’d have a field set that has the
root_dirInferredDependenciesp
powerful-eye-58407
05/04/2023, 3:46 PMthank you, that's exactly what I need, at least my rule is triggered properly now, I can start implementing the logic
powerful-eye-58407
05/04/2023, 3:48 PM Copy code
class LibraryRootsField(StringSequenceField):
alias = "library_roots"
help = "Paths to directory containing source code to be packaged"
class PythonLibraryConfiguration(Target):
alias = "python_library_configuration"
core_fields = (*COMMON_TARGET_FIELDS, LibraryRootsField)
@dataclass(frozen=True)
class PythonLibraryDependenciesInferenceFieldSet(FieldSet):
required_fields = (LibraryRootsField,)
library_roots: LibraryRootsField
class InferPythonLibraryDependencies(InferDependenciesRequest):
infer_from = PythonLibraryDependenciesInferenceFieldSet
@rule
def infer_python_library_dependencies(
request: InferPythonLibraryDependencies,
) -> InferredDependencies:
<http://logger.info|logger.info>("infer_python_library_dependencies called")
# TODO logic goes here...
return InferredDependencies(...)powerful-eye-58407
05/04/2023, 3:48 PMfor reference, I ended up with something as above
👍 1
powerful-eye-58407
05/04/2023, 3:50 PMthank you @curved-television-6568, it would take me much more time to figure this out by myself 🙂
c
curved-television-6568
05/04/2023, 3:50 PMfyi, you’d likely want to lookup the owning targets for the source files you find, as that is what should go into the output result of that rule.
👍 1
curved-television-6568
05/04/2023, 3:56 PMthere’s a lot going on here, but this might be useful/reusable:
https://github.com/pantsbuild/pants/blob/2417cfeaf5de9b8c4bcf2a847bffc03551c2f773/[…]thon/pants/backend/python/dependency_inference/module_mapper.py
curved-television-6568
05/04/2023, 3:59 PMoh, nvm the last comment, if you use specs, you get targets already..
curved-television-6568
05/04/2023, 3:59 PMh
happy-kitchen-89482
05/04/2023, 6:13 PMPossibly even simpler - could you replace your fine-grained per-directory python_sources with one at the source root, with
sources=["**/*.py"]p
powerful-eye-58407
05/04/2023, 7:11 PMPossibly even simpler - could you replace your fine-grained per-directory python_sources with one at the source root, withThat would work without plugins, that's true. I already have the structure with one BUILD file per directory and some of them required extra changes to infer 3rd party dependencies, that's why I was wondering how to do it in a different way, without removing all existing BUILD files in subdirs.(maybe excluding tests if you have tests in the same dirs as sources) and then have a single dep on that target?sources=["**/*.py"]