it would be feasible for a custom goal to walk the graph formed by invoking

build-dags

a bunch of times

to do that, you would write

@rules

that invoke

build-dags

, and then recurse on the discovered dependencies

in terms of inspiration: one question is whether airflow dags allow cycles: if they don’t, then you should be fine just using

await Get(Targets, Addresses(stuff_discovered_by_running_the_build_dags_tool))

as to including this into

./pants package

: that might also be feasible…? assuming you’ve built the above code to invoke that tool, you can have another

@rule

consume it, and install it as a generator of

files

targets: https://www.pantsbuild.org/docs/plugins-codegen

that would leave you in a situation where you’d have a custom target type (maybe

airflow_library_dag

next to your

pex_binary

which would generate a

files

target

to add a dependency on the custom target type to your `pex_binary`s, you would probably want to declare a macro (maybe named

airflow_binary

) which would always declare both underlying targets at once, and with a dependency between the binary and the dag

Is it feasible to import a library declared as a dependency of a this new type of target?

you mean have the

airflow_library_dag

type depend on python code…?

on the python code of the library itself

I am still quite lost, unfortunately, so I broke the problem in two parts: 1. Generate a YAML file with the list of tasks generated by each of my libraries 2. Put together all the YAML files, run logic and generate a DAG file for airflow However, I am struggling already in 1. What I want is that my

GenerateSourcesRequest

goes from

PythonSources

to

FilesGeneratorTarget

(if I understood correctly). However, in the

rule

I need to import those python sources to run the correct function, get a dictionary and write it to disk. Does this even make sense? I could create a pex on the fly that calls the desired function, but I am not sure if this makes sense.

I need to import those python sources to run the correct function, get a dictionary and write it to disk. Does this even make sense?

that’s correct: to get the python sources of your targets, you’d do something like:

sources = Get(
        SourceFiles,
        SourceFilesRequest(
            sources_fields=[t.get(Sources) for t in targets],
            for_sources_types=(PythonSources,),
        ),
    )

then, to run a

Process

, you’d do something like https://www.pantsbuild.org/docs/rules-api-process#process

and it’s a reasonably quick function