https://pantsbuild.org/ logo
#general
Title
# general
a

astonishing-london-2419

09/15/2021, 8:52 AM
The second part is a bit more complicated. Each of the applications of the monorepo is run using Airflow, and for this each application dynamically generates a set of tasks that are then put together by the tooling. The process currently is as follows: 1. We call the
build-dags
command in our custom tool 2. For each of the libraries inside the
app
folder, the tool loads
library.dag
and executes a
generate_tasks
function, which returns a list of airflow tasks and their dependencies. 3. Once all tasks have been collected, the tool figures out the DAGs (you can have cross-app DAGs) and writes an output YAML file that is deployed from the CI in airflow. How to achieve that with
pants
? My idea is to break this into two steps: 1. On
./pants package
, not only the
pex
are generated but also YAML files with the tasks for each app/library. This would mean creating a new target
airflow_tasks
that would load the
library.dag
, run the
generate_tasks
function and dump the result in a per-app YAML. 2. Create a new goal
./pants build-dags
(or maybe it could be done with
package
if
pants
supports “ordered” builds) that has the
airflow_tasks
as inputs and then puts together all the YAMLs and builds the DAGs. I guess this is how I would do it, but looking at the plugins docs it all seems a bit daunting. Any example plugins I can look at to understand how to do this myself?
2 Views