#17858 AWS MWAA plugin New discussion created by cburroughs AWS MWAA or "Amazon Managed Workflows for Apache Airflow" https://docs.aws.amazon.com/mwaa/latest/userguide/what-is-mwaa.html is an AWS service for managed service for Apache Airflowhttps://airflow.apache.org/. Airflow is usually described as a pure python orchestrator or "really fancy crontab". You define in python "go do this job at this time" with various configuration knobs for retries and dependency graphs. The jobs themselves have plugable "operators" (docker, k8s, bash) which notable include more pure python. Any python code needs to have the right 3rd party dependencies available in the managed environment. The deploy procedure is described in: https://docs.aws.amazon.com/mwaa/latest/userguide/working-dags-dependencies.html https://docs.aws.amazon.com/mwaa/latest/userguide/best-practices-dependencies.html#best-practices-dependencies-python-wheels So to deploy into the managed environment one: • upload your python files to s3 • upload a requirements.txt file for pip • (optional) upload a zipfile of wheels A plants plugin would do something like: • Gather the right python files to upload somewhere in

dist

• Create a zipfile of the wheels they depend on. (Properly resolved with a constraints file) • Generate a dummy requirements.txt file that just points at the wheels. Slack 🧵 https://pantsbuild.slack.com/archives/C046T6T9U/p1667574027454759 pantsbuild/pants