hey guys, I’m setting up airflow and was wondering if there are any best practices or recommendations in getting airflow set up in our monorepo. specifically: 1. we’re using poetry to track all our common dependencies, but since airflow pins its dependencies and doesn’t support poetry, we’re not able to simply do a

poetry add apache-airflow

since it leads to an unsatisfiable list of dependency constraints 2. because of this, airflow requires you provide a

requirements.txt

with a list of 3rd-party libraries used in your DAGs. does pants have a way to generate a requirements file from a

python_sources

target? am i thinking about this the wrong way? 3. haven’t started working on adding support for 1st-party code that’s imported into DAGs but I have a feeling this is not going to be very straightforward. any recommendations here?

@high-energy-55500 I'm looking answers to your questions. I'm also encountering the same situation. Were you able to generate the requirements.txt file automatically using pants? Also, I'm curious about how you packaged your first-party code—was it as a pex file or a tar file?

@white-twilight-61019 the gist of it is that airflow is a PITA to work with. we ended up needing to use a bunch of hacks to get airflow working. recently we've switched to multiple resolves which addresses some of the pain points with incompatible packages required by airflow, but adds a whole other layer of complexity. if I could go back, i would have preferred we went with Prefect or some other software. Airflow is one of the most frustrating pieces of software i've ever dealt with