Hi Pants experts I am trying to build airflow dags into pex

Question

Hi Pants experts I am trying to build airflow dags into pex files see this <https github com mingshi wang airflow pants example|repo> The problem is I am unable to add dependencies to the contraint txt when I run build support generate constraints sh Can anyone please show me what is wrong Thanks

hundreds-father-404 · Answer

Hello What do you mean you are unable to add dependencies Are you getting an error

powerful-florist-1807 · Answer

I didn t get any error I tried to run build support generate constraints sh but I the generated contraints txt misses the dependencies in the requirements txt

hundreds-father-404 · Answer

Ah ha it looks like our example repo is a bit out of date with the `generate constraints sh` The version you re using only will use requirements actually used by your project whereas this version uses everything from requirements txt as well Update the script to this <https www pantsbuild org docs python third party dependencies tip set up a virtual environment optional>

powerful-florist-1807 · Answer

Great I will give it a try

enough-analyst-54434 · Answer

But also your project does use 1 of those requirements and we still miss it It will need a module mapping

enough-analyst-54434 · Answer

apache airflow gt airflow

hundreds-father-404 · Answer

I ll cherry pick to 2 5 0rc2 which means that < powerful florist 1807> you can remove all that `module mapping` stuff from your `BUILD` file when you upgrade because Pants s default will cover it h t to < curved television 6568> for adding this feature

powerful-florist-1807 · Answer

Great

powerful-florist-1807 · Answer

Finally I got the airflow dag runnable from PEX Thanks < hundreds father 404> < enough analyst 54434>

powerful-florist-1807 · Answer

Hi < hundreds father 404> < enough analyst 54434> I refactored a class MyBashOperator from examples src python dags dag py into a separate file under the same folder python dags task py But when I built and ran the pex pants package examples src python pipeline main py it complains MyBashOperator cannot be imported because module dags is not found I have verified the root is correctly set to examples src python Do you know what could be wrong

powerful-florist-1807 · Answer

This can be repro ed in the <https github com mingshi wang airflow pants example|repo>

hundreds-father-404 · Answer

Can you run ` pants roots` and report back what you get

hundreds-father-404 · Answer

See <https www pantsbuild org docs troubleshooting import errors and missing dependencies> for the general guide on import errors

powerful-florist-1807 · Answer

Sure it says examples src python

powerful-florist-1807 · Answer

I guess I need to build a module of examples src python dags

hundreds-father-404 · Answer

What do you mean build a module

hundreds-father-404 · Answer

Also what does ` pants dependencies examples src python dags dag py` say

powerful-florist-1807 · Answer

Sure it says Exception Unmatched glob from file arguments examples src python python dags dag py

hundreds-father-404 · Answer

I gave you a bad command one too many `python `

powerful-florist-1807 · Answer

OK

powerful-florist-1807 · Answer

I reran it and see

powerful-florist-1807 · Answer

mingshiwang Mingshi Wang airflow pants example pants dependencies examples src python dags dag py apache airflow examples src python dags task py

hundreds-father-404 · Answer

K so that is behaving how we want Great I m looking at <https github com mingshi wang airflow pants example blob main examples src python pipeline main py> and I don t understand how it works it doesn t ever import your `dags` folder

powerful-florist-1807 · Answer

Oh main py doesn t import dags It is just a wrapper of airflow cli

powerful-florist-1807 · Answer

` pants print stacktrace ldebug run examples src python pipeline main py list dags sd examples src python dags`

powerful-florist-1807 · Answer

This is how I used it to start a dag via the CLI

powerful-florist-1807 · Answer

BTW I am working with Leo to improve the python dev experience at Coinbase and this is my POC

hundreds-father-404 · Answer

Ah I see you aren t including the files you want to run in the PEX itself Okay I ll take a look tomorrow and clone your repo Busy week with Pycon

powerful-florist-1807 · Answer

< hundreds father 404> I solved the issue which is due to a bug in Airflow 1 10 10 Thanks for all your help

hundreds-father-404 · Answer

Oh yay Sorry I hadn t gotten to it but great to hear