https://pantsbuild.org/ logo
Join Slack
Powered by
Hi Pants experts, I am trying to build airflow dag...
# general
p
Hi Pants experts, I am trying to build airflow dags into pex files (see this repo). The problem is I am unable to add dependencies to the contraint.txt when I run "build-support/generate_constraints.sh". Can anyone please show me what is wrong? Thanks!
h
Hello! What do you mean you are unable to add dependencies? Are you getting an error?
p
I didn't get any error. I tried to run "build-support/generate_constraints.sh", but I the generated contraints.txt misses the dependencies in the requirements.txt
h
Ah ha, it looks like our example repo is a bit out of date with the 
generate_constraints.sh
. The version you're using only will use requirements actually used by your project, whereas this version uses everything from requirements.txt as well Update the script to this: https://www.pantsbuild.org/docs/python-third-party-dependencies#tip-set-up-a-virtual-environment-optional
p
Great! I will give it a try
e
But also, your project does use 1 of those requirements and we still miss it. It will need a module mapping.
apache-airflow -> airflow
👍 2
h
I'll cherry-pick https://github.com/pantsbuild/pants/pull/12068 to 2.5.0rc2, which means that @powerful-florist-1807 you can remove all that 
module_mapping
stuff from your 
BUILD
file when you upgrade because Pants's default will cover it (h/t to @curved-television-6568 for adding this feature!)
p
Great!
Finally I got the airflow dag runnable from PEX. Thanks @hundreds-father-404 @enough-analyst-54434!
🙌 1
Hi @hundreds-father-404 /@enough-analyst-54434 I refactored a class "MyBashOperator" from "/examples/src/python/dags/dag.py" into a separate file under the same folder "examples/src/python/dags/task.py". But when I built and ran the pex (./pants package examples/src/python/pipeline/main.py), it complains "MyBashOperator" cannot be imported because module "dags" is not found. I have verified the root is correctly set to "examples/src/python". Do you know what could be wrong?
This can be repro'ed in the repo
h
Can you run 
./pants roots
and report back what you get?
p
Sure it says "examples/src/python"
👍 1
I guess I need to build a module of 'examples/src/python/dags'
h
What do you mean build a module?
Also what does 
./pants dependencies examples/src/python/dags/dag.py
say?
p
Sure - it says " Exception: Unmatched glob from file arguments: "examples/src/python/python/dags/dag.py""
h
I gave you a bad command, one too many 
python/
p
OK
mingshiwang@Mingshi-Wang airflow-pants-example % ./pants dependencies examples/src/python/dags/dag.py //:apache-airflow examples/src/python/dags/task.py
I reran it and see
h
K, so that is behaving how we want. Great. I'm looking at https://github.com/mingshi-wang/airflow-pants-example/blob/main/examples/src/python/pipeline/main.py and I don't understand how it works - it doesn't ever import your 
dags
folder?
p
Oh - main.py doesn't import dags. It is just a wrapper of airflow cli
./pants --print-stacktrace -ldebug run examples/src/python/pipeline/main.py  -- list_dags -sd examples/src/python/dags
This is how I used it to start a dag via the CLI
BTW - I am working with Leo to improve the python dev experience at Coinbase, and this is my POC
h
Ah I see, you aren't including the files you want to run in the PEX itself Okay, I'll take a look tomorrow and clone your repo. (Busy week with Pycon!)
🙏 1
p
@hundreds-father-404 I solved the issue which is due to a bug in Airflow 1.10.10. Thanks for all your help!
❤️ 1
h
Oh yay!! Sorry I hadn't gotten to it, but great to hear :)
2 Views
Open in Slack
PreviousNext
Powered by