https://pantsbuild.org/ logo
#general
Title
# general
w

worried-salesclerk-37834

09/24/2021, 8:27 PM
Not sure of the best way to do this... I use airflow via Cloud Composer. In order to deploy the Directed Acyclic Graphs (DAGs), you zip up all of the python files containing those definitions. So there's no entrypoint or anything, just a zip of files. What I would like to do is also depend on our common library, so that some of the dags can use code from there. It doesn't look like I can just make an archive with "dependencies" on the dags' python library and have it transitively pull in my common library's code.
python_distribution
looks like it might be a way forward, but it looks like I'd have to make a separate distribution for the common library? It's relatively large, so I was hoping not to pull in the entire codebase. Is there a way I can make a zip of a current
python_library
that also pulls in all of its transitive dependency code?
h

happy-kitchen-89482

09/24/2021, 8:52 PM
Hmm
You build this with an
archive
target today?
How are you constructing those zips today?
w

witty-crayon-22786

09/24/2021, 8:54 PM
w

worried-salesclerk-37834

09/24/2021, 8:55 PM
We're just using terraform today to zip the directory since it's entirely self-contained now. We have to copy code we want to use into it.
We're not using dag-factory, though that does look interesting...
h

happy-kitchen-89482

09/24/2021, 9:18 PM
I see, so the goal here would be to not have to copy the code?
I don't use airflow so I want to be sure I understand the use-case
w

worried-salesclerk-37834

09/24/2021, 9:36 PM
Yep! Would be great to use dependency inference just like our cloud functions and pex binaries.
h

happy-kitchen-89482

09/24/2021, 9:55 PM
I see, so possibly a regular pex (which is just a zip file with the .pex suffix) would work?
As a proof-of-concept you could make a
pex_binary
with a fake entry point, that depends on a
python_library
containing the DAG code, and dep inference should do the right thing, I think
w

worried-salesclerk-37834

09/25/2021, 1:54 AM
Boom! That worked perfectly. Would have never thought of the fake entrypoint. Thanks!
h

happy-kitchen-89482

09/25/2021, 2:51 AM
Ah great!
In fact I think you can omit the entry point entirely, so you don't even need the fake one
👀 1
w

worried-salesclerk-37834

09/25/2021, 4:29 AM
Pants' pex_binary docs say you can set the entrypoint argument to "", but when I do I get an "InvalidFieldException" saying that I have to set it to some value.
Code was modified 7 months ago: https://github.com/pantsbuild/pants/blame/main/src/python/pants/backend/python/target_types.py#L207 (unless the logic came from somewhere else). Docs: https://www.pantsbuild.org/docs/reference-pex_binary#codeentry_pointcode Not sure which one is expected, but I can submit a change to either to make them align.
h

happy-kitchen-89482

09/25/2021, 10:30 PM
I think you can just omit that field entirely?
w

worried-salesclerk-37834

09/27/2021, 1:55 PM
Oh! I was reading the 2.6 docs when I was using 2.8. Yep, that totally works and jives with the docs that I should have been reading. Thanks for your patience!
3 Views