https://pantsbuild.org/ logo
#general
Title
# general
h

happy-kitchen-89482

04/13/2022, 8:50 AM
TIL that pex is mentioned in the Spark documentation as one way to deploy to a cluster: https://spark.apache.org/docs/latest/api/python/user_guide/python_packaging.html#using-pex
❤️ 2
😁 2
r

refined-addition-53644

04/13/2022, 10:58 AM
Somehow missed it when I needed to ship some python code on all the executors. I had to manually install conda env on all those machines 😭
h

hundreds-father-404

04/13/2022, 1:40 PM
Benjy, should we add a page to our Python integration docs?
h

happy-kitchen-89482

04/13/2022, 2:27 PM
Probably, although that is not supported on Databricks, only on OSS Spark, so we should probably figure out that side of thing as well first
3 Views