TIL that pex is mentioned in the Spark documentation as one way to deploy to a cluster: https://spark.apache.org/docs/latest/api/python/user_guide/python_packaging.html#using-pex

Somehow missed it when I needed to ship some python code on all the executors. I had to manually install conda env on all those machines 😭

Benjy, should we add a page to our Python integration docs?

Probably, although that is not supported on Databricks, only on OSS Spark, so we should probably figure out that side of thing as well first