Is there a Pants plugin for Databricks Integration...
# generalb
bored-energy-25252
08/17/2022, 9:58 AMIs there a Pants plugin for Databricks Integration? Like:
• uploading the whl/jar to databricks libraries
• manage the databricks clusters
Or could pants community start a project like that?
h
happy-kitchen-89482
08/17/2022, 2:53 PMNot yet, but we’d welcome a motivated community member who knows about databricks and wants to look into this, and we’d be happy to help. We’ve had some users ask about this before, and I even reached out to a friend at Databricks to ask if they could add support for PEX files, as that would be a lot simpler than their current Python deployment story. (OSS Spark works with PEX, but commercial Databricks does not for some reason)
happy-kitchen-89482
08/17/2022, 2:54 PMI don’t think any of the core maintainers know enough about databricks to have a strong opinion about what this should do, what is useful and so on. But if you could write up a short design doc to start a conversation, that could be a great start
👍 1
b
bored-energy-25252
10/13/2022, 2:57 AMApache Spark supports PEX starting from PySpark >= 3.1.1 or PySpark >= 3.2.0
bored-energy-25252
10/13/2022, 2:57 AMh
happy-kitchen-89482
10/13/2022, 6:53 AMYep! Unfortunately Databricks does not though
b
bored-energy-25252
10/13/2022, 6:54 AMI’m trying to test if PEX works in Databricks. It will help us save a lot of pip install time.
h
happy-kitchen-89482
10/13/2022, 2:53 PMThat would be amazing if so, last I looked it was not a thing, but maybe that has changed!
happy-kitchen-89482
10/13/2022, 2:54 PMI have reached out to folks I know at Databricks to ask them about supporting PEX, they asked around internally but it didn’t go anywhere
happy-kitchen-89482
10/13/2022, 2:54 PM[This was a few months ago]
b
bored-energy-25252
11/29/2022, 4:31 PMJust got it work on open source pyspark: https://github.com/da-tubi/pants-pyspark-pex
Not far away from making it work on Databricks
3 Views