https://pantsbuild.org/ logo
Join Slack
Powered by
Heya! I am using pantsbuild to build a docker imag...
# general
f
Heya! I am using pantsbuild to build a docker image that we are going to use on Databricks. I kind of want to include every python_source in my repo as a dependency, is that even possible?
✅ 1
It would seem that 
InjectedDependencies
is no longer a thing in Pants?
r
Is your repo a single Python package or multiple? I've never tried adding 
python_source
targets to an image directly, but compiling a 
pex_binary
target and then adding that to the image works nicely.
f
Yeah sorry, that’s what I am doing. The 
python_source
targets and other dependencies are depended on by a pex which is copied into the image.
@square-psychiatrist-19087 helped me out with a plugin that makes it possible to depend on python_source targets recursively. Very useful stuff!~
s
yeah, maybe it makes sense to contribute the plugin to pants repo, it looks like it is sometimes useful
🙌 1
f
I am admittedly trying to do something weird here, but to me it was extremely useful!
g
I literally just did this in pants as of a few days ago. I can share my full source with you.
@famous-xylophone-36532 You can technically include every Python file by having a single target that just includes all python files.
Copy code
python_sources(
    name="all-python",
    sources=["**/*.py"],
)
f
Wow that's fun, we solved this in exaxtly the same way by using venv from pex. Cool! I want to avoid the catch all python sources thing since we are using this repo for so many other things.
g
@famous-xylophone-36532 yeah, I would advise against it as well 🙂
@famous-xylophone-36532 Did you do anything different from me that I could learn from you?
f
Hmm, no not really, other than the fact that Azure Databricks wants to create its own venv rather than use your provided one for some reason, so I had to alias 
my-pex.pex venv
to the 
virtualenv
command.
I might be doing a plugin for Databricks Asset Bundles, I just need to understand them a bit more first 😄
Will be open sourced as soon as I can convince the right people @ my clients.
g
I had to alias 
my-pex.pex venv
to the 
virtualenv
command.
@famous-xylophone-36532 interesting... Were you setting 
PYSPARK_PYTHON
?
f
Yep!
g
That's very odd. We didn't have such issues using Databricks.
2 Views
Open in Slack
PreviousNext
Powered by