I have a naive question. I'm trying to run some code on a ray cluster. In order to get this to work I need to tell ray a little bit about the packages I use. I need to essentially pass a dict like {"pip": [pip requirements], "env_vars": {...}, "working_dir": "."} to

ray.init(runtime_env=...)

I have an "entrypoint"

item-rank/src/item_rank/publish.py

so I can do something like

./pants dependencies --transitive item-rank/src/item_rank/publish.py

. If I keep all of my requirements in a single

requirements.txt

file I can parse the output and get something like

"db-dtypes~=1.0.4",
"gcsfs~=2022.10.0",
"google-cloud-bigquery-storage==2.16.0",

from

//:reqs#db-dtypes
//:reqs#gcsfs
//:reqs#google-cloud-bigquery

with a little bit bash golf. How might I translate

archipelago/src/archipelago/foo.py
capstan/src/capstan/bar.py

into env vars like

PYTHONPATH=$PYTHONPATH:archipelago/src:capstan/src

e.g. the relevant source roots (I suppose this is just "./pants roots"). I've done this manually and it seems to work (I'm running into permission issues so there are still errors that will take until Monday to resolve, but it seems like this is a viable route) Edit - I never really asked a question, this comment is really more that I'm curious how other folks have approached integrating with ray. I feel like I'm traveling down the wrong path here yet I should be able to query pants about everything I need to construct an environment for my ray workers. A better solution would be if ray understood pex but that doesn't appear to be a supported feature yet

As you said,

pants roots

is what you want for the PYTHONPATH

Thanks

Neat

There's https://github.com/ray-project/ray/issues/15518 but I don't think I have the skill / time to contribute to that project

The fact that ray, and databricks, and aws lambda, and gcp cloud functions, all want you to provide raw requirement and entry point metadata to deploy python, tells you how immature the python deployment story still is

To databrick's credit, pyspark supports pex. Unfortunately my team has a huge aversion to spark / pyspark

Pyspark does but databricks doesn't, for some reason

Yusuf thanks for sharing, we also use ray and databricks. We don’t have a solution for ray yet, but we do publish pex files to use with databricks. We use some custom init scripts to inject some code in the python site packages to init the pex environment.

I did a little digging on this. Ray allows you to configure runtime environments via

RuntimeEnvPlugin

. They use this internally for implementing

pip

https://github.com/ray-project/ray/blob/master/python/ray/_private/runtime_env/pip.py#L387,

conda

https://github.com/ray-project/ray/blob/master/python/ray/_private/runtime_env/conda.py#L257 etc