silly-queen-7197
04/14/2023, 11:48 PMray.init(runtime_env=...)
I have an "entrypoint" item-rank/src/item_rank/publish.py
so I can do something like ./pants dependencies --transitive item-rank/src/item_rank/publish.py
. If I keep all of my requirements in a single requirements.txt
file I can parse the output and get something like
"db-dtypes~=1.0.4",
"gcsfs~=2022.10.0",
"google-cloud-bigquery-storage==2.16.0",
from
//:reqs#db-dtypes
//:reqs#gcsfs
//:reqs#google-cloud-bigquery
with a little bit bash golf.
How might I translate
archipelago/src/archipelago/foo.py
capstan/src/capstan/bar.py
into env vars like PYTHONPATH=$PYTHONPATH:archipelago/src:capstan/src
e.g. the relevant source roots (I suppose this is just "./pants roots"). I've done this manually and it seems to work (I'm running into permission issues so there are still errors that will take until Monday to resolve, but it seems like this is a viable route)
Edit - I never really asked a question, this comment is really more that I'm curious how other folks have approached integrating with ray. I feel like I'm traveling down the wrong path here yet I should be able to query pants about everything I need to construct an environment for my ray workers. A better solution would be if ray understood pex but that doesn't appear to be a supported feature yethappy-kitchen-89482
04/15/2023, 12:06 AMpants roots
is what you want for the PYTHONPATHpants dependencies --transitive path/to/file.py | \
xargs pants list --filter-target-type=python_requirement | \
xargs pants peek | \
jq .[].requirements
peek
goal gives you detailed info about each input targetsilly-queen-7197
04/15/2023, 12:11 AM./pants dependencies --transitive item-rank/src/item_rank/publish.py | \
xargs ./pants list --filter-target-type=python_requirement | \
xargs ./pants peek | jq '[.[] | .requirements[]] | reduce .[] as $item ([]; . + [$item])'
experimental_shell_command
dependencyhappy-kitchen-89482
04/15/2023, 12:20 AMsilly-queen-7197
04/15/2023, 12:23 AMhappy-kitchen-89482
04/15/2023, 12:23 AMsilly-queen-7197
04/15/2023, 12:26 AMhappy-kitchen-89482
04/15/2023, 3:23 AMgentle-gigabyte-52115
04/17/2023, 7:56 AMsilly-queen-7197
04/17/2023, 10:07 PMRuntimeEnvPlugin
. They use this internally for implementing pip
https://github.com/ray-project/ray/blob/master/python/ray/_private/runtime_env/pip.py#L387, conda
https://github.com/ray-project/ray/blob/master/python/ray/_private/runtime_env/conda.py#L257 etcPexPlugin
utilizing the same strategy might be possible.