Any pointers before I start building a pants plugin? In order to run Ray jobs, that can use code from around the monorepo, deployed to Anyscale, I would like to be able to create a

ray_job

target. This would do three things: • It would take an

script

python source file, and construct a pex binary containing the script and its source dependencies. • It would generate a wrapper script that submits the job to anyscale and include that in the root of the pex, as the entrypoint of the pex • It would generate a

requirements.txt

that can be submitted along with the job to anyscale, and include that in the root of the pex. This would allow me to

pants run path/to/ray_job

and have that execute on a cloud compute cluster. Conceptually, I guess it's close to the way the lambda_function target works, in that the lambda function target takes a handler, generates a wrapper, and constructs a zip, but every time I try to read the source of a pants plugin, I just bounce off. 1. Is this a reasonable thing to undertake 2. Can anyone help me grok the steps involved?

My default recommendation(s) when it comes to custom plugins are (not necessarily in this order): 1. Read the docs: https://www.pantsbuild.org/stable/docs/writing-plugins/overview 2. Read the tutorials: https://www.pantsbuild.org/stable/docs/tutorials/create-a-new-goal 3. Read some information from outside the pantsbuild website (there used to be a page for this, which had stuff other people's blogs had worked on: e.g. https://sureshjoshi.com/development/first-pants-plugin) 4. Refer to any of the similar backends/plugins in the pants repo - as that's the largest source of code 5. Really consider whether you need to write a plugin - or maybe `adhoc_tool/`macros is/are enough

Hey Suresh, I'm a big fan of your tutorial fwiw! I have been down this path before, and yeah, maybe I could hack it with an adhoc_tool and a shell script.

I'm a big fan of your tutorial fwiw

Thanks! I have some stuff coming down the pipe which should make plugins a little bit simpler to write - but I'm trying to also envision a world where they're not needed in the same way we have them today.

The thing I never figured out the last time I was thinking through something like this was "how do I synthesise a requirements.txt"

Good question - you CAN do stuff like this without a dedicated plugin, but I can see why you might want it. Ignoring the details of what this example does - you can see that there is a world where you use some scripts to synthesize what you want: https://github.com/sureshjoshi/pants-shell-command-example/blob/main/src/BUILD.pants But, by "generate a requirements.txt" Im guessing you mean more like - something from the pex is used, as in, you want the output of pex_binary to emit this? Again, likely do-able, but it would essentially be using shell scripting + adhoc tools

Context: Anyscale is a platform for running distributed jobs for ML purposes. To submit a ray job, you create a

working directory

which includes all your code, and upload it, with a requirements.txt for 3rd party deps. Let's say I have a python script at src/ray/my_job.py, and it depends on src/data/cleaning.py, which in turn depends on some numerical library. I would like to create a pex where the entry point is a dynamic python file, or a static python file that reads some env vars. That file would use the anyscale sdk to upload the current working directory, containing the sources, and submit src/ray/my_job.py, plus a requirements.txt to Anyscale. so the file tree is something like entrypoint.py requirements.txt src data cleaning.py ray my_job.py where requirements.txt is synthesised and entrypoint is, at the least, configured via my target.

Small note: I recommend using deploy goal for this. Take a look at the kubernetes backend, it's pretty small and uses the deploy goal

Shoulda read the docs