Hey! I'm trying to handle resource files that are not saved on disk, namely, 1. Some resourced are generated with some python code 2. Some resources are stored on s3-like storage on Google Cloud. I understand I can add specific targets for these file type, however, it is not clear to me what kind of rule I should add so that these new target types would be either generated or downloaded when injected as dependencies to other targets, such as

python_sources

target. For example I want to have the following target:

python_source(name="code", dependencies=[":s3_resouces", ":generated_resourced"])

and then be able to

pants run "path/to/file:code"

I've tried to follow docs on code generators, which seem to be a very similar use-case, however, it focuses on using the generated code outside of pants, where I want to be able to utilise the engine to track those dependencies automatically and cache them too ideally. Any advice on how to handle it ?

I think you can achieve that without a plugin, file target can take http_source, so you can fetch a file from s3. Then use experimental_wrap_as_python_sources and feed it into pex_binary as a dependency

Thanks @square-psychiatrist-19087. 1. I don't think file works here as s3 requires authentication and in our case a special tool to download (

gsutil

for google cloud). Just passing a google url gives the following error

IntrinsicError: Error downloading file: builder error for url (gs://<our_url>): URL scheme is not allowed

2. Thanks, I'll check it out!

have you tried adhoc_tool?

looking at it right now, yeah, seems like exactly what I need, thank you

you will probably need system_binary to define the gsutil binary and then pass it to adhoc_tool as runnable

and for python code-generator I would just pass

python_sources

target as runnable ?

you should pass the code generator itself as a runnable

I see, thanks 🙏

For gsutil, you should be able to make a pex instead. Easier + more portable.