This is probably a dumb question, but here goes: I have a

pex_binary

target, that, when run, reads some files from the local file system. I can't count on those files being present on the local file system, so would like to always run a script to populate those files before the

pex_binary

is run with

pants run :the_pex_binary

. I have a feeling this may be somewhat antithetical to Pants' design, or out of scope, but wanted to check.

This feels like something maybe the new workspace environment can help with eventually. But, if I’m reading this, you want a side-effect to run before your pex_binary. Will the pex_binary contain the newly created files?

you want a side-effect to run before your pex_binary.

that's right. The binary won't contain the newly created files. At least, it doesn't need to. But if that's a way to do this, I'm game to try it.

Well, I guess, if it's not necessarily in the pex - why not have a shim inside your code?

yeah I'm trying to avoid code changes since the context is a migration from a previous build system. But I could do that. what if I go the route of putting the files in the PEX? It's sort of unclear to me how those files are accessible through general file system APIs.

I mean, there might be a way to use the shell script or another thing, but it feels like cramming determinism into a non-deterministic system (or vice versa, I'm not sure). Like: I want reproducible builds, but I conditionally want to change some part of the filesystem, which, after I make my build - might then change again ... before my next build? It's a bit abstract, so I'm trying to make it more concrete. The shim inside a pex file, assumes that you want these files created at runtime (which is what it sounded like), but it's also running through pants. Not suggesting it can't be done, but I haven't done it using Pants. But, using either a macro or a plugin, the world's your oyster

yeah, fair. So we have a previous build in Earthly. It's got some tasks like "run service X". The service, on startup, loads some BentoML models from the filesystem into memory. So in a dependent Earthly task, we made sure those BentoML models were present in the expected place by reading a YAML file and downloading them using the BentoML CLI (which fetches them from GCS). Then the service would run, see the files, and load them properly. Previously this was all happening in a Docker container, which I could probably do again, but am trying to avoid this time around.

Ah interesting, any reason to move off Earthly? I've never used it. Re question: just need to think for a sec

many reasons: • everything running in Docker is incredibly slow (you're always rebuilding images) • it's so general as to be useless - you need to manually define all your tasks like linting/formatting/tests/run/etc. It's not really aware of much besides Docker. • you still need something like Poetry to manage your deps, and once we hit about 8 pyproject.tomls in the repo it just became too much (it's a monorepo, but a small one)

Oh wow, okay, I guess i really didn't know much about earthly - I thought it was more than just docker. Cool, thanks for letting me know! So, me personally, just speaking for myself - this feels like the kinda thing I would end up doing in Docker, or in pre-run bindings in a

scie

https://github.com/a-scie/jump I'm pretty sure this would all be do-able in a plugin. Natively though, I'm a little less sure. You've tried shell, which didn't work out (https://www.pantsbuild.org/2.21/docs/shell#testing-your-packaging-pipeline), and adhoc tool would be my next place to look. I'm curious if

archive

might be an option

ah ok haven't looked at that (archive) yet, thanks for the pointer. It could be that I'm just doing shell wrong too. Docker does seem like the easy route, but I think the difficulty I would have is running the tests. They are integration tests that basically start the service and throw requests at it. So it does all this file loading then too.

Yep, fair, I personally avoid docker when I can - as there's a burden in using it, that I often don't need.

hmm what does it do exactly?

As I understand it, it would allow using a remote URL as an HTTP source, pulling that into the pipeline - but not re-downloading if it exists. So, not quite reading a yaml file, but 🤷

I see. This archive/assets docs page is helping a lot actually. I've been thinking about this wrong, not surprisingly.

A

file

target is for loose files that are copied into the chroot where Pants runs your code. You can then load these files through direct mechanisms like Python's

open()

or Java's

FileInputStream

this sounds like what I want. I can basically put the BentoML model files into the repo dir in the expected structure, and have the code load from there. and as far as I understand it,

shell_command

essentially gives you

files

, is that right? So I could then use that to run the BentoML CLI to create the

files

instead of having to put them in manually.

and as far as I understand it,

shell_command

essentially gives you

files

, is that right? So I could then use that to run the BentoML CLI to create the

files

instead of having to put them in manually.

I don't know about BentoML - but if there is some way to have loose files in your repo, using files/resources is how you would collect them into the system

hah ok thanks, really appreciate your help. I'm going to plug away at it some more, will keep you posted.

So, this ended up being a pretty cool little side track, using some targets I'd never needed to touch before

oh man this is super helpful, thanks a ton! I should be able to get something going.

If you have a CLI tool, then the shell command, or adhoc tool are probably safe bets - and then you have to carefully pass dependencies from

output_xyz

to the next layer. That was also the first time I'd needed to use

experimental_wrap_as_resource

since otherwise, a

file

is generated, and it's kinda loose in the pex_binary target

yeah I think I'll need to use a

file

in the end, the code that loads the model is doing regular FS access

That's interesting - I guess since the pex won't include it, it just doesn't bother?

the warning goes away with this, but it still doesn't run. So confused...

pex_binary(
    name="bin",
    dependencies=[
        ":lib",
        ":archive"
    ],
    entry_point="main.py"
)

archive(
  name="archive",
  format="zip",
  files=[":run-downloader"],
)

python_sources(
    name="lib",
    dependencies=[
        ":local-file",
        ":downloaded-image",
    ],
    sources=["**/*.py"],
)

# experimental_wrap_as_resources(
#     name="wrapped_downloader_output",
#     inputs=[":run-downloader"],
# )

shell_command(
    name="run-downloader",
    command="./downloader.sh",
    execution_dependencies=[":scripts", ":manifest"],
    output_files=["dilbert-rng.gif"], # This must match your expected file(s) (or use output_directory)
    tools=["curl", "head"],
)

and you're still running in my repo?

yep

I guess a pex can't contain an archive?

yeah exactly

🤷 But as I mentioned, might be worth checking out adhoc tool and seeing if maybe that has some of the hooks you're interested in. It's a lot more powerful for making pipelines

fair enough, I'll keep looking. Thanks agaiin!