I need to build a docker image containing an artifact that I'll download from S3. What's the appropriate way to treat that artifact as a pants target? Do I need to build a plugin here, or is a shell script sufficient?

Could/Would you download it inside the container image itself?

maybe but that's a little fiddly for reasons. It would be neater to treat it as an input to the docker image.

Not the only solution, I situationally tend to do stuff that can also be run alone, if needed, rather than always having everything through Pants. However, if you want to do it via Pants, there is a target in the docs that should work. Sec....

Fantastic, I'll take a look - thanks as always!

Let me know how that works, as I can't recall how stuff like special auth would work (I guess as part of the URL, or env vars). I've used that target before, just trying to find it.

I'm unconvinced that this is the way, given that we'll need to do aws auth, but it's nice to have the option. Ta! I'll let you know what we end up doing.

https://github.com/sureshjoshi/pants-shell-command-example/blob/main/README.md I believe I sent this to you earlier this year, but essentially someone was asking about file resources in shell_commands, and one of the examples I gave used an

http_source

to grab an image. https://github.com/sureshjoshi/pants-shell-command-example/blob/75b8f6e9b9222becda8049b6d99d065288c6ecb6/src/BUILD.pants#L46 Nothing new there, but just an example

Hello, following on from @average-breakfast-91545's question, I'd like to throw yet more complexity into the mix. Turns out it's not just an artifact from S3 but also some metadata we have to query from Dynamo. We already have python code to do this, so I thought perhaps I'd be able to run a python script to manage dynamo and s3 and output the file we need. At first I tried

shell_command

but that wasn't running for the

package

goal so I switched to

package_shell_command

instead. I thought

python

would work like any other binary but

package_shell_command

is timing out running

python --version

Before I go down a rabbit hole trying to do a simpler reproduction of the problem: Does anyone know if this is expected to work?

Mixing shell commands with packaging and all of that should be possible, but it comes down to “which set of targets to use”, as there are many involving shell/adhoc_tools and they all sort of do something different. I think a lot of this is more historical/legacy mixing rather than “this is the optimal way to set this up for these different potential workflows” Can you break down the list of steps you’re trying to do, in what order?

Sure. 1. Run a python script to do some aws things - query dynamo, pull a file from S3, store that ✨ somewhere ✨ 2. Build a pex that includes all dependencies for a python app 3. Build a pex that includes all source code for that python app 4. Build a docker image that includes deps + source code + the file output form step #1 Steps 2, 3 and 4 all work fine. We previously would do the aws things at runtime but we're deploying into an environment where that will no longer be possible, so trying to shift the work to build time instead. Hopefully that all makes sense

For context, we're packaging an ML model, which is a binary blob, but which has runtime deps on both code and libs which need to be versioned along with the artifact.

Okay, yeah, this all sounds do-able, so long as the final asset ends up in the pants-pipeline at some point. Let me see if I can find examples of where I've done something like this

Yeah, I tried a couple of different variations:

package_shell_command(
    name="model_download",
    command="exec -a $0 python model_download.py --out output --model-name foo",
    tools=["python", "bash"],
    output_directories=["output/"],
    execution_dependencies=["model_download.py"],
    log_output=True,
)

# BUILD
#############
shell_sources(name="scripts")

package_shell_command(
    name="model_download_shell",
    command="./test.sh",
    tools=["bash", "aws", "which", "touch", "python"],
    output_directories=["output/"],
    log_output=True,
)

# test.sh
#############
#!/bin/bash

echo 'This is a test'
which python
python --version

You can see in the shell variation I was just trying to get anything python flavoured to work

huh... that's interesting

I've just tried running again with --keep-sandboxes and running the shell script outputted, same thing. 30s timeout.

Interesting. Let me try to whip something up here quickly, to see if I can run into this

I'll give this a whirl and report back, thanks for your help!

👍 Might still not be what you're looking for, as I'm literally running a script which then runs python - so, kinda dumb, but it's just a workflow that I know "should" work, and then streamlining that workflow could be something to look into once you have a workflow that is adequate for your purposes There are a few other mechanisms which could work here too, I just didn't have a pre-made repo including them 😄

Interestingly this example still didn't work for me. First I had to add

bash

into the tool list and then I got the same timeout error:

Process 'the `shell_command` at `src/service/edge/models/download:run-download`' failed with exit code -15.
stdout:

stderr:


Exceeded timeout of 30.0 seconds when executing local process: Running the `shell_command` at `src/service/edge/models/download:run-download`

It didn't even print the python version 🤔

Oh, that's something. What env var does pyenv make pythons available at? This might be a case where the logic of the

tools

field doesnt match some of the other locations

I'm not exactly sure what you mean. It leaves this in my environment:

$ env | grep PYENV
PYENV_ROOT=/Users/ryanloader/.pyenv
PYENV_SHELL=zsh
PYENV_VERSION=inference

Unfortunately it doesn't link a version explicitly in there because there's another level of indirection:

$ ls ~/.pyenv/versions/
3.11.14        inference       ingest       serve

I just mean some pants facilities are a bit better at accounting for pyenv