So I'm on Pants 2.16.0rc1, testing out the

shell_command

. I have a python project, with python tests, and inside the BUILD file for said tests I've added

python_tests(...)

shell_command(
    command="./just_run_touch_foo.sh",
    tools=["bash", "touch"],
    execution_dependencies=[":scripts"],
    output_directories=["."],
    workdir=".",
    description="this just runs `touch foo"
)

shell_sources(
    name="scripts"
)

What should I expect to happen here? Hoping to see a file

foo

sitting in the directory after the tests run. (Or, at least, inside the sandbox while tests run. So far I have seen neither.) At the current time, my attempt at running the shell command above results in

19:40:00.77 [WARN] No applicable files or targets matched. The `test` goal works with these target types:

  * experimental_test_shell_command
  * python_test
  * shunit2_test

However, you only specified target arguments with these target types:

  * shell_command

I'm pretty sure I don't want to run

experimental_test_shell_command

, as this actually runs a test inside bash script. What I want eventually is to download some canonical dataset that will be used in my unit tests...

shell_command

is designed as a target for generating files for other targets to use: you will need a dependency from the Python test target to the shell command one, so that pants knows the files are needed.

About that

file

target... Is there any way I can send an authentication section in a header, like what you'd do with curl?

Not that I know of. Being able to pick up auth details from a environment variable (or something along those lines?) seems like a sensible feature request for http_source, if you have a moment to file an issue: https://github.com/pantsbuild/pants/issues/new?assignees=&labels=enhancement&template=feature_request.md&title=

you may want to implement a custom

url_handler

in that case.. https://github.com/pantsbuild/pants/tree/dcdd1bcc5c7b2f4992658fce28e469c3bc294bd9/src/python/pants/backend/url_handlers

@broad-processor-92400 added an issue let me know what you think.

Thanks!

No problem!

until your issue to support additional headers is resolved, a shell command that uses

curl

is a decent option: you’d only want to ensure that you included some version information and/or `shasum`ing of the downloaded file in your script.

Okay. Do you have an example

shell_command

that I can refer to? As noted above, I'm seeing an error "No applicable files or targets matched." I think it's because I'm doing something wrong with the BUILD file for this directory. I'm on Pants 2.16.0rc1

That sounds like you still might be using

pants run ...

? You'll need to use

pants export-codegen ...

to validate/debug it, and set dependencies in appropriate targets to use it for tests, e.g.

shell_command(
  name="data",
  command="curl ... && shasum ...",
  tools=["curl", "shasum"],
  output_file=["some_file.txt"]
)

python_tests(
  ...,
  dependencies=[":data"]
)

This will mean that running those tests knows that it needs to run the data command and materialize the files within the testing sandbox

So, what I have now is an actual shell script with a shebang at the top. What you're saying is to not do that, but put the actual shell command under the

command=

option?

ah, a shell script should be fine too. The key is having the dependencies set up: a

shell_command

target describes how to generate some files for other targets to use.

Okay, coming back to this from Friday, and I'm no longer getting an error.

It will be cached after the first run

Here is the BUILD file:

python_tests(
    name="tests",
    dependencies=[":data"]
)

files(
    name="data",
    sources=["test-data/*.h5"]
)

shell_command(
    command="./just_run_touch_file.sh",
    tools=["bash", "touch"],
    execution_dependencies=[":scripts"],
    output_directories=["."],
    workdir=".",
    description="run a shell command",
    dependencies=[":scripts"],
)

shell_sources(
    name="scripts"
)

and here is the script:

#!/usr/bin/bash

echo "hello"

touch foo

I don't see either "hello" nor the file

foo

Ah, `shell_command`s output is normally hidden (if they succeed). You can add

shell_command(..., log_output=True)

to help with debugging. You might also need to change the script slightly (e.g. add a comment

# 1

that you can just increment to bust the cache for now)

Also, I really appreciate your time... is there an example project that I could learn from? Somehow the docs are not clicking for me.

The file

foo

will, theoretically, be placed into the same directory as the

BUILD

file, in the test sandbox. You can peek into those sandboxes with https://www.pantsbuild.org/docs/reference-global#keep_sandboxes. For instance, if the tests are failing

pants --keep-sandboxes=on_failure test ::

and the cd/exploring the directory that is logged.

Tried that, but no luck. I'll continue debugging later on. Thanks for your help @cold-soccer-63228 @witty-crayon-22786