Is there a way I can easily run multiple `experimental_run_shell_command`s in parallel? For example, both of these

experimental_run_shell_commands

were identified as changed:

 ./pants --changed-since=HEAD --changed-dependees=transitive --filter-tag-regex='^cdk$' list
aws/projects/project_1:cdk
aws/projects/project_2:cdk

But if I try to

run

them, I get an error:

 ./pants --changed-since=HEAD --changed-dependees=transitive --filter-tag-regex='^cdk$' run
12:32:43.46 [ERROR] 1 Exception encountered:

  TooManyTargetsException: The `run` goal only works with one valid target, but was given multiple valid targets:

  * aws/projects/project_1:cdk
  * aws/projects/project_2:cdk

Please select one of these targets to run.

I can hack something with

xargs

like this:

export PANTS_CONCURRENT=True && ./pants --changed-since=HEAD --changed-dependees=transitive --filter-tag-regex='^cdk$' list | xargs -L1 -P 2 ./pants run

but I was wondering if there's a "better"/more

pants

-y way to do this?

Interactive Processes can only be run serially. So multiple pants commands (without using the daemon) is the only way

Interactive Processes

My command doesn't need to take any input - does that mean it's non-interactive and I should be using a different method other than

experimental_run_shell_command

?

That's the internal name for anything being

run

. What do your processes do? AFAIK we don't have a mechanism to have the user request multiple processes run in parallel on-demand and without caching 🤔

Ah, gotcha. That's helpful context. My script calls out to a

cdk

(AWS CDK infrastructure-as-code tool) to run a command (

diff

or

deploy

infrastructure). That command needs the Python context from Pants since the CDK code itself is defined in Python.

Hmmm you could maybe plug into the

package

command. There's a

deploy

one as well. That'd involve a plugin today, but could also be extended

Let me look at the docs for those. IIRC I looked at

deploy

and I thought it was helm-specific.

It is today, but anything is pluggable 😌

Ah, so via

experimental-deploy

is how I'd plug in? https://www.pantsbuild.org/docs/reference-experimental-deploy (not really any docs, but maybe I can reference the Helm impl for an example)

There's a bit of a paradigm shift going on in regard to shell processes. I think we're ooching towards really opening up the floodgates with those. I could easily see, from recent changes, a way to specify an

experimental_shell_publish_command

which: can run in parallel, as part of

publish

CC @ancient-vegetable-10556 /@happy-kitchen-89482 /@witty-crayon-22786 while we're splashing at

shell

stuff

Sure. For some more perspective, I'm coming from `yarn`/`lerna` monorepos and they can

run

any arbitrary

script

in the

package.json

with a concurrency flag. It's super nice!

yarn lerna run --since '' --concurrency 10 cdk -- deploy '**'

(Oh my bad, I mean

publish

not

package

. brain fog got me there)

Yes, it would probably look a lot like https://github.com/pantsbuild/pants/commit/6d603dd1e487e495f28160f7d76166455cd96a36

^ Yup that was my mental reference

For what it’s worth, and I haven’t spent too much time reading whether this needs to run outside the sandbox, but you can do this:

experimental_shell_command(name="a", command="first_command")

experimental_shell_command(name="b", command="second_command")

experimental_run_shell_command(name="c", dependencies=[":a", ":b"], command="/bin/true")

and then

./pants run path/to:c

They'd also be cached, which isn't ideal 😕

They’re only cached so long as their input dependencies don’t change

I think this might be the behavior I want. If: • the Python source code (that defines the infrastructure) doesn't change; or, • the underlying 3rdparty dependencies don't change; or, • the shared "library" methods don't change then the CDK shell command(s) should not be run at all

then try a format like the above; noting that you’ll probably need some hackery to handle reverts

Hmm, gotcha. Yeah the caching kinda scares me. With infra-as-code, I'd rather run "too often" and let the provisioning engine determine it's a no-op than not triggering when we should.

Certainly.

Alright, cool. I think I'll go that route for now. @ancient-vegetable-10556 and @bitter-ability-32190, thank you for taking the time to help me out! I hope that, in exchange, this use-case is useful for consideration (it sounds like maybe you're thinking about this anyway).

I have considered making it possible to mark `experimental_shell_command`s as non-cacheable, but that is for a later date!

FWIW, I looked at

bazel

before

pants

and

pants

was so much more understandable for me!

I migrated our monorepo and haven't looked back lol

@famous-river-94971 glad it's working out for you! Would it be okay to quote you, with or without attribution, on Twitter?

Hey Carina - sure, feel free! I don't want to ruffle any feathers with Bazel lovers, so just use the quote, no attribution necessary.

Sure!

There's definitely situations bazel is a better choice for build+test. And even in those cases pants can still help out with fmt+lint+check. While we were migrating, that's the boat we were in.