if I want to implment a publish rule that does it in a 'big bang' for all matched targets and not individually, what do I need to do? Can I do this? Struggling to make head or tails of the goal implementation 😅 \

Can you add a bit more detail, what is "it" in this case? What's a matched target in this case? Like, something that runs on all of your

my_publish

targets at once?

I have a set of individual targets that can be published to the PaaS. But they're interdependent, so when I publish

//a/b/product/foo

that might depend on

//a/b/modules/bar

they need to be deployed together with one

Process

call But if I also say i want to publish

//a/b/product/foo

and

//a/b/product/wiz

that both depend on the same stuff... they also need to be deployed together in one call (else the remote idempotentce stuff fucks out and throws a fault code when you try do it in parallel) And I have just realized i'm confusing Publish with Deploy

Yeahhhh... this feels more like an

experimental_deploy

on first read to me.

It’s a matter of joining the various `Process`es together into one.. how to do this would be dependent on the deployment command (and the backend providing the integration to it) I’d guess..

how do you mean joining Processes?

Not sure if it's what Andreas means, but I would assume that in your deployment, you'd Get (MultiGet?) on each of the pre-requisites (e.g.

package

these 5 targets), pass them all in as a digest to a single

Process

, and that magic

Process

is your "deployment" piece (whatever the business logic or tool that is)

ah yeah. that parts easy. Problem is if I do

./pants deploy //foo/bar/baz //foo/bar/wiz

and they both have overlapping dependencies

Is there any relationship between baz and wiz?

not directly, but they might share a common deployable dependency

So, is there a requirement that if you deploy baz, you change ALL of the dependants, or is it case by case, as you've written in that example?

it's more of a quirk in the deployment tool. Because

baz

depends on

common

you need to 'deploy'

(baz, common)

in one call (regardless or not if common has changed) And if

wiz

also depends on it, you need to deploy

(wiz, common)

But there's some parallelism controls on the 'remote' end and it can cause errors if you do

(baz, common), (wiz, common)

asynchronously but none if you do

(baz, wiz, common)

Ahhh, okay okay. Interesting... I thought deploy would take in multiple field sets, like some of the other goals

looks like it takes a singular. And it returns a

DeployProcess

that specifies it's

Process

and dependent targets that will also need deploying.

Hmm, okay, I haven't looked at that in a while - I see the problem now

hey, it's got

experimental-

for a reason

So, if you run deploy on the two targets, does it fork off into two separate Processes from the start?

from my understanding,

./pants e-deploy //foo //bar

will get a

DeployProcess

for each target Then publish any dependencies specified by that process. and then individually call the process for each of those targets. so yes, two separate processes.

👍 I thought Deploy worked like Check or something, where the targets are handed to you on a silver platter. Honestly, I thought there was a way to optionally have that be the case for all goals

Does feel like this is a recurring pattern. How do you decide to do things per-process, per-target, or per-file (dubiously the same as per-target I guess)

The ones I know of just kinda make sense to be bundled I guess. Lint, Fmt, Check packaging and stuff does feel like it could be semantically distinct things. Like, they're almost terminal processes.

it looks like publish supports both

Hmm, weird. I'd have to dig in to the code again - make sure I'm understanding this correctly. It's just been a while since I've looked at those guts

Haven’t read all the comments here, but from what I’ve gleaned so far, this could be akin to collecting the requirements for a

python_distribution

target. When asked to deploy

baz

, find out it depends on

common

and include that in the same “unit of work”. When asked to deploy

baz

and

wiz

things gets a little bit more interesting, as the backend preparing the deployment won’t know about more than one at a time (if the

deploy

goal is implemented in the same manner as

publish

) so there’ll need to be some sort of funnel in the deploy goal that can take the units of deployment and merge them into one, for those units that makes sense (likely on a per deployment-tool basis perhaps) — as @wide-midnight-78598 correctly alluded from my previous comment, this might be merging input digests for all the Processes into one so you can have a single Process with all the resources for it from all relevant targets to deploy. That’s what it looks like to me at least 🙂