https://pantsbuild.org/ logo
Join Slack
Powered by
Are `UnionRules` only for @goal_rule? I'm investig...
# plugins
g
Are 
UnionRules
only for @goal_rule? I'm investigating using them to route a "build" request in my OCI plugin such that 
ImageBundleRequest
is either translated into a 
Build...
or 
Pull...
variant. However; that doesn't seem to work correctly, complaining that my union subtype isn't created by a rule or in-place. Which is true - it's never mentioned explicitly.
✅ 1
f
Unions are for more than just goals. Just declare the union’s base class with the 
@union
decorator.
(You can of course also search for that decorator to find all of the unions in the Pants rules.)
Then as you have already seen use 
UnionRule
to declare implementations of the union.
g
I've done that, the Union seems to work. It just doesn't understand that the request can ever be made (and thus the receiving rule can never be called).
(Recognizing your username, I'm following a bit what you do with the codegen pattern in the go backend, so a rule generating requests.)
Copy code
No installed rules return the type BuildImageBundleRequest, and it was not provided by potential callers of @rule(backends.oci.util_rules.build_image_bundle:51:build_oci_bundle_package(BuildImageBundleRequest, UmociTool) -> FallibleImageBundle, gets=[Get(Targets, UnparsedAddressInputs), Get(DownloadedExternalTool, ExternalToolRequest), Get(Targets, DependenciesRequest), Get(FieldSetsPerTarget, FieldSetsPerTargetRequest), Get(Digest, MergeDigests), Get(FallibleProcessResult, Process)]).
And this is how I map them - I've tried a bunch of variations here, with a raw targets, etc...
Copy code
@rule
def ibr_to_fibr(
    request: ImageBundleRequest, union_membership: UnionMembership
) -> FallibleImageBundleRequest:
    tgt = request.target
    concrete_requests = [
        request_type(request_type.field_set_type.create(tgt))
        for request_type in union_membership[FallibleImageBundleRequest]
        if request_type.field_set_type.is_valid(tgt)
    ]
    if len(concrete_requests) > 1:
        raise ValueError(
            f"Multiple registered builders from {ImageBundleRequest.__name__} can "
            f"build target {tgt.name}. It is ambiguous which implementation to "
            f"use.\n\n"
            f"Possible implementations:\n\n"
            f"{bullet_list(sorted(generator.__name__ for generator in concrete_requests))}"
        )

    if not concrete_requests:
        return None
    first_concrete = concrete_requests[0]
    return first_concrete(tgt)
f
what about rule registrations? are you able to put up the whole plugin somewhere?
also rules shouldn't be returning 
None
, the engine expects the rule to return the declared type. you should probably just raise an exception there just like the 
>1
case
and are you seeing this error message in a test or in regular usage?
g
This is just from trying to 
pants list ::
. One sec and I'll push the very WIP code.
👍 1
https://github.com/tgolsson/pants-backend-oci , the rules in question are in 
util_rules
f
(and then obviously the subclass need not have the dataclass decorator any more)
tangentially, very nice to see an OCI backend! container images without the docker!
g
None of those solve it unfortunately 😞 I now get more errors though, so maybe it did change something
And yeah! Tried the docker backend but deterministic shas are a must for me. Plus I don't run docker anywhere... Buildah, podman, nothing else.
f
what are the errors?
also I note you have a rule that tries to produce the union base class type
I don't believe that will work
the pattern I've used and seen elsewhere in the rules is that you have a union type that can take an input (like a field set) and produce a concrete output type that can then be used later in the process.
for example, with 
GoCodegenBuildRequest
, the rules for each union impl produce a 
BuildGoPackageRequest
which is not a union and contains all information necessary to build a Go package
whereas it looks like 
FallibleImageBundleRequest
is marked as a union
note: the engine is type invariant except for unions
g
Yeah. I guess I could do that, though at that point I might as well manually check each variant since it's the build flow that changes, not the build args. I wanted do be agnostic over how the preceding image is built, without having to maintain a central point of knowledge.
In practice, there's going to be three main ways you have a base image in the end: "empty", an externally pulled image, or the output from a previous step. So it's not an egregious amount of centralization, just felt like union rules was the perfect solution for this. 😛
f
union could be. maybe there is a way to represent each step of the build pipeline with a single "build step" type? and much like 
Digest
represents a filesystem tree, can an image be represented by some value during processing? maybe the sha-256 of the intermediate image.
and the union for the build step would have as a field, the identity of any previous image to apply the step to
g
The problem is the recursiveness itself. When building image Foo with base Bar, I need to know how to source Bar. So I can't know the digest for the intermediate step until that has been built.
(I.e, I want to say "produce B and give me the digest for that, and I'll build on top of it")
I think your solution solves the opposite question - given that I know the result of building a base, how do I build the successor? But that seems backwards to me.
f
if the prior image step input is an optional, say 
ImageHandle | None
as the type, then the base steps just either ignore the prior image handle or raise an exception that they were invoked incorrectly
the base steps are thus equivalent to an intermediate step
you may find some inspiration in the linter/formatter rules since they seem to me to have a similar "recursive" nature by being applied in order to set of sources
the union base class for a step could be called an 
ImageTransformer
the dataclass for an individual step application can store which 
ImageTransformer
to use to get the previous image
then each rule for an implementation of 
ImageTransformer
would run an 
await Get(ImageStepApplication, ImageTransformer, request.prior_step_transformer_class(request.prior_step))
essentially each step is implemented by asking the engine to apply the previous step's transformer, then do its own processing
I'm vague still on how to write that down precisely as Pants rules, but formulating the problem this way is similar to how build dependency graphs in Pants are traversed to produce a build
the difference being you have 1 dependency for an OCI pipeline versus N dependencies for stuff like the Go build or JVM builds
g
Hmm. So 
request.prior_step_transformer_class(request.prior_step)
would be creating the union-variant that I'm currently trying to create with a rule?
I'm worried that'll lead to the same issue because the subtype isn't explicitly mentioned in the scope of the function or produced by any other rule.
f
well you would need a builder function to construct the list of transformer steps together
it would presumably know all of the steps, right?. it can then construct the chain without actually applying the image tranformations. the transformations would only happen once the top of the stack is requested for its image.
this seems similar to me to how running 
flatMap
on a Scala 
Future
just records the operation but doesn't actually apply it until the base 
Future
resolves
but here in this case, the top of the stack just needs to be requested for its image, and then that rule asks for its step prior image, until you hit a base transformer that just produces an image without asking the engine for a prior image application.
it's essentially what we do in the Go backend. We use 
BuildGoPackageTargetRequest
to invoke the builder logic which then constructs a tree of 
FallibleBuildGoPackageRequest
instances which just record what builds will happen.
then when the actual build is needed, the applicable goal will ask the engine to convert the 
FallibleBuildGoPackageRequest
for the root of the tree into a 
BuiltGoPackage
any way, that is probably too complicated
you will note that the rule for 
BuildGoPackageTargetRequest
just hard-codes the 2 or 3 cases it needs to convert
g
Sigh; sometimes I consider switching to goat farming 🤌 I think what the error is trying to tell me isn't any of the things I thought. What it is saying is that there's no third rule anywhere to "initialize" this chain of requests...
There was a bunch of issues in my rule graph, and the errors were mostly pointing me in the wrong direction (and being quite unhelpful). Using 
-ltrace
gave me a graphviz view to see what it was trying to resolve, so I knew where to start pulling. Now it at least moves forward.
f
yeah the rule graph errors are not helpful at all. I still need help from other Pants maintainers to make sense of them. 😞
Open in Slack
PreviousNext
Powered by