The projects runs smoothly on pants v2.5. To make it work with v.2.6 I already changed

TwoStepPex

and

PexInterpreterConstraints

but even if I start from scratch (by deleting

~/.cache/pants

) I get an

panic at 'Loop subgraph

error

the full error reads as follows:

16:29:21.41 [ERROR] panic at 'Loop subgraph: digraph {
    0 [ label = "NodeIndex(46885): Deleted(reason: Monomorphized, ParamsLabeled(node: @rule(pants.engine.process:244:fallible_to_exec_result_or_raise(FallibleProcessResult, ProductDescription) -> ProcessResult), in: (Process), out: ((PrefilledSetupKwargsRequest, Process, PytestCoverageDataCollection, Specs))))" ]
    1 [ label = "NodeIndex(46896): Deleted(reason: Monomorphized, ParamsLabeled(node: @rule(pants.engine.process:233:get_multi_platform_request_description(MultiPlatformProcess) -> ProductDescription), in: (Process), out: ((PrefilledSetupKwargsRequest, Process, PytestCoverageDataCollection, Specs))))" ]
    2 [ label = "NodeIndex(46897): Deleted(reason: Monomorphized, ParamsLabeled(node: @rule(pants.engine.process:266:remove_platform_information(FallibleProcessResultWithPlatform) -> FallibleProcessResult), in: (Process), out: ((PrefilledSetupKwargsRequest, Process, PytestCoverageDataCollection, Specs))))" ]
    3 [ label = "NodeIndex(46910): ParamsLabeled(node: @rule(pants.engine.process:504:find_binary(BinaryPathRequest) -> BinaryPaths, gets=[Get(BashBinary, BashBinaryRequest), Get(PyDigest, CreateDigest), Get(ProcessResult, Process), Get(FallibleProcessResult, Process)]), in: (BinaryPathRequest), out: ((BinaryPathRequest, PrefilledSetupKwargsRequest, PytestCoverageDataCollection, Specs)))" ]
    4 [ label = "NodeIndex(46919): Deleted(reason: Monomorphized, ParamsLabeled(node: @rule(pants.engine.process:484:find_bash(BashBinaryRequest) -> BashBinary, gets=[Get(BinaryPaths, BinaryPathRequest)]), in: (BashBinaryRequest), out: ((BashBinaryRequest, PrefilledSetupKwargsRequest, PytestCoverageDataCollection, Specs))))" ]
    5 [ label = "NodeIndex(46920): Deleted(reason: Monomorphized, ParamsLabeled(node: @rule(pants.engine.process:498:get_bash() -> BashBinary, gets=[Get(BashBinary, BashBinaryRequest)]), in: (()), out: ((BashBinaryRequest, PrefilledSetupKwargsRequest, PytestCoverageDataCollection, Specs))))" ]
    0 -> 1 [ label = "ProductDescription" ]
    0 -> 2 [ label = "FallibleProcessResult" ]
    3 -> 2 [ label = "Get(FallibleProcessResult, Process)" ]
    3 -> 0 [ label = "Get(ProcessResult, Process)" ]
    3 -> 4 [ label = "Get(BashBinary, BashBinaryRequest)" ]
    3 -> 5 [ label = "Get(BashBinary, BashBinaryRequest)" ]
    5 -> 4 [ label = "Get(BashBinary, BashBinaryRequest)" ]
    5 -> 5 [ label = "Get(BashBinary, BashBinaryRequest)" ]
    4 -> 3 [ label = "Get(BinaryPaths, BinaryPathRequest)" ]
}
', /__w/pants/pants/src/rust/engine/rule_graph/src/builder.rs:706
16:29:21.41 [ERROR] Please set RUST_BACKTRACE=1, re-run, and then file a bug at <https://github.com/pantsbuild/pants/issues>.
16:29:21.48 [ERROR] Rust panic

I already re-run with RUST_BACKTRACE=1 but I don't know where the corresponding log-file is stored

I have only inherited the project and the plugin was created by a former colleague.

That can be tested by disabling the custom rules.

That said, it would be easier to catch them up all the way to current Pants (2.8.x or even the upcoming 2.9.x) rather than having to step them through every intermediate version...

you can find both plugins here. Please let me know if the provided files are not sufficient - frankly, I'm not very familiar with pants. Many thanks in advance for your support, it's highly appreciated

I figured out: If I out-comment the UnionRule for our CustomSetupKwargsRequest, at least the bootstrapping process works fine:

def rules():
    return [
        *collect_rules(),
        # Register a custom KwargsRequest as a possible alternative to the default
        # Since the `is_applicable` method of our custom type always returns True,
        # Pants will always choose PrefilledSetupKwargsRequest over
        # SetupKwargsRequest.
        # UnionRule(SetupKwargsRequest, PrefilledSetupKwargsRequest),
    ]


@dataclass
class Pep440Version:
    version: str


class PrefilledSetupKwargsRequest(SetupKwargsRequest):
    """Custom Pants type for providing default arguments for the `setup_py` target."""

    @classmethod
    def is_applicable(cls, target: Target) -> bool:
        return True


@rule
async def setup_kwargs_plugin(request: PrefilledSetupKwargsRequest) -> SetupKwargs:
    if "version" in request.explicit_kwargs:
        raise ValueError(
            f"The version argument is specified in the setup_py arguments of "
            f'"{request.target.address}", but it is supposed to be derived from the '
            f"VCS tags. Please remove the version specifier and let this plugin handle "
            f"the versioning."
        )
    address_path = request.target.address.spec_path
    pep440_version = await Get(Pep440Version, SourceRootRequest(PurePath(address_path)))
    return SetupKwargs(
        {"version": pep440_version.version, **request.explicit_kwargs},
        address=request.target.address,
    )

but I don't see a reason so far why this piece should cause the rule dependency graph creation to fail

@hundreds-father-404 That said, you are far more of an expert on this than I am...

Hm so I wanted to first get Pants 2.6 and 2.7 working. I know Benjy had recommended going to straight to 2.8, but I personally recommend instead going one minor version at a time to reduce the size of diff. I couldn't reproduce with https://github.com/Eric-Arellano/example-python/tree/test-plugin @fresh-account-42296 would you be interested in pair programming on getting 2.6 to work? Rule graph errors are particularly confusing, which we know and really want to fix. It might be easier for us to debug together

@hundreds-father-404, many thanks for the recap of the pair programming -> it helps a lot to go through it once more. And I hope it'll help me to fix the remaining issue(s) with the sphinx plugin

Hi @happy-kitchen-89482, I communicated your suggestion to upstream the sphinx plugin to the projekt manager and he was quite enthusiastic. However, they have to carve out a process how to deal with code with the community. So presumably, rather next month than tomorrow and another person than myself will get in touch again to push this idea forward. Till then, I'll try my best to get it up and running

Found the issue!!! This causes issues in

artifacts.py

packages = await MultiGet(
        Get(BuiltPackage, PackageFieldSet, field_set)
        for field_set in field_sets_per_target.field_sets
    )

You're essentially invoking the

package

goal inside the codegen plugin hook. That creates a cycle between package <-> codegen. Packaging often depends on codegen as a precursor. And now you're saying that that codegen first depends on building the relevant packages. Now, sometimes you can have circular dependencies in a graph like this, which is probably why it worked in Pants 2.5. But often it causes issues like this To fix this, two options: 1. Give up on your

artifacts

target 2. Make the codegen rule less general so that you don't try to build any arbitrary package, only ones you've "blessed" to not use codegen. You could still use

await Get(FieldSetsPerTarget, FieldSetsPerTargetRequest)

, but then inspect the returning

FieldSets

, like split

PexBinaryFieldSet

vs

ArchiveFieldSet

. Rather than using the generic "union"

Get(BuiltPackage, PackageFieldSet

, use a couple more specific calls like

Get(BuiltPackage, ArchiveFieldSet)

. Warn or error if the field has any targets that are not "blessed" cc @witty-crayon-22786 - does this all sound right?

one thing to keep in mind though is that we made a significant improvement to rule graph solving in 2.8.x, which might allow the existing rules to work. because there is nothing inherently wrong with a rule graph cycle (e.g.: to codegen for some targets i must first package for some other targets, and vice versa): only a runtime cycle can’t be resolved (e.g.: to codegen for this target i must first package for this target, and vice versa)

i ended up with a pressing TODO item today, but if Eric is able to investigate whether that change in 2.8.x helps, we might be able to backport it (maybe only to 2.7.x though)

Hi 🙂 Trying out your suggestion @hundreds-father-404, to make the calls in our

artifacts

plugin more specific I ran into another problem (which is probably very easy to solve). As an intermediate attempt, I adapted the code as follows:

packages = []
    for field_set in field_sets_per_target.field_sets:
        if isinstance(field_set, NpmBuildFieldSet):
            package = await Get(
                BuiltPackage, NpmBuildFieldSet, field_set
            )
            packages.append(package)
        elif isinstance(field_set, PexBinaryFieldSet):
            package = await Get(
                BuiltPackage, PexBinaryFieldSet, field_set
            )
            packages.append(package)
        else:
            logger.warn(f"[WARN] No package is built currently for {field_set} as input PackageFieldSet")

Running this code in one target that only uses the

NpmBuildFieldSet

works fine. However, running this code on another target that also uses

PexBinaryFieldSet

, I receive the following error:

Exception: Invalid Get. Because the second argument to `Get(BuiltPackage, PexBinaryFieldSet, PexBinaryFieldSet(address=Address(fleetsim:app), entry_point=<class 'pants.backend.python.target_types.PexEntryPointField'>(alias='entry_point', address=fleetsim:app, value={repr(self.value)}, default={repr(self.default)}), output_path=<class 'pants.core.goals.package.OutputPathField'>(alias='output_path', value='fleetsim/fleetsim.pex', default=None), always_write_cache=<class 'pants.backend.python.target_types.PexAlwaysWriteCacheField'>(alias='always_write_cache', value=False, default=False), emit_warnings=<class 'pants.backend.python.target_types.PexEmitWarningsField'>(alias='emit_warnings', value=None, default=None), ignore_errors=<class 'pants.backend.python.target_types.PexIgnoreErrorsField'>(alias='ignore_errors', value=False, default=False), inherit_path=<class 'pants.backend.python.target_types.PexInheritPathField'>(alias='inherit_path', value=None, default=None), shebang=<class 'pants.backend.python.target_types.PexShebangField'>(alias='shebang', value=None, default=None), zip_safe=<class 'pants.backend.python.target_types.PexZipSafeField'>(alias='zip_safe', value=False, default=True), platforms=<class 'pants.backend.python.target_types.PexPlatformsField'>(alias='platforms', value=None, default=None), execution_mode=<class 'pants.backend.python.target_types.PexExecutionModeField'>(alias='execution_mode', value='zipapp', default='zipapp'), include_tools=<class 'pants.backend.python.target_types.PexIncludeToolsField'>(alias='include_tools', value=False, default=False)))` is annotated with `@union`, the third argument should be a member of that union. Did you intend to register `UnionRule(PexBinaryFieldSet, PexBinaryFieldSet)`? If not, you may be using the wrong type (PexBinaryFieldSet) for the third argument.

Is there something I oversaw? Or did I misunderstood your suggestion?