https://pantsbuild.org/ logo
#general
Title
# general
f

fresh-account-42296

12/10/2021, 3:49 PM
I have a problem upgrading pants from version 2.5 to 2.6
The projects runs smoothly on pants v2.5. To make it work with v.2.6 I already changed
TwoStepPex
and
PexInterpreterConstraints
but even if I start from scratch (by deleting
~/.cache/pants
) I get an
panic at 'Loop subgraph
error
the full error reads as follows:
Copy code
16:29:21.41 [ERROR] panic at 'Loop subgraph: digraph {
    0 [ label = "NodeIndex(46885): Deleted(reason: Monomorphized, ParamsLabeled(node: @rule(pants.engine.process:244:fallible_to_exec_result_or_raise(FallibleProcessResult, ProductDescription) -> ProcessResult), in: (Process), out: ((PrefilledSetupKwargsRequest, Process, PytestCoverageDataCollection, Specs))))" ]
    1 [ label = "NodeIndex(46896): Deleted(reason: Monomorphized, ParamsLabeled(node: @rule(pants.engine.process:233:get_multi_platform_request_description(MultiPlatformProcess) -> ProductDescription), in: (Process), out: ((PrefilledSetupKwargsRequest, Process, PytestCoverageDataCollection, Specs))))" ]
    2 [ label = "NodeIndex(46897): Deleted(reason: Monomorphized, ParamsLabeled(node: @rule(pants.engine.process:266:remove_platform_information(FallibleProcessResultWithPlatform) -> FallibleProcessResult), in: (Process), out: ((PrefilledSetupKwargsRequest, Process, PytestCoverageDataCollection, Specs))))" ]
    3 [ label = "NodeIndex(46910): ParamsLabeled(node: @rule(pants.engine.process:504:find_binary(BinaryPathRequest) -> BinaryPaths, gets=[Get(BashBinary, BashBinaryRequest), Get(PyDigest, CreateDigest), Get(ProcessResult, Process), Get(FallibleProcessResult, Process)]), in: (BinaryPathRequest), out: ((BinaryPathRequest, PrefilledSetupKwargsRequest, PytestCoverageDataCollection, Specs)))" ]
    4 [ label = "NodeIndex(46919): Deleted(reason: Monomorphized, ParamsLabeled(node: @rule(pants.engine.process:484:find_bash(BashBinaryRequest) -> BashBinary, gets=[Get(BinaryPaths, BinaryPathRequest)]), in: (BashBinaryRequest), out: ((BashBinaryRequest, PrefilledSetupKwargsRequest, PytestCoverageDataCollection, Specs))))" ]
    5 [ label = "NodeIndex(46920): Deleted(reason: Monomorphized, ParamsLabeled(node: @rule(pants.engine.process:498:get_bash() -> BashBinary, gets=[Get(BashBinary, BashBinaryRequest)]), in: (()), out: ((BashBinaryRequest, PrefilledSetupKwargsRequest, PytestCoverageDataCollection, Specs))))" ]
    0 -> 1 [ label = "ProductDescription" ]
    0 -> 2 [ label = "FallibleProcessResult" ]
    3 -> 2 [ label = "Get(FallibleProcessResult, Process)" ]
    3 -> 0 [ label = "Get(ProcessResult, Process)" ]
    3 -> 4 [ label = "Get(BashBinary, BashBinaryRequest)" ]
    3 -> 5 [ label = "Get(BashBinary, BashBinaryRequest)" ]
    5 -> 4 [ label = "Get(BashBinary, BashBinaryRequest)" ]
    5 -> 5 [ label = "Get(BashBinary, BashBinaryRequest)" ]
    4 -> 3 [ label = "Get(BinaryPaths, BinaryPathRequest)" ]
}
', /__w/pants/pants/src/rust/engine/rule_graph/src/builder.rs:706
16:29:21.41 [ERROR] Please set RUST_BACKTRACE=1, re-run, and then file a bug at <https://github.com/pantsbuild/pants/issues>.
16:29:21.48 [ERROR] Rust panic
I already re-run with RUST_BACKTRACE=1 but I don't know where the corresponding log-file is stored
h

happy-kitchen-89482

12/10/2021, 4:05 PM
So you have a custom plugin, it sounds like?
f

fresh-account-42296

12/10/2021, 4:57 PM
yes, we have a custom plugin defined. Do you think that this causes the problem?
I have only inherited the project and the plugin was created by a former colleague.
h

happy-kitchen-89482

12/10/2021, 5:30 PM
No, just checking that I understand the context of the question
e

enough-analyst-54434

12/10/2021, 5:43 PM
@happy-kitchen-89482 your intuition must be correct. Since the released Pants doesn't do this, the custom rules added to the rule graph must IIUC.
That can be tested by disabling the custom rules.
f

fresh-account-42296

12/10/2021, 5:56 PM
Many thanks for the hint. It definitely shows the right direction to look at: We have to custom plugins, "documentation" (using Sphinx) and "versioning" that cause the issues. If I exclude them from the backend_packages list in the pants.toml file,
./pants --version
runs through
h

happy-kitchen-89482

12/10/2021, 6:03 PM
So this is an unfortunate consequence of the Plugin API not being stable yet, sorry about that! If you're able to publish your custom plugins to a repo somewhere we can take a look at the code and see what would need to change to catch them up.
That said, it would be easier to catch them up all the way to current Pants (2.8.x or even the upcoming 2.9.x) rather than having to step them through every intermediate version...
f

fresh-account-42296

12/10/2021, 8:41 PM
Many thanks for your response @happy-kitchen-89482 👍 I'll anyways plan to upgrade to 2.8 so this is not an issue at all. I'll let you know as soon as possible when I managed to upload the plugins to a public github repo
you can find both plugins here. Please let me know if the provided files are not sufficient - frankly, I'm not very familiar with pants. Many thanks in advance for your support, it's highly appreciated
h

hundreds-father-404

12/13/2021, 7:42 PM
Hey Ralf, checking how this upgrade is going?
f

fresh-account-42296

12/14/2021, 7:06 AM
Hey Eric, many thanks for reaching out. I did not hear back from Benjy so far after I uploaded the two plugins. In the meantime, I started to read the v2.8 doc and how to implement custom plugins and I compare this to the current code -> in the best case, I'll find the problem causing piece myself. The versioning plugin I uploaded is a rather small unit - so I hope to make some progress there today
I figured out: If I out-comment the UnionRule for our CustomSetupKwargsRequest, at least the bootstrapping process works fine:
Copy code
def rules():
    return [
        *collect_rules(),
        # Register a custom KwargsRequest as a possible alternative to the default
        # Since the `is_applicable` method of our custom type always returns True,
        # Pants will always choose PrefilledSetupKwargsRequest over
        # SetupKwargsRequest.
        # UnionRule(SetupKwargsRequest, PrefilledSetupKwargsRequest),
    ]


@dataclass
class Pep440Version:
    version: str


class PrefilledSetupKwargsRequest(SetupKwargsRequest):
    """Custom Pants type for providing default arguments for the `setup_py` target."""

    @classmethod
    def is_applicable(cls, target: Target) -> bool:
        return True


@rule
async def setup_kwargs_plugin(request: PrefilledSetupKwargsRequest) -> SetupKwargs:
    if "version" in request.explicit_kwargs:
        raise ValueError(
            f"The version argument is specified in the setup_py arguments of "
            f'"{request.target.address}", but it is supposed to be derived from the '
            f"VCS tags. Please remove the version specifier and let this plugin handle "
            f"the versioning."
        )
    address_path = request.target.address.spec_path
    pep440_version = await Get(Pep440Version, SourceRootRequest(PurePath(address_path)))
    return SetupKwargs(
        {"version": pep440_version.version, **request.explicit_kwargs},
        address=request.target.address,
    )
but I don't see a reason so far why this piece should cause the rule dependency graph creation to fail
h

happy-kitchen-89482

12/14/2021, 8:40 PM
Hi, sorry for the delay, I missed your message
@hundreds-father-404 That said, you are far more of an expert on this than I am...
h

hundreds-father-404

12/14/2021, 11:44 PM
Thanks for posting the links! Trying to hook them up to an example repo to figure this out
Hm so I wanted to first get Pants 2.6 and 2.7 working. I know Benjy had recommended going to straight to 2.8, but I personally recommend instead going one minor version at a time to reduce the size of diff. I couldn't reproduce with https://github.com/Eric-Arellano/example-python/tree/test-plugin @fresh-account-42296 would you be interested in pair programming on getting 2.6 to work? Rule graph errors are particularly confusing, which we know and really want to fix. It might be easier for us to debug together
f

fresh-account-42296

12/15/2021, 6:45 AM
Good morning, @hundreds-father-404, many thanks for our effort. I guess it doesn't make things easier that you can't reproduce the problem :) pair programming sounds great - it would be my pleasure. Let me know when you have time.
❤️ 1
h

hundreds-father-404

12/15/2021, 4:35 PM
Yay it's working now! To recap from pair programming: • Issue was a rule that went from
SourceRootRequest -> SourceRoot-> Pep440Version
. For whatever reason, the engine didn't like that when used inside the
CustomSetupPyKwargs
plugin hook. It wasn't actually necessary though, they were only using
SourceRootRequest
as a "newtype" for
PurePath
and because they liked a classmethod it had. Fix was to add
Pep440VersionRequest
to "newtype"
PurePath
• There are other migration things they still have to do. Ralf plans to upgrade to 2.6 and land that, and then 2.7, then 2.8 • I encouraged Ralf to still reach out for help if they hit any other roadblocks 🙂 Ralf, fyi I had a WIP yesterday to upgrade your Sphinx plugin to Pants 2.8 with the changes made to the
SourcesField
. I didn't finish it, but gives you a head start. Will DM the diff
h

happy-kitchen-89482

12/15/2021, 8:25 PM
Yay! Do we want to upstream that sphinx plugin into core Pants BTW?
h

hundreds-father-404

12/15/2021, 8:27 PM
@fresh-account-42296 would y'all be open to that?
f

fresh-account-42296

12/15/2021, 8:30 PM
that sounds great (to me) ❤️ two caveats: I did not yet manage to make it running for 2.6 (and beyond) and I'll have to talk to the product owner of the project which I will do tomorrow
@hundreds-father-404, many thanks for the recap of the pair programming -> it helps a lot to go through it once more. And I hope it'll help me to fix the remaining issue(s) with the sphinx plugin
❤️ 1
Hi @happy-kitchen-89482, I communicated your suggestion to upstream the sphinx plugin to the projekt manager and he was quite enthusiastic. However, they have to carve out a process how to deal with code with the community. So presumably, rather next month than tomorrow and another person than myself will get in touch again to push this idea forward. Till then, I'll try my best to get it up and running
h

happy-kitchen-89482

12/16/2021, 6:01 PM
Great! Thanks for pushing that forward
f

fresh-account-42296

12/20/2021, 9:43 AM
Hi @hundreds-father-404, sorry for being pushy... but would it be possible from your side to give a rough estimate when you guys think you have time to look into the issue with the plugins? The only reason I am asking is that I want to manage expectations on the side of the product team properly. Any time frame is plausible.
h

happy-kitchen-89482

12/20/2021, 10:48 AM
Hey @fresh-account-42296 No worries about being pushy! Eric will get back to you after he starts his work day in a few hours. Remind me what you're waiting on?
f

fresh-account-42296

12/20/2021, 10:51 AM
Hey Benjy - I know, I am little too early for Eric to reply ☺️ Last week, I sent Eric an archive with our plugins that he wanted to share we one of your team mates - so that you guys can have a look if the weird dependencies that we found in two of our plugins might point towards a bug in the rule graph algorithm
h

hundreds-father-404

12/20/2021, 4:20 PM
Hi, happy monday. I'm planning to look today and will pull in Stu today or tomorrow if I can't figure it out A lot of us our out for Christmas and New Years tho, so if the problem requires changing Pants code itself then it might take a bit. I'm hoping instead it's something we can work around by changing your plugin code
f

fresh-account-42296

12/20/2021, 4:26 PM
Hi Eric, many thanks for your update. That's all I need 👍 And if it is really that Pants needs to be changed: No worries at all regarding the time frame - at least, we have a running system for now
❤️ 1
h

hundreds-father-404

12/20/2021, 11:15 PM
Reminder to myself / a tip: when deactivating other backends to try to get a minimal diff, make sure you still are activating any relevant rules those backends provide, like that you activate
pants.backend.python
to be able to use
VenvPex
in your rules
❤️ 1
Found the issue!!! This causes issues in
artifacts.py
Copy code
packages = await MultiGet(
        Get(BuiltPackage, PackageFieldSet, field_set)
        for field_set in field_sets_per_target.field_sets
    )
You're essentially invoking the
package
goal inside the codegen plugin hook. That creates a cycle between package <-> codegen. Packaging often depends on codegen as a precursor. And now you're saying that that codegen first depends on building the relevant packages. Now, sometimes you can have circular dependencies in a graph like this, which is probably why it worked in Pants 2.5. But often it causes issues like this To fix this, two options: 1. Give up on your
artifacts
target 2. Make the codegen rule less general so that you don't try to build any arbitrary package, only ones you've "blessed" to not use codegen. You could still use
await Get(FieldSetsPerTarget, FieldSetsPerTargetRequest)
, but then inspect the returning
FieldSets
, like split
PexBinaryFieldSet
vs
ArchiveFieldSet
. Rather than using the generic "union"
Get(BuiltPackage, PackageFieldSet
, use a couple more specific calls like
Get(BuiltPackage, ArchiveFieldSet)
. Warn or error if the field has any targets that are not "blessed" cc @witty-crayon-22786 - does this all sound right?
❤️ 1
w

witty-crayon-22786

12/21/2021, 12:17 AM
thanks for investigating! sitting down to dinner, but will look first thing tomorrow
h

hundreds-father-404

12/21/2021, 12:18 AM
Thanks, will DM you the zip file with my changes included
w

witty-crayon-22786

12/21/2021, 4:08 PM
so, yea: that diagnosis seems likely.
one thing to keep in mind though is that we made a significant improvement to rule graph solving in 2.8.x, which might allow the existing rules to work. because there is nothing inherently wrong with a rule graph cycle (e.g.: to codegen for some targets i must first package for some other targets, and vice versa): only a runtime cycle can’t be resolved (e.g.: to codegen for this target i must first package for this target, and vice versa)
👍 2
i ended up with a pressing TODO item today, but if Eric is able to investigate whether that change in 2.8.x helps, we might be able to backport it (maybe only to 2.7.x though)
h

hundreds-father-404

12/21/2021, 4:16 PM
2.8 is a bit nontrivial for them to upgrade to because the Target API changes (mea culpa), but good suggestion. I could try locally backporting to 2.7 and use pant_from_sources if the above workarounds aren't desirable
f

fresh-account-42296

12/21/2021, 4:28 PM
Hi @hundreds-father-404 and @witty-crayon-22786, many thanks for your work and effort! That's great news. I have to more thoroughly work through your suggestions to better understand what they would mean for our project (and for me in terms of implementation effort 😁)
❤️ 1
Hi 🙂 Trying out your suggestion @hundreds-father-404, to make the calls in our
artifacts
plugin more specific I ran into another problem (which is probably very easy to solve). As an intermediate attempt, I adapted the code as follows:
Copy code
packages = []
    for field_set in field_sets_per_target.field_sets:
        if isinstance(field_set, NpmBuildFieldSet):
            package = await Get(
                BuiltPackage, NpmBuildFieldSet, field_set
            )
            packages.append(package)
        elif isinstance(field_set, PexBinaryFieldSet):
            package = await Get(
                BuiltPackage, PexBinaryFieldSet, field_set
            )
            packages.append(package)
        else:
            logger.warn(f"[WARN] No package is built currently for {field_set} as input PackageFieldSet")
Running this code in one target that only uses the
NpmBuildFieldSet
works fine. However, running this code on another target that also uses
PexBinaryFieldSet
, I receive the following error:
Copy code
Exception: Invalid Get. Because the second argument to `Get(BuiltPackage, PexBinaryFieldSet, PexBinaryFieldSet(address=Address(fleetsim:app), entry_point=<class 'pants.backend.python.target_types.PexEntryPointField'>(alias='entry_point', address=fleetsim:app, value={repr(self.value)}, default={repr(self.default)}), output_path=<class 'pants.core.goals.package.OutputPathField'>(alias='output_path', value='fleetsim/fleetsim.pex', default=None), always_write_cache=<class 'pants.backend.python.target_types.PexAlwaysWriteCacheField'>(alias='always_write_cache', value=False, default=False), emit_warnings=<class 'pants.backend.python.target_types.PexEmitWarningsField'>(alias='emit_warnings', value=None, default=None), ignore_errors=<class 'pants.backend.python.target_types.PexIgnoreErrorsField'>(alias='ignore_errors', value=False, default=False), inherit_path=<class 'pants.backend.python.target_types.PexInheritPathField'>(alias='inherit_path', value=None, default=None), shebang=<class 'pants.backend.python.target_types.PexShebangField'>(alias='shebang', value=None, default=None), zip_safe=<class 'pants.backend.python.target_types.PexZipSafeField'>(alias='zip_safe', value=False, default=True), platforms=<class 'pants.backend.python.target_types.PexPlatformsField'>(alias='platforms', value=None, default=None), execution_mode=<class 'pants.backend.python.target_types.PexExecutionModeField'>(alias='execution_mode', value='zipapp', default='zipapp'), include_tools=<class 'pants.backend.python.target_types.PexIncludeToolsField'>(alias='include_tools', value=False, default=False)))` is annotated with `@union`, the third argument should be a member of that union. Did you intend to register `UnionRule(PexBinaryFieldSet, PexBinaryFieldSet)`? If not, you may be using the wrong type (PexBinaryFieldSet) for the third argument.
Is there something I oversaw? Or did I misunderstood your suggestion?
h

hundreds-father-404

12/21/2021, 10:41 PM
Ugh dang it, I was hoping this wouldn't happen. Fixed in https://github.com/pantsbuild/pants/pull/13833, which is currently only in Pants 2.9 but could be backported K, how feasible is upgrading from Pants 2.6 to 2.8 for you all? You'd comment out the problematic snippet for now, and then hopefully Stu is right that the improvements to the rule graph will fix the problems you're having in 2.6 and 2.7 Otherwise, we could explore backporting #13833 to probably Pants 2.7
f

fresh-account-42296

12/21/2021, 10:55 PM
let's give it a try! I'll try to go straight to 2.8. But only after some sleep 😴 😄