Hey, I have a

MultipleSourcesField

where I just provide the

default

field. However, in

pants peek

no sources are listed, I just see

sources_raw

. Do I need to write custom code to hydrate the sources?

What rules are available by default in a RuleRunner? None? Some? For context, I'm writing a unit test which has to go through a target generator. The target generator works outside the test, but inside the test it fails with:

E         No installed rules return the type GenerateCargoTargetsRequest, and it was not provided by potential callers of @rule(pants_cargo_porcelain.target_types:172:generate_cargo_generated_target(GenerateCargoTargetsRequest, RustSubsystem, RustupTool) -> GeneratedTargets, gets=[Get(SourceFiles, [SourceFilesRequest]), Get(RustToolchain, [RustToolchainRequest]), Get(ProcessResult, [CargoProcessRequest])]).

My runner is dumb as rocks:

rule_runner = RuleRunner(
        rules=[
            *register.rules(),
            *clippy_register.rules(),
            QueryRule(Partitions, [CargoClippyRequest.PartitionRequest]),
            # QueryRule(LintResult, [CargoClippyRequest.Batch]),
        ],
        target_types=register.target_types(),
    )

Which at this point is everything in the plugin.

register.rules()

bundles the following from my target types.

def rules():
    return [
        *collect_rules(),
        UnionRule(GenerateTargetsRequest, GenerateCargoTargetsRequest),
    ]

But I'm guessing this happens "one step up" where something has to do a union lookup...

Is there any way to interact more directly with the cache from a plugin? Ignoring named caches for now. Right now as I understand it, the cache is Request -> Digest. Let's say I want to cache a file. The file is one of many outputs from a rule. I want to be able to recall one or more of those files later, but not all. The recalling rule might be completely unrelated to the original generator. We can assume files uniquely a specific set of data, so no collisions/mismatch. (This might be one of those Stackoverflowy X-Y problems)

I'm working on a custom plugin and I did this:

content = FileContent(path=versions_file_path, content="some string")
digest = await Get(Digest, CreateDigest([content]))

That resulted in the infamous

Filesystem changed during run: retrying 'MyGoal'

infinite loop. I was really confused. Upon further inspection of the run logs, I found this:

09:14:56.95 [31m[ERROR][0m panic at 'called `Result::unwrap()` on an `Err` value: "Field `content` was not convertible to type alloc::vec::Vec<u8>: PyErr { type: <class 'TypeError'>, value: TypeError(\"'str' object cannot be interpreted as an integer\"), traceback: None }"', src/intrinsics.rs:422

So I updated the code above to call

.encode()

on the string before passing it as

content

and things started working. However, I feel like my mistake should have resulted in an error / failure instead of the infinite

Filesystem changed

loop. Should I open a bug for this?

Hi, I need some inputs/thoughts on protobuf-codegen for go-grpc.

grpc-gateway

, https://github.com/grpc-ecosystem/grpc-gateway can generate a REST API for grpc services, but it has to be generated in go. This means changing/modifying https://github.com/pantsbuild/pants/blob/main/src/python/pants/backend/codegen/protobuf/go/rules.py

codegen.protobuf.go

. My current approach is, to copy the official source code of codegen.go.protobuf.rules to local

pants_plugin

, and use this custom code generator. For this custom plugin, I am generating a local

go.mod

and

go.sum

which includes

grpc-gateway

packages, and any relevant changes like downloading binaries (https://github.com/pantsbuild/pants/blob/main/src/python/pants/backend/codegen/protobuf/go/rules.py#L596) or adding plugin here https://github.com/pantsbuild/pants/blob/main/src/python/pants/backend/codegen/protobuf/go/rules.py#L500 Do you think it's the right approach ? If yes, then do you think this can/should be added to official code generator(controlled by some variable, like generate_grpc_gateway)? If yes, then once I have my local plugging stabilized, I can create a PR.

What's the right way to determine if two

Address

objects are equivalent if one is relative and one is absolute? I tried comparing their

address.spec

, which documentation claims will prepend

//

to any build root level targets, but I am not seeing that behavior in actuality, the relative specs are still missing the

//

.

Hey people! I would like to use Pantsbuild to Oras artifacts. Was thinking about maybe writing a plugin with a custom target to do this, but I am a little bit stumped by how to do this. Currently reading the docs and looking at code for the Helm backend which should be quite similar to what I am trying to do. Any recommendations for more docs (outside of the pants website), or maybe even which backends / existing plugins are simple and good examples for how to write plugins?

Are there any sanctioned/Pants-based mechanics to prevent a rule from running in parallel? I've got a race condition in my Rust plugin while running tests; where I think that multiple rule runners end up installing the toolchain at the same time, and

rustup

crashing out because of that.

Getting this error when trying to use my custom target type and field set with

TargetRootsToFieldSets

- what am I missing here?

10:45:22.46 [ERROR] 1 Exception encountered:

Engine traceback:
  in select
    ..
  in plugins.splat.goals.my_plugin_goal
    `my-plugin-goal` goal

Traceback (most recent call last):
  File "/Users/nick.dellosa/Library/Caches/nce/4ce2ab7b680d1a68593ba94c1a1eb8a45d8b6cc2fd7c215163282f168119ea3d/bindings/venvs/2.18.0.dev7/lib/python3.9/site-packages/pants/engine/internals/selectors.py", line 626, in native_engine_generator_send
    res = rule.send(arg) if err is None else rule.throw(throw or err)
  File "/Users/nick.dellosa/Projects/data-platform/pants-plugins/plugins/myplugin/goals.py", line 385, in splat_deploy
    target_roots_to_splat_deploy_field_sets = await Get(
  File "/Users/nick.dellosa/Library/Caches/nce/4ce2ab7b680d1a68593ba94c1a1eb8a45d8b6cc2fd7c215163282f168119ea3d/bindings/venvs/2.18.0.dev7/lib/python3.9/site-packages/pants/engine/internals/selectors.py", line 118, in __await__
    result = yield self
  File "/Users/nick.dellosa/Library/Caches/nce/4ce2ab7b680d1a68593ba94c1a1eb8a45d8b6cc2fd7c215163282f168119ea3d/bindings/venvs/2.18.0.dev7/lib/python3.9/site-packages/pants/engine/internals/selectors.py", line 626, in native_engine_generator_send
    res = rule.send(arg) if err is None else rule.throw(throw or err)
  File "/Users/nick.dellosa/Library/Caches/nce/4ce2ab7b680d1a68593ba94c1a1eb8a45d8b6cc2fd7c215163282f168119ea3d/bindings/venvs/2.18.0.dev7/lib/python3.9/site-packages/pants/engine/internals/specs_rules.py", line 552, in find_valid_field_sets_for_target_roots
    field_set_types=union_membership[request.field_set_superclass],
  File "/Users/nick.dellosa/Library/Caches/nce/4ce2ab7b680d1a68593ba94c1a1eb8a45d8b6cc2fd7c215163282f168119ea3d/bindings/venvs/2.18.0.dev7/lib/python3.9/site-packages/pants/engine/unions.py", line 134, in __getitem__
    return self.union_rules[union_type]  # type: ignore[return-value]
  File "/Users/nick.dellosa/Library/Caches/nce/4ce2ab7b680d1a68593ba94c1a1eb8a45d8b6cc2fd7c215163282f168119ea3d/bindings/venvs/2.18.0.dev7/lib/python3.9/site-packages/pants/util/frozendict.py", line 77, in __getitem__
    return self._data[k]
KeyError: <class 'plugins.myplugin.targets.MyPluginFieldSet'>

Anyone know of a guide or tutorial on adding dependency inference? I'm thinking about adding a plugin to compile C code

I am currently at the stage where I’ve managed to register a Target and a rule, and I can run

pants publish :my-target

. But nothing happens! How do I know if Pants is actually running my rule? If I comment out the rule Pants complains about missing paths from all of the other kinds of publishable targets in my repo (Docker image and Helm chart).

Hmm, I can’t figure out why the ORAS binary is not made available to the publish process. It is downloaded as a tar.gz archive (successfully I think), but I cannot call the contained binary. Does this look correct? https://github.com/Peder2911/pants-oras-plugin/blob/main/pants-plugins/oras/subsystem.py

What's the best way to mock digests for unit testing?

Hmm, so my ORAS plugin would have to consider remote state in the registry when figuring out what to push, which means that my rules are going to be pretty impure. Is this a fundamental problem or would it be OK? I am guessing that the Get(ProcessResult) calls in the rules would invalidate the cache and make sure that I don’t reuse stale remote state?

@famous-xylophone-36532 OK; so. Dumb question here maybe but after reading the ORAS docs, how do you actually use it? If you're using actual containers elsewhere, how do they use these resources? I can find a lot of tools (configsync, crane, ...) that leverage attachments or OCI artifacts, but nothing that actually downloads a blob and runs something on them. It feels like I should be able to use this like GCS to deliver artifacts, but I can't see how that works. 😛

How do I get the digest from a Dependencies field?

I have several fields that need to be formatted

name <email@domain>

. Validating email seems like a pretty common thing, but a quick search for email in the pants codebase didn’t turn up anything. Are there any helpful utils I should use? Or is regex the way to go?

The documentation here mentions that the DigestEntries object can be used to manipulate the paths in a digest, but it does not show how. https://www.pantsbuild.org/docs/rules-api-file-system#digestentries-light-weight-handles-to-files. Does anyone have an example of this? I want to shuffle some files around before adding them.

It would be really nice to embed a table in one my

help_text

strings. Are there any other help texts that do that? Would it be formatted like a markdown table?

Is there a suggested way to get a reproducible pants install in the prescence of thirdparty requirements? For example: with a

requirements.txt

next to

register.py

, or even with

[GLOBAL].plugins

. Background: I just realised today that the

plugins = ["hdrhistogram"]

line we have in our

pants.ci.toml

on recommendation from https://www.pantsbuild.org/docs/using-pants-in-ci seems to be unpinned. Presumably I could go and replace everything with transitively pinned

==

requirements, but that feels... unsatisfying; is there another way?

Is anyone else interested in the nfpm backend (which can build rpm, deb, apk, archlinux packages)? Interested enough to help with it? Here is the plugin: https://github.com/pantsbuild/pants/pull/19308

Im trying to create an in-repo plugin where the rest of my python sources are using

interpreter_constraints = ['==3.10.*']

from

pants.toml

but I am having a hard time wrapping my head around the interpreter requirements for pants itself. I can't ever seem to generate lockfiles as no candidate of

pants

is available for python 3.10. I've tried setting an

interpreter-constraints = ['==3.9.*']

on the

python_sources

in the plugin build file but it had no effect. Is there a more complete example I could look at?

Hmm, I'm in a huge mess with circular dependencies in Rust workspaces, and I'm unsure of what tools Pants offers to help disambiguate this. I have multiple ways of fixing it by just adding more targets, but they would no longer map to the conceptual model, which feels bad. I can solve this for "output" targets like binaries, tests and libraries, as they are separate targets. The dependency chain there ends up going for example

binary_crate -> workspace -> {all_member_packages}

. However, for "non-output" targets it becomes weird. For example,

clippy

operates on a package but needs the whole workspace. So then we get

package -> workspace -> {all_member_packages}

- and those member packages then depend back on the package we're analysing via the workspace. I can definitely solve this by just adding more indirection, but having a specific target just for linting feels conceptually wrong. Maybe @fast-nail-55400 has some thoughts from working with go?

I'm upgrading from Pants 2.16.0 to 2.18.0. We have a number of internal plugins in the repo. There were six plugin unit tests that were failing. I fixed five of them by swapping out

pants.backend.python.goals.setup_py

with `package_dists`; looks like the file was refactored in see this commit. However, I still have one test that is failing. The output looks something like this:

E       ValueError: Encountered 27 rule graph errors:
E         No installed rules return the type AddPrefix, and it was not provided by potential callers of @rule(<intrinsic>(AddPrefix) -> Digest).
E           If that type should be computed by a rule, ensure that that rule is installed.
E           If it should be provided by a caller, ensure that it is included in any relevant Query or Get.
E         No installed rules return the type CreateArchive, and it was not provided by potential callers of @rule(pants.core.util_rules.archive:47:create_archive(CreateArchive) -> Digest, gets=[Get(Digest, [CreateDigest]), ZipBinary, BashBinary, TarBinary, Get(Digest, [MergeDigests]), Get(ProcessResult, [Process])]).
E           If that type should be computed by a rule, ensure that that rule is installed.
E           If it should be provided by a caller, ensure that it is included in any relevant Query or Get.
E         No installed rules return the type CreateDigest, and it was not provided by potential callers of @rule(<intrinsic>(CreateDigest) -> Digest).
E           If that type should be computed by a rule, ensure that that rule is installed.
E           If it should be provided by a caller, ensure that it is included in any relevant Query or Get.

...

Any ideas on what other changes might be causing this?

I have a weird rule error I don't understand, at least not fully. I have the following two rules:

@rule(desc="Find all cargo packages in project")
def find_all_cargo_targets(all_targets: AllTargets) -> AllCargoTargets:
    ...

@rule(desc="Assign packages to workspaces")
async def assign_packages_to_workspaces(
    all_cargo_targets: AllCargoTargets,
) -> CargoPackageMapping:
    ...

And I'm now implementing a new goal;

generate-lockfiles

, which should make use of this.

@rule
async def determine_rust_user_resolves(
    _: KnownCargoUserResolveNamesRequest,
    cargo_targets: AllCargoTargets,
) -> KnownUserResolveNames:
    ...

works, but if I replace the second argument with a

CargoPackageMapping

or try to query for one inside that rule, it breaks the rule graph. I'm assuming this is because of something that the

assign_package_to_workspaces

rule does, but the only things it looks for is

SourceFiles

,

DigestContents

, and

RawSpecs

. Neither of those should - I think? - need resolves to be known. Or maybe they do, in order to validate fields..?

I'm wondering if the rule graph solver could detect and diagnose the cycle that led to islands, rather than the islands themselves. I've no idea how the solver works though, but that seems to be what happened above. And my intuition for how to fix it (look for added edges, not missing) seems to have helped. Maybe would be better UX? If possible.

Our pants repo has a number of processes which build PEX files and then invoke those PEX files using

VenvPexProcess

and

ProcessResult

. Currently, we just do this after we get the `ProcessResult`:

print(result.stdout.decode())
print(result.stderr.decode())

This is great and all, but is there a way to output log lines as they're being produced? This would be super useful for long-running processes

I have a repo with some custom pants plugins and I'm seeing some bizarre behavior after making a change that I don't think should have caused any failures. I have three rules A, B, and C. B uses

await Get(ProcessResult, CRequest....)

to call rule

C

.

C

has dependencies on other rules, but they're all rules that are provided by pants; none of them are custom rules from our repo. Until today,

A

didn't depend on any other rules. Everything has been working quite well. Today, I updated

A

to depend on

C

in the same way that

B

depends on

C

. However, this is resulting in rule graph errors anytime I run any pants commands. I'll attach the stacktrace in the thread, but the main error message is this:

ValueError: Encountered 225 rule graph errors:

. However, the exact number of alleged rule graph errors is random; rerunning the same command potentially produces a different number of rule graph errors than the previous run even if I don't make any changes. If I comment out the

await Get(ProcessResult, CRequest...)

line in

A

, the rule graph errors go away and pants is happy again. I've tried adding the

-ldebug --print-stacktrace

flags, but this doesn't result in any additional helpful output being produced. Any ideas on what's happening here? I'm really confused as (1) calling this rule from

B

is fine but causes everything to start on fire when I try to use it from

A

(2) nothing is calling this function yet. Is pants scanning the source code of each rule function in order to infer dependencies?

The docs have this bit about Streaming results (advanced). Is it possible to set this up for my own custom rules or goal rules? The docs don't say it's possible, so I'm poking around in the pants source code to see how it's configured for fmt, lint, and test

If you are going to use pants to lint/test/etc your in-repo plugins (the plugins have BUILD files), there isn't currently a way to use the pants Python interpreter for that right?? You need Python 3.9 on the workstation just like "regular" targets?