Hello :wave:! Looking for a little guidance with ...
# plugins
l
Hello 👋! Looking for a little guidance with a plugin I wrote internally, I think I've addressed my problem for now but hoping to get some insight into properly fixing my code (maybe I'm just approaching it all wrong to begin with). After updating from 2.21 to 2.24 (identified error from 2.23) I'm getting this error:
Copy code
native_engine.IntrinsicError: Get(InterpreterConstraints, InterpreterConstraintsRequest, InterpreterConstraintsRequest(addresses=Addresses([Address(src/******/__init__.py:lib), Address(src/******/app.py:lib), Address(src/******/conf.py:lib), Address(src/******/schemas.py:lib), Address(src/******/__init__.py:lib), Address(src/******/cli.py:lib), Address(src/******/core.py:lib), Addr
ess(src/******/cors.py:lib), Address(src/******/models.py:lib)]), hardcoded_interpreter_constraints=None)) was not detected in your @rule body at rule compile time. Was the `Get` constructor called in a non async-function, or was it inside an async function defined after the @rule? Make sure the `Get` is defined before or inside the @rule body.
I've drilled down and identified my usage of
create_python_repl_request
as the cause. I'm able to fix my problem by simply copying the whole rules function code into my module which isn't a big problem - but obviously nicer to simply leverage the existing code.
w
I've seen this error when there is a local async rule, and it's not above the call site
Like, what used to be
@rule_helper
I think, but now is just
async def whatever()
l
hmm
This is my plugin code:
Copy code
import logging
from dataclasses import dataclass
from typing import ClassVar

from pants.backend.python.goals.repl import PythonRepl, create_python_repl_request
from pants.backend.python.subsystems.setup import PythonSetup
from pants.backend.python.util_rules.pex_environment import PexEnvironment
from pants.core.goals.package import BuiltPackage, BuiltPackageArtifact, OutputPathField, PackageFieldSet
from pants.core.util_rules.environments import EnvironmentField
from pants.engine.env_vars import CompleteEnvironmentVars
from pants.engine.fs import CreateDigest, Digest, FileContent
from pants.engine.process import Process, ProcessResult
from pants.engine.rules import Get, collect_rules, rule
from pants.engine.target import (
    COMMON_TARGET_FIELDS,
    Dependencies,
    DependenciesRequest,
    DictStringToStringField,
    Field,
    StringField,
    Target,
    Targets,
)
from pants.engine.unions import UnionRule
from pants.util.logging import LogLevel

logger = logging.getLogger(__name__)


class GeneratorFunction(StringField):
    alias = "generator_function"
    required = True


class GeneratorEnvironmentField(DictStringToStringField):
    alias = "generator_environment"
    required = False


class GeneratedFile(Target):
    alias = "generated_file"
    core_fields: ClassVar[tuple[type[Field], ...]] = (
        *COMMON_TARGET_FIELDS,
        GeneratorFunction,
        GeneratorEnvironmentField,
        OutputPathField,
        EnvironmentField,
        Dependencies,
    )


@dataclass(frozen=True)
class GeneratedFileFieldSet(PackageFieldSet):
    required_fields = (GeneratorFunction, OutputPathField)

    generator_function: GeneratorFunction
    generator_environment: GeneratorEnvironmentField
    output_path: OutputPathField
    dependencies: Dependencies


@rule(level=<http://LogLevel.INFO|LogLevel.INFO>)
async def package_generated_file(
    field_set: GeneratedFileFieldSet,
    pex_env: PexEnvironment,
    python_setup: PythonSetup,
    complete_env: CompleteEnvironmentVars,
) -> BuiltPackage:
    targets = await Get(Targets, DependenciesRequest(field_set.dependencies))
    repl_impl = PythonRepl(targets=targets)
    # Because we're (ab)using another rule here that we need to call the inner
    # function of the rule, rather than just the rule itself due to issues passing
    # python setup to rule.
    repl_request = await create_python_repl_request.__wrapped__(repl_impl, pex_env, python_setup)
    env = {**complete_env, **repl_request.extra_env, **(field_set.generator_environment.value or {})}
    process_result = await Get(
        ProcessResult,
        Process(
            argv=(*repl_request.args, *field_set.generator_function.value.split(" ")),
            env=env,
            description=f"Generating file using {GeneratorFunction.alias}",
            input_digest=repl_request.digest,
            immutable_input_digests=repl_request.immutable_input_digests,
            append_only_caches=repl_request.append_only_caches,
        ),
    )
    output_path = field_set.output_path.value_or_default(file_ending="")
    generated_digest = await Get(Digest, CreateDigest([FileContent(output_path, process_result.stdout)]))
    artefact = BuiltPackageArtifact(output_path)
    return BuiltPackage(digest=generated_digest, artifacts=(artefact,))


def rules():
    return (
        *collect_rules(),
        UnionRule(PackageFieldSet, GeneratedFileFieldSet),
    )


def target_types():
    return [
        GeneratedFile,
    ]
I figure I'm probably doing something non-standard (funky) with
create_python_repl_request
here
Would it be something to do with async then?
w
await create_python_repl_request.__wrapped__(repl_impl, pex_env, python_setup)
Yeah, I dont know what that is. Supposed to be a
Get
I'm guessing, unless you're using the newer call-by-name, which has its own quirkls
l
Ah, no it's not a call-by-name, just me trying to call another rule
I guess this is just me not fully gasping the nature of the rules here
The plugin essentially just calls a user defined script to generate some artefact, I'm using the repl to run that script in a sandbox
w
Yeah, okay, so, if you want to be part of the Rule system, you kinda have to call
Get()
I can check when I get back to my computer, to see what that is - I’ve never seen it before
l
Thanks! I guess I'll have to go read the rules docs a bit harder too 😄
w
I think I saw briefly some of the chatter the past week or two, but what does this do?
l
ah, I guess I'm not doing the call the right way in any case
So the plugin just takes the path of a python script to run (on packaging) which outputs content for a file to put in dist
specifically we use this to generate an OpenAPI spec json file from the code
w
Ah, okay, I thought we already had an OpenAPI generator?
l
In this case it's a custom implementation - an existing project we migrated to pants
w
Gotcha. Okay, so off the top of my head - not suggesting this is ideal for connecting all the dots, I use adhoc_tool by default instead of plugins
However, this one sounds like it also has to be wrapped back in to the overall python package
Why did you want a repl in the first place though?
l
Ah, don't really need a repl specifically - that was just me abusing it to get a sandbox to run the script in 😅
so that all the dependencies are pulled in with a venv
w
Ah, okay, so aside from this - adhoc_tool might be an option - and it’s like, 99% of the time what I would do. With a pex, you can take that script, make a
pex_binary
out of it, in case you need to run a python file like a script. What I’m trying to think of is the least-code way to do what you want
l
Yeah, I'll look into adhoc_tool then!
w
https://gist.github.com/sureshjoshi/98fb09f2a340f7c1dad270c4887865a0#file-build-pants-sveltekit Here;s a pretty full-fledged one I used to run pnpm with Svelte - the important part is near the bottom, where I use the output back into an
archive
- maybe this could work with a python package too, but I’ve just never done it
l
Yeah I think you're right, adhoc_tool is probably exactly what I need
I've seen it before but I guess I hadn't connected the pieces in my mind
w
Oh, there you go: https://www.pantsbuild.org/stable/docs/ad-hoc-tools/integrating-new-tools-without-plugins Says in the first paragraph that you can use runnable targets with adhoc_tool, and any python file can be runnable with pants run. So, you might even be able to skip the pex (depending on what it is). All of my single-file scripts follow PEP whatever-it-is that list the requirements at the top, and then I use UV or Pex to run them
l
Beautiful!
Thanks so much!
👍 1