I am looking to create a `read_file_contents(<p...
# plugins
f
I am looking to create a
read_file_contents(<path/in/repo>)
plugin or target extension .. Such that this is possible.
Copy code
scala_artifact(
    group="org.typelevel",
    artifact="cats-core",
    version=read_file_contents("VERSION"),
    packages=["cats.**"],
)
Copy code
docker_image(
    name="demo",
    repository="example/demo",
    image_tags=[read_file_contents("VERSION"), "example"]
)
Copy code
python_distribution(
    name="mydist",
    dependencies=[
        # Dependencies on code to be packaged into the distribution.
    ],
    provides=python_artifact(
        name="mydist",
        version=read_file_contents("VERSION"),
    ),
    wheel_config_settings={"--global-option": ["--python-tag", "py37.py38.py39"]},
)
what type of plugin, or target am i needing to create - and is there something similar -
parametrize
almost feels similar, but it is generating new targets (replacing the original I assume)
ah - StringField .. So am I right in guessing I would want to create a StringField returning type of plugin (does not seem to be a StringFieldRequest type .. digging.. )
Okay - now I am stuck >> i think i want to make it as a subclass of StringField, however • how do I register a new Field I have tried two approaches ..
Copy code
@dataclass(frozen=True)
class FileReadRequest(ABC):

    @classmethod
    def is_applicable(cls, _: Target) -> bool:
        return True

@rule
async def read_file(request: FileReadRequest) -> StringField:

    original_kwargs = request.explicit_kwargs.copy()
    build_file_path = request.target.address.spec_path

    version_file = original_kwargs.pop("filename", None)
    return StringField(value=version_file)
(this is just returning a "fake" filename - as a new tag to docker )I have the file reading code, as we use that for the SetupKwargs) ..
Copy code
# register.py

from myplugin.general import file_reading

def rules():
    return [file_reading.read_file]
Copy code
docker_image(
  name="myspecial",
  repository="myspecial",
  dependencies= [
    ...
  ],
  tags=[
       "cicd",
       read_file(filename="extra_tag"),
  ]
)
but that says
Copy code
❯ pants package tools/containers/myspecial::
22:37:19.13 [ERROR] 1 Exception encountered:

Engine traceback:
  in `package` goal

MappingError: Failed to parse ./tools/containers/myspecial/BUILD:
ParseError: tools/containers/myspecial/BUILD:12: Name 'read_file' is not defined.
The other attempt - I was trying to subclass StringField
Copy code
class FileReadingField(StringField):
    alias = "read_file"
    value: Optional[str]
    default = None
    help = "Version from a fil"

   @classmethod
   def compute_value(
and use compute_value but • how do i register this new Field ? • i will need to run the async read file approach, but can that be done within a classmethod ?
Copy code
async def read_file_from_repo(file_path: str) -> str:
        """returns the file contents"""
        digest_contents = await Get(
            DigestContents,
            PathGlobs(
                [file_path],
                description_of_origin="`read_file()` plugin",
                glob_match_error_behavior=GlobMatchErrorBehavior.error,
            ),
        )
        c = digest_contents[0].content.decode()
        return c
@square-psychiatrist-19087 thanks for pointing to that other thread. :-) what I do realise is I need to mKe something that is the same as
env(...)
I did find it, but it doesn't look obvious as to how it is registering itself .. need to look deeper
s
You said that you have VERSION files for your projects, where do they come from? Usually it's easier to store information in pants BUILD files, and then put it into other files, for example you could do
Copy code
VERSION="0.0.1"
shell_command(command=f"echo ${VERSION} > VERSION", output_files=["VERSION"])
f
There is a much bigger thread on this one, but summary, we are using semver, conventional commits "per package" where tagging happens at a "directory" level. • My describing what we do currently - https://pantsbuild.slack.com/archives/C046T6T9U/p1708820848157369 • This is the python only method I wrote. https://github.com/rbuckland/pants-playground
Where I am "ultimately" heading is 1. create a plugin that reads files, (with a regex filter) for arbitrary general use ◦ can be used for all sorts of lifting things into BUILD files 2. create a per package, versioning plugin, that generates
VERSION
and
CHANGELOG.md
files, (using
convco
most likely, as a library under the hood) but my first step is (1)
> You said that you have VERSION files for your projects, where do they come from? currently - convco, (https://github.com/convco/convco/) via a macro
:bump_version()
this looks like ..
Copy code
# BUILD
version_and_changelog()

poetry_requirements(
    name = "reqs",
...
that adds
:bump_version
:gen_changelog
which the CI/CD pipeline runs, intelligently, for "paths" that only had a change
h
@fresh-continent-76371 the critical thing to consider here is invalidation - what will cause the target to be re-evaluated when the content of the file changes?
To plug in to the existing Pants mechanisms for this, you'll probably want the file to be wrapped by a
files()
target, and instead of
version=read_file_contents()
you'll need some rule that acts on that file target
If this is done naively then changing the file contents won't cause anything to re-run...
So you need some general target processing rule that looks at
read_file_contents()
and sucks in the content of a files() target
Something like that, I'm fuzzy on the details
f
Yeah - I also figured that- my issue right now is th3 actual design Is it a subclass of StringField .. is it a *Request -> StrField . At the moment the error is simply that read_file_contents is not valid..
h
But the important part is that the file has to be in the
sources
of a
files()
target
f
Target type or rule? :-) or something else 🤔
h
You're looking to modify BUILD file parsing itself, it sounds like
👍 1
f
Even though I have written two plugins and a bunch of macros .. I genuinely am still confused :-(
You're looking to modify BUILD file parsing itself, it sounds like
o yes I think so
I want to write an all purpose , for all backends ..
read_file_contents(filename=.., [filenames=..], [regex=..])
Which returns a str/StrField
h
So probably what you want is something like:
Copy code
file(source="VERSION")

scala_artifact(
    group="org.typelevel",
    artifact="cats-core",
    version=read_file_contents(":version"),
    packages=["cats.**"],
)
(we can maybe macro this up so that you don't need the explicit
file()
target later) Where
read_file_content()
effectively creates a dependency on the file target. Then invalidation will work.
But a
scala_artifact
doesn't take dependencies today, so that would have to change
There are some details to figure out here, but I think you want to model this as a dependency on a
file()
target.
Start with making this work via explicit file target and explicit dependency, and then we can figure out how to make it less verbose
f
Is there not a way to have a plugin/ return a string .. and the Dependency resolver works out it needs to execute it ? And that way what ever wants a string , it goes there ? I am trying to inject a string first Eg
Copy code
docker_image(
   name="myspecial",
   repository="myspecial",
   dependencies= [
     ...
   ],
   tags=[
        "cicd",
        read_file_contents(filename="VERSION"),
   ]
 )
h
again, the main issue is how would Pants know to reevaluate the target graph when that file changes?
That is done via sources and dependencies
If you just change the content of
VERSION
, the BUILD file hasn't changed, so Pants naively wouldn't do anything
There is logic to reevaluate when sources of (transitive) dependencies change
So we probably want to leverage that logic here
f
If you just change the content of
VERSION
, the BUILD file hasn't changed, so Pants naively wouldn't do anything
This I do understand.. but my issue is actual more kindergarten at the moment. The error is that pants says
Copy code
MappingError: Failed to parse ./tools/containers/myspecial/BUILD:
ParseError: tools/containers/myspecial/BUILD:12: Name 'read_file' is not defined.
h
Yes, that makes sense
f
I was hoping to have a basic example of a plugin that just returns `fooba`r I presume either I have stuffed registration up.. or that pants won't link the read_ .. as it needs it in the Dependency tree
h
Look at how the python backend registers the special symbol "python_artifact" to see how that is done
In src/python/pants/backend/python/register.py
👍 1
under "BuildFileAliases"
f
_`build_file_aliases()` is part of the API is it ?_ (I think that is what I was looking for )
h
yes, that is how you register special BUILD file symbols like this
f
I am slowing getting there . I managed to register the build_alias
read_file_contents()
Copy code
resource(
    name = "version",
    source = "VERSION",
)


docker_image(
  name="my-complex-app",
  dependencies=[
    "lib/my-simple-lib:simple_lib_wheel",
  ],
  image_tags=[
       "abc",
       "123",
       read_file_contents(sources=[":version"]),
  ]
)
read_file_contents - is a
str
Copy code
@dataclass(frozen=True)
class ReadFileContents(str):
    """read_file_contents() string provider"""

    alias = "read_file_contents"
    help = help_text(
        """
        A string read from a file.
        """
    )
    regex: Optional[str] = None
    static: Optional[str] = None
    sources: Optional[ContentOfMultipleSourcesField] = None

    def __new__(cls, value: str = "dummy-value-from-new", **kwargs):
        print(f"new() {value=} {kwargs=}", end="")
        return super().__new__(cls, value)

    def __init__(self, value: str = "dummy-value-from-init", **kwargs):
        print(f"init {value=} {kwargs=}", end="")
        super().__init__()
        # self.regex = kwarg.get("regex")
        # self.static = kwarg.get("static")
        # self.sources = kwarg.get("sources")

    def __hash__(self):
        return super().__hash__()
However, I am then attempting what I think is a "rule" to intercept that
read_file_contents(sources=[":version"]),
so I can "overide" the value... but alas, the rule doesn't seem to run.
Copy code
❯ pants package  app::
15:39:44.87 [INFO] stdout: "new() value='dummy-value-from-new' kwargs={'sources': [':version']}"
15:39:44.87 [INFO] stdout: ""
15:39:44.87 [INFO] stdout: "init value='dummy-value-from-init' kwargs={'sources': [':version']}"
15:39:44.87 [INFO] stdout: ""
15:39:46.96 [INFO] Completed: Building docker image my-complex-app:abc +2 additional tags.
15:39:46.97 [INFO] Wrote dist/app.my-complex-app/my-complex-app.docker-info.json
Built docker images: 
  * my-complex-app:abc
  * my-complex-app:123
  * my-complex-app:dummy-value-from-new
Docker image ID: sha256:d4a58212b8d4e26ba6f53f22c2f0dbbc9050c6f81cfe981a2e32fd5a611ea405
(.venv)
Copy code
from pants.engine.fs import DigestContents, GlobMatchErrorBehavior, PathGlobs
from pants.engine.rules import Get, collect_rules, rule
from pants.engine.target import (
    COMMON_TARGET_FIELDS,
    Target
)
from pants.engine.unions import UnionRule
from pants.core.target_types import FileSourceField, StringField, MultipleSourcesField
from pants.util.strutil import help_text
from dataclasses import dataclass
from typing import Optional
import os

# class RegexFilterField(StringField):
#     alias = "regex"
#     required = False
#     help = help_text(
#         """
#         Regex pattern, used to extract the string from the file contents.
#         """
#     )

class ContentOfMultipleSourcesField(MultipleSourcesField):
    required = False
    help = help_text(
        """
        A list of files to read the contents from.
        All contents are concatenated together.
        """
    )


@dataclass(frozen=True)
class ReadFileContents(str):
    """read_file_contents() string provider"""

    alias = "read_file_contents"
    help = help_text(
        """
        A string read from a file.
        """
    )
    regex: Optional[str] = None
    static: Optional[str] = None
    sources: Optional[ContentOfMultipleSourcesField] = None

    def __new__(cls, value: str = "dummy-value-from-new", **kwargs):
        print(f"new() {value=} {kwargs=}", end="")
        return super().__new__(cls, value)

    def __init__(self, value: str = "dummy-value-from-init", **kwargs):
        print(f"init {value=} {kwargs=}", end="")
        super().__init__()
        # self.regex = kwarg.get("regex")
        # self.static = kwarg.get("static")
        # self.sources = kwarg.get("sources")

    def __hash__(self):
        return super().__hash__()


@dataclass(frozen=True)
class ReadFileContentsRequest:
    target: Target

    @classmethod
    def is_applicable(cls, _: Target) -> bool:
       return True


@rule
async def do_read_the_file(request: ReadFileContentsRequest) -> str:

    return ReadFileContents("winner")

def rules():
    return (
        do_read_the_file,
    )
I was kind of hoping this would work. if I registered a
from ReadFileContent --> str
image.png
Copy code
❯ pants package  app::
19:55:15.89 [ERROR] 1 Exception encountered:

Engine traceback:
  in `package` goal

InvalidTargetException: app/my-complex-app/BUILD:8: The 'image_tags' field in target app/my-complex-app:my-complex-app must be an iterable of strings (e.g. a list of strings), but was `['abc', '123', ReadFileContents(sources=[':version'], regex=None)]` with type `list`.

(.venv)
so registering a rule that does a conversion, pants does not pick that up.
h
No, that's not how BUILD file parsing works.
Have you looked at the example of custom
python_artifact
?
Now that I think of it, that's actually not too far from what you're trying to do
But it only works because there is already a rule that goes from
SetupKwargsRequest -> SetupKwargs
and you're hooking into that via a
union
. That rule is invoked when building a python_distribution.
But what you're trying to do is generic - you want some rule to kick in at BUILD file parsing time, not when a specific goal runs
So you do need to make some changes to BUILD file parsing itself
@fresh-continent-76371 If you're free to hop on this call tomorrow: https://pantsbuild.slack.com/archives/C0D7TNJHL/p1710089695896389 it might be worth discussing with the group, folks may have ideas
f
Yeah can do. I'm Sydney, Australia
Am I right in thinking that if there were a registered
StringRequest --> str
then this method would work
h
Not quite, some rule that processes BUILD files and produces their content would have to invoke that rule and do the actual substitution