fresh-continent-76371
03/09/2024, 10:22 AMread_file_contents(<path/in/repo>)
plugin or target extension .. Such that this is possible.
scala_artifact(
group="org.typelevel",
artifact="cats-core",
version=read_file_contents("VERSION"),
packages=["cats.**"],
)
docker_image(
name="demo",
repository="example/demo",
image_tags=[read_file_contents("VERSION"), "example"]
)
python_distribution(
name="mydist",
dependencies=[
# Dependencies on code to be packaged into the distribution.
],
provides=python_artifact(
name="mydist",
version=read_file_contents("VERSION"),
),
wheel_config_settings={"--global-option": ["--python-tag", "py37.py38.py39"]},
)
what type of plugin, or target am i needing to create - and is there something similar - parametrize
almost feels similar, but it is generating new targets (replacing the original I assume)fresh-continent-76371
03/09/2024, 10:38 AMfresh-continent-76371
03/09/2024, 11:44 AM@dataclass(frozen=True)
class FileReadRequest(ABC):
@classmethod
def is_applicable(cls, _: Target) -> bool:
return True
@rule
async def read_file(request: FileReadRequest) -> StringField:
original_kwargs = request.explicit_kwargs.copy()
build_file_path = request.target.address.spec_path
version_file = original_kwargs.pop("filename", None)
return StringField(value=version_file)
(this is just returning a "fake" filename - as a new tag to docker )I have the file reading code, as we use that for the SetupKwargs) ..
# register.py
from myplugin.general import file_reading
def rules():
return [file_reading.read_file]
docker_image(
name="myspecial",
repository="myspecial",
dependencies= [
...
],
tags=[
"cicd",
read_file(filename="extra_tag"),
]
)
but that says
❯ pants package tools/containers/myspecial::
22:37:19.13 [ERROR] 1 Exception encountered:
Engine traceback:
in `package` goal
MappingError: Failed to parse ./tools/containers/myspecial/BUILD:
ParseError: tools/containers/myspecial/BUILD:12: Name 'read_file' is not defined.
fresh-continent-76371
03/09/2024, 11:49 AMclass FileReadingField(StringField):
alias = "read_file"
value: Optional[str]
default = None
help = "Version from a fil"
@classmethod
def compute_value(
and use compute_value
but
• how do i register this new Field ?
• i will need to run the async read file approach, but can that be done within a classmethod ?
async def read_file_from_repo(file_path: str) -> str:
"""returns the file contents"""
digest_contents = await Get(
DigestContents,
PathGlobs(
[file_path],
description_of_origin="`read_file()` plugin",
glob_match_error_behavior=GlobMatchErrorBehavior.error,
),
)
c = digest_contents[0].content.decode()
return c
fresh-continent-76371
03/09/2024, 6:48 PMenv(...)
I did find it, but it doesn't look obvious as to how it is registering itself .. need to look deepersquare-psychiatrist-19087
03/09/2024, 9:08 PMVERSION="0.0.1"
shell_command(command=f"echo ${VERSION} > VERSION", output_files=["VERSION"])
fresh-continent-76371
03/09/2024, 9:15 PMfresh-continent-76371
03/09/2024, 9:17 PMVERSION
and CHANGELOG.md
files, (using convco
most likely, as a library under the hood)
but my first step is (1)fresh-continent-76371
03/09/2024, 9:20 PM:bump_version()
this looks like ..
# BUILD
version_and_changelog()
poetry_requirements(
name = "reqs",
...
that adds
:bump_version
:gen_changelog
which the CI/CD pipeline runs, intelligently, for "paths" that only had a changehappy-kitchen-89482
03/09/2024, 10:44 PMhappy-kitchen-89482
03/09/2024, 10:46 PMfiles()
target, and instead of version=read_file_contents()
you'll need some rule that acts on that file targethappy-kitchen-89482
03/09/2024, 10:47 PMhappy-kitchen-89482
03/09/2024, 10:48 PMread_file_contents()
and sucks in the content of a files() targethappy-kitchen-89482
03/09/2024, 10:48 PMfresh-continent-76371
03/09/2024, 10:48 PMhappy-kitchen-89482
03/09/2024, 10:49 PMsources
of a files()
targetfresh-continent-76371
03/09/2024, 10:49 PMhappy-kitchen-89482
03/09/2024, 10:49 PMfresh-continent-76371
03/09/2024, 10:49 PMfresh-continent-76371
03/09/2024, 10:50 PMYou're looking to modify BUILD file parsing itself, it sounds like
o yes I think so
fresh-continent-76371
03/09/2024, 10:53 PMread_file_contents(filename=.., [filenames=..], [regex=..])
Which returns a str/StrFieldhappy-kitchen-89482
03/09/2024, 10:54 PMfile(source="VERSION")
scala_artifact(
group="org.typelevel",
artifact="cats-core",
version=read_file_contents(":version"),
packages=["cats.**"],
)
(we can maybe macro this up so that you don't need the explicit file()
target later)
Where read_file_content()
effectively creates a dependency on the file target. Then invalidation will work.happy-kitchen-89482
03/09/2024, 10:55 PMscala_artifact
doesn't take dependencies today, so that would have to changehappy-kitchen-89482
03/09/2024, 10:55 PMfile()
target.happy-kitchen-89482
03/09/2024, 10:56 PMfresh-continent-76371
03/09/2024, 10:59 PMdocker_image(
name="myspecial",
repository="myspecial",
dependencies= [
...
],
tags=[
"cicd",
read_file_contents(filename="VERSION"),
]
)
happy-kitchen-89482
03/09/2024, 11:00 PMhappy-kitchen-89482
03/09/2024, 11:00 PMhappy-kitchen-89482
03/09/2024, 11:00 PMVERSION
, the BUILD file hasn't changed, so Pants naively wouldn't do anythinghappy-kitchen-89482
03/09/2024, 11:01 PMhappy-kitchen-89482
03/09/2024, 11:01 PMfresh-continent-76371
03/09/2024, 11:04 PMIf you just change the content of, the BUILD file hasn't changed, so Pants naively wouldn't do anythingVERSION
This I do understand.. but my issue is actual more kindergarten at the moment. The error is that pants says
MappingError: Failed to parse ./tools/containers/myspecial/BUILD:
ParseError: tools/containers/myspecial/BUILD:12: Name 'read_file' is not defined.
happy-kitchen-89482
03/09/2024, 11:05 PMfresh-continent-76371
03/09/2024, 11:05 PMhappy-kitchen-89482
03/09/2024, 11:06 PMhappy-kitchen-89482
03/09/2024, 11:06 PMhappy-kitchen-89482
03/09/2024, 11:06 PMfresh-continent-76371
03/09/2024, 11:13 PMhappy-kitchen-89482
03/10/2024, 12:44 AMfresh-continent-76371
03/10/2024, 4:44 AMread_file_contents()
resource(
name = "version",
source = "VERSION",
)
docker_image(
name="my-complex-app",
dependencies=[
"lib/my-simple-lib:simple_lib_wheel",
],
image_tags=[
"abc",
"123",
read_file_contents(sources=[":version"]),
]
)
read_file_contents - is a str
@dataclass(frozen=True)
class ReadFileContents(str):
"""read_file_contents() string provider"""
alias = "read_file_contents"
help = help_text(
"""
A string read from a file.
"""
)
regex: Optional[str] = None
static: Optional[str] = None
sources: Optional[ContentOfMultipleSourcesField] = None
def __new__(cls, value: str = "dummy-value-from-new", **kwargs):
print(f"new() {value=} {kwargs=}", end="")
return super().__new__(cls, value)
def __init__(self, value: str = "dummy-value-from-init", **kwargs):
print(f"init {value=} {kwargs=}", end="")
super().__init__()
# self.regex = kwarg.get("regex")
# self.static = kwarg.get("static")
# self.sources = kwarg.get("sources")
def __hash__(self):
return super().__hash__()
fresh-continent-76371
03/10/2024, 4:45 AMread_file_contents(sources=[":version"]),
so I can "overide" the value... but alas, the rule doesn't seem to run.fresh-continent-76371
03/10/2024, 4:46 AM❯ pants package app::
15:39:44.87 [INFO] stdout: "new() value='dummy-value-from-new' kwargs={'sources': [':version']}"
15:39:44.87 [INFO] stdout: ""
15:39:44.87 [INFO] stdout: "init value='dummy-value-from-init' kwargs={'sources': [':version']}"
15:39:44.87 [INFO] stdout: ""
15:39:46.96 [INFO] Completed: Building docker image my-complex-app:abc +2 additional tags.
15:39:46.97 [INFO] Wrote dist/app.my-complex-app/my-complex-app.docker-info.json
Built docker images:
* my-complex-app:abc
* my-complex-app:123
* my-complex-app:dummy-value-from-new
Docker image ID: sha256:d4a58212b8d4e26ba6f53f22c2f0dbbc9050c6f81cfe981a2e32fd5a611ea405
(.venv)
fresh-continent-76371
03/10/2024, 4:46 AMfrom pants.engine.fs import DigestContents, GlobMatchErrorBehavior, PathGlobs
from pants.engine.rules import Get, collect_rules, rule
from pants.engine.target import (
COMMON_TARGET_FIELDS,
Target
)
from pants.engine.unions import UnionRule
from pants.core.target_types import FileSourceField, StringField, MultipleSourcesField
from pants.util.strutil import help_text
from dataclasses import dataclass
from typing import Optional
import os
# class RegexFilterField(StringField):
# alias = "regex"
# required = False
# help = help_text(
# """
# Regex pattern, used to extract the string from the file contents.
# """
# )
class ContentOfMultipleSourcesField(MultipleSourcesField):
required = False
help = help_text(
"""
A list of files to read the contents from.
All contents are concatenated together.
"""
)
@dataclass(frozen=True)
class ReadFileContents(str):
"""read_file_contents() string provider"""
alias = "read_file_contents"
help = help_text(
"""
A string read from a file.
"""
)
regex: Optional[str] = None
static: Optional[str] = None
sources: Optional[ContentOfMultipleSourcesField] = None
def __new__(cls, value: str = "dummy-value-from-new", **kwargs):
print(f"new() {value=} {kwargs=}", end="")
return super().__new__(cls, value)
def __init__(self, value: str = "dummy-value-from-init", **kwargs):
print(f"init {value=} {kwargs=}", end="")
super().__init__()
# self.regex = kwarg.get("regex")
# self.static = kwarg.get("static")
# self.sources = kwarg.get("sources")
def __hash__(self):
return super().__hash__()
@dataclass(frozen=True)
class ReadFileContentsRequest:
target: Target
@classmethod
def is_applicable(cls, _: Target) -> bool:
return True
@rule
async def do_read_the_file(request: ReadFileContentsRequest) -> str:
return ReadFileContents("winner")
def rules():
return (
do_read_the_file,
)
fresh-continent-76371
03/10/2024, 4:47 AMfresh-continent-76371
03/10/2024, 8:57 AMfrom ReadFileContent --> str
fresh-continent-76371
03/10/2024, 8:57 AMfresh-continent-76371
03/10/2024, 8:58 AM❯ pants package app::
19:55:15.89 [ERROR] 1 Exception encountered:
Engine traceback:
in `package` goal
InvalidTargetException: app/my-complex-app/BUILD:8: The 'image_tags' field in target app/my-complex-app:my-complex-app must be an iterable of strings (e.g. a list of strings), but was `['abc', '123', ReadFileContents(sources=[':version'], regex=None)]` with type `list`.
(.venv)
fresh-continent-76371
03/10/2024, 8:58 AMhappy-kitchen-89482
03/10/2024, 4:44 PMhappy-kitchen-89482
03/10/2024, 4:46 PMpython_artifact
?happy-kitchen-89482
03/10/2024, 4:46 PMhappy-kitchen-89482
03/10/2024, 4:48 PMSetupKwargsRequest -> SetupKwargs
and you're hooking into that via a union
. That rule is invoked when building a python_distribution.happy-kitchen-89482
03/10/2024, 4:49 PMhappy-kitchen-89482
03/10/2024, 4:49 PMhappy-kitchen-89482
03/10/2024, 4:59 PMfresh-continent-76371
03/10/2024, 5:11 PMfresh-continent-76371
03/10/2024, 5:15 PMStringRequest --> str
then this method would workhappy-kitchen-89482
03/10/2024, 5:20 PM