I have a custom target that is an alternative to t...
# plugins
f
I have a custom target that is an alternative to the built-in
files
target generator and have issues with the caching ๐Ÿงต
Copy code
from pants.engine.target import COMMON_TARGET_FIELDS, SingleSourceField, Target


class ProjectVersionSourceField(SingleSourceField):
    alias = "source"
    help = "Path to the file with the project version."
    default = "VERSION"
    required = False


class ProjectVersionTarget(Target):
    alias = "version_file"
    core_fields = (*COMMON_TARGET_FIELDS, ProjectVersionSourceField)
    help = "A project version target representing the VERSION file."
I also have a
python_test
target in addition to the instance of that custom target:
Copy code
version_file(
    name="main-project-version",
    source="VERSION",
)

python_tests(
    name="tests",
    dependencies=[
        ":main-project-version",
    ],
)
When I run
pants test tests/cli
, it runs fine, caching the results. I expect that after I modify the contents of the
VERSION
file used by that target, the test target should be considered changed and the
--changed-since
does suggest that this is the case since the dependencies have been declared correctly. However, caching mechanism is broken - no matter the changes made to the
VERSION
file, the test never actually runs, relying on the local cache. I think I may need to request some kind of rule for this target so that the digest of the files could be evaluated to invalidate the cache, but not sure what should it be? Replacing the custom target with the
file
built-in target makes caching works fine again upon subsequent changes to the
VERSION
file.
I suspect this may be something related to the
await Get(Digest, PathGlobs([f"{target.address.spec_path}**/*"]))
, but I am not sure what union rule should that be part of, if any? ๐Ÿ˜•
I have a repro to check out and modify the plugin code, if required https://github.com/AlexTereshenkov/cheeseshop-query/pull/41
c
Is your VERSION file actually part of the test run sandbox, when using your target?
if not, I suspect you need a codegen to translate the VERSION file from a
version_file
into a
file
file..
f
Is your VERSION file actually part of the test run sandbox, when using your target?
no, it is not - i.e. it is not copied into the sandbox. It only should invalidate the dependents as their behavior may change
if not, I suspect you need a codegen to translate the VERSION file from a
version_file
into a
file
file.. (edited)
oh I see. Is that something trivial I'd be able to see how we do for the built-in codegen or is it something esoteric?
c
you can look at this for a rather slim example of hooking into codegen: https://github.com/pantsbuild/pants/blob/main/src/python/pants/backend/docker/util_rules/dockerfile.py in this case, I kind of abuse it, as I codegen from and to the same type.. but you'd want to adjust the codegen to take your version file source field as input and produce a file (or resource?) source field type as output.
f
right, so to invalidate cache, a dependent target must be part of the sandbox?
c
yea, otherwise it won't "see" the change, and thus not become invalidated.
f
right, so it won't help to try add some logic to my custom target that would re-read the contents somehow and thus invalidate
c
hmm... what you need is for some hash value that goes into the rule to change...
f
just trying to see what would be easier - write the codegen to generate dummy file targets or cache invalidation rule ๐Ÿ˜•
c
I think the far easier option is to make the VERSION file show up in the sandbox... ๐Ÿ˜‰
โœ… 1
there're only a few things that goes into the test execution rule besides files and env vars that could invalidate your test run... (I'm not too familiar with the implementation of that, so I draw a blank what else there may be..) so I have no idea how to invalidate the cache with anything else than dumping a file in there.. ๐Ÿ˜„
f
that's cool, thank you! I'll take a look at your dockerfile example