we need to do

2.7.1

final: oops.

are they any good patterns for “freezing” a Python class? context: I wrote code to construct a trie and, during the construction process, the nodes need to be mutable but in order to work with the engine they need to be immutable and hashable. I’m thinking just a “frozen” flag to prevent mutation. Would that be sufficient to use it with the engine?

for whom it may concern:

# python3.8:
>>> cast(dict[str, str], dict())
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: 'type' object is not subscriptable

# python3.9
>>> cast(dict[str, str], dict())
{}

@witty-crayon-22786 I'm working on the UX for

check

for Java and Go. 🧵

@ancient-vegetable-10556:

src/python/pants/jvm/goals/coursier_integration_test.py

is failing in CI for me: https://github.com/pantsbuild/pants/runs/4028069406?check_suite_focus=true#step:13:420. I rebased my branch cleanly on top of

main

which includes #13295. Any ideas on what is failing?

I’m looking at implementing https://github.com/pantsbuild/pants/issues/13360 🧵

I’ve run into a minor hickup, however.. as I’m generating the source file (a

Dockerfile

) for the

docker_image

target, the engine complains that I don’t have the required file present for the source field:

E             File "/private/var/folders/8j/c8jf_msj009947wyw82xvdkw0000gn/T/process-executioneZewVW/src/python/pants/engine/internals/graph.py", line 792, in hydrate_sources
E               sources_field.validate_resolved_files(snapshot.files)
E             File "/private/var/folders/8j/c8jf_msj009947wyw82xvdkw0000gn/T/process-executioneZewVW/src/python/pants/engine/target.py", line 1570, in validate_resolved_files
E               f"The {repr(self.alias)} field in target {self.address} must have "
E           pants.engine.target.InvalidFieldException: The 'source' field in target src/docker:docker must have 1 file, but it had 0 files.

However, it will be in the Digest.. any way to work around this? This is what the test looks like:

def test_synthetic_dockerfile(rule_runner: RuleRunner) -> None:
    rule_runner.write_files(
        {
            "src/docker/BUILD": dedent(
                """\
                docker_image(dependencies=[":dockerfile"])
                dockerfile(
                  name="dockerfile",
                  instructions=[
                    "FROM python:3.8",
                    "FROM alpine as interim",
                    "FROM interim",
                    "FROM scratch:1-1 as output",
                  ]
                )
                """
            ),
        }
    )

    assert_build_context(
        rule_runner,
        Address("src/docker"),
        expected_files=["src/docker/Dockerfile"],
        expected_version_context={
            "baseimage": {"tag": "3.8"},
            "stage0": {"tag": "3.8"},
            "interim": {"tag": "latest"},
            "stage2": {"tag": "latest"},
            "output": {"tag": "1-1"},
        },
    )

So there’s a codegen for the

dockerfile

target, which produces the

Dockerfile

which I want the

docker_image

to pick up..

Yeah, that's a Target API error. Set

expected_num_files = 0

. It's happening because

SingleSourceField

sets it to 1 as a "reasonable default" You can use

range(0, 2)

if 0-1 is allowed

Right, so I better run my own validation later against the digest explicitly then, because I do want the expect num files to be 1..

exactly. The `SourcesField.validate_resolved_files`(?) method only runs on the sources declared in the BUILD file. Not any generated sources

Yep, that worked, now I bomb on a later step. Thanks!

But also because you control the codegen rule, it's arguably not as important to validate. And you could do that validation via a test, rather than runtime assertion

well, it could be either or, so I do want to assert it at runtime.

That is, if you don’t use the

dockerfile

target, but use a file on disk.

In that case, you should have already gotten the validation via

expected_num_files

Oh! Are you reusing the same sources field for both the synthetic target and normal target? Instead, you can have the synthetic target subclass and override

expected_num_files

. Your rules still use

DockerSingleSourceField

, and they work just fine with the subclass

SyntheticDockerSingleSourceField

The Target API was designed that way precisely for situations like this

Uhm, maybe I should change how I’m trying to do this, if you look at my test again, notice that the source field of

docker_image

is the same, regardless of which variant of Dockerfile I choose to use… so I’d say it doesn’t work to do what you suggest in this case (which could be an indication that I should change how I try to do this…)

The expected num files for the source field of the synth target is 0 already..

My motivation was to break out the

dockerfile

spec part of the

docker_image

, that way it’ll be easier to support linting of the synthetic dockerfile (I assume) and also to run codegen export to inspect it, if desired, etc..

To try to clarify, I'm proposing this modeling:

class DockerSourceField(SingleSourceField):
   default = "Dockerfile"

class SyntheticDockerSourceField(DockerSourceField):
  default = None
   expected_num_files = 0
   alias = "_source"

Then your rules use

tgt.get(DockerSourceField)

and are agnostic to knowing

SyntheticDockerSourceField

exists, beyond needing to set

enable_codegen=True

when you're hydrating sources

Perhaps it’s just too late for me to get it… but I’ve pushed what I have (working) here: https://github.com/pantsbuild/pants/pull/13386/commits/6025b3e843b469accc1e22f53fb7493eab912bb9#diff-230e01d4da02868508b[…]cdb86ee8680b8d26R133-R137 It’s the next step, to wire this codegen target up with

docker_image

that proved problematic (but I see a way forward, albeit, perhaps not the most elegant yet..)

As what I have in that draft so far, I was hoping that simply use

enable_codegen=True

… aaahhhh, ok, now perhaps it finally dawned on me, ok, give me a minute to digest pun not intended for a while.. 🙂 (or rather, I should go to bed, and continue this at a later time…)

(mentally calculating number of hours between my time zone and yours) Yeah that latter one might be good. 🙂

Ah okay, you have two completely separate fields. That works too This looks good fwict! Is it not working? Yes to have to set

enable_codegen=True

and

for_sources_types=(DockerImageSourceField,)

Yeah, the codegen part works splendid. It’s when I try to pull the generated source in to the other target instead of having the source globbed that I ran into issues… but I’ll get back to this, I think my head isn’t working properly at this hour any more…

Also, I need to get up in ~4hrs for day 2 on our kick off here.. 😛

Thanks for iterating with me, though 🙂

Good luck with your kick off! ❤️

I assume

FrozenDict

always returns its elements in the same order when running

repr

on it?

Given that named_caches should be append-only, what is the philosophical opposition to allowing it to poke into folders outside of

~/.cache/pants

? As part of https://github.com/pantsbuild/pants/issues/13390, I think it'd be sweet if Pants could share the

GOPATH

(downloaded modules) already on your machine. Ditto pip

I'm very interested in https://github.com/pantsbuild/pex/projects/1 It looks like not much has happened with that recently. I'm guessing the focus shifted to the new PEX release, right? Or is there some other blocker for pex-based lockfiles? Are there plans to continue working on that soon, or are there still other things that come first?