Yeah, this could be in the Pants docker support. The zip comparison should help isolate this quickly.

Is your

pex_binary

target depending on a

python_distribution

? That'd do it.

That's the one designed in path for 1st party to show up as 3rdparty I'm aware of.

I think I might also be failing to understand docker's caching. If the input file (the

.pex

) has changed, how does it know to not try and re-create the image?

Also, my test here is: • Build the image • run

docker images

• Add a comment to a first-party source • re-build image • re-run

docker images

I see two

<none>

images being build each time in addition to the tagged image

It skips through the layers until it hits a change.

So the deps portion COPY layer should not change if dpes don't change.

SO you should hit a skip for that layer.

Then the src layer - which is very purposefuly ordered to COPY after the deps layer, does produce a new layer.

So you rebuild, but only that 1 src layer at the end.

Its the

as

--from

pair that gets you this fancyness. This trick did not always exist and is relatively new (maybe 5 years old, but not always around).

Even if only the firstparty changed, the overall file changed, right? Sorry, I think I'm missing something obvious

You should see docker saying you get a cache hit on that deps layer even if slow.

It still "builds" the images for

deps

and

srcs

only to produce the same

deps

image, so the work involved when doing the final thing is re-used.

And it is ordered as such because

deps

changing is much less frequent than

srcs

changing

Maybe sharpen the pain / reason for questions. Are the steps slower than you'd expect?

I would expect 5

So, if you had deps, run1, run2, srcs as your layers, run1 and run2 would truly short circuit here.

Since a RUN layer just takes the hash of the RUN command string.

Unlike a COPY layer that takes the hash of the copied-in items.

Maybe that sharpens it up?

I need to move my

COPY

instructions higher up

Im hopping off work, but I think thatll bear fruit

I'm also seeing long build times, but I think thats comparitvely normal? My images are ~7GB. I'll have to see why

I also wonder what it'd look like to export 2 PEXes. One with only deps nd one with only sources. Then do the multi-stage build using their respoective PEXs

Basically an internal speed hack for a well known case or else a documented hack Pants allows you to set up if you know what you're doing.

And to loop this up unzipping the 2 images for the deps has the exact same layer hashes.

I think maybe our docker invocation should maybe set some setting so we dont see those images? I'll ping Andreas

Ah epiphany. We might be able to offshore this work using Pants itself. `experimental_shell_command`'s output as a dep, where the command does the

--compile

this is promising:

experimental_shell_command(
    name="unpacked_deps",
    command=f"PEX_TOOLS=1 python3.8 {'/'.join(('..' for _ in build_file_dir().parts))}/{'.'.join(build_file_dir().parts)}/binary.pex venv --scope=deps --compile app-deps",
    dependencies=[":binary"],
    outputs=["app-deps/lib"],
    tools=["python3.8"],
)

But honestly I dream of a solution where we don't build a pex just to immediately unpack it

OK so this does speed up build, doesn't produce extraneous images, and works like a charm. How does this play with play with

venv

mode? Ideally the last image layer (I'm learning!) has the code prepped for immediate execution (while not having the files duplicated in the

COPY

destination and the final destination). I guess too, I don't care about what mode it runs in specifically just that it has near-native startup time (which

venv

mode promises)

I suppose I can tweak my

experimental_shell_command

to use

--scope=all

and then use multiple

COPY

instructions for the relevant slices of the venv

Exactly! Do that instead.

Actually, no - you'd want two scopes still I'd think so the logic of what to copy stays in PEX.

So then it's just a bummer we still build an "intermediate" PEX in the sandbox, but honestly not doing that is really weird because we are packaging that PEX

These experimental shell commands are also a bit eyebrow-raising. But for now we're just experimenting. Making this cleaner is doable over time

Ah Pants getting in the way 😕

Error expanding output globs: Failed to read link "/tmp/process-executionhCEYrB/mcd/techlabs/projects/aidt/asr_service/app-srcs/bin/python3.8": Absolute symlink: "/usr/bin/python3.8"

I think @witty-crayon-22786 fixed this, but it isn't in 2.12.

We're kind of cheating here I suppose. Making the venv on the dev box, and then just copying it in to the docker image and assuming everything is kosher. Multi-stage unpacks in the dest image of choice. I'm gonna need to stew on this one

The beauty of the PEX unpack inside the container is this is not a worry.

That would allow no COPY and just RUN against the mounted in PEX - not sure if that works out faster in the end or not.

I had a great experience with podman / buildah / crun in ~2019. Haven't touched it since.

And I think your overlords are big backers of this project IIRC.

Ah no, Guiseppe is RedHat.

Where the Docker "context" - docker's terminology, is the dir tree rooted at the Dockerfile

So, that would totally explain the behavior.

This all makes more sense if you try to imagine how you would implement (caching layers in) a docker build yourself.

layout=packed

seems to match

layout=zipapp

, I think if I only copy the PEX-specific bits and use

layout=packed

we might be on to something

Yeah this gets the cache turned to 11 (using a packed PEX)

FROM ... as deps
COPY my/binary.pex/__main__.py /bin/app.pex/__main__.py
COPY my/binary.pex/.bootstrap /bin/app.pex/.bootstrap
COPY my/binary.pex/PEX-INFO /bin/app.pex/PEX-INFO
COPY my/binary.pex/.deps /bin/app.pex/.deps
RUN PEX_TOOLS=1 python3.8 /bin/app.pex venv --scope=deps --compile /bin/app

plus

FROM ... as srcs
COPY my/binary.pex /bin/app.pex
RUN PEX_TOOLS=1 python3.8 /bin/app.pex venv --scope=srcs --compile /bin/app

if I edit a firstparty file inside the PEX and re-run it completely skips the

deps

stage

So now I just need to verify none of

.deps

or the other

deps

stage inputs change by editing a 1stparty file with meaningless (like a comment) changes

Shucks... I see the

deps

stage running. Oddly it doesnt run the

COPY

but does run the

RUN

🤔

So because the PEX info file contains the code hash, we'll never get a true cache for deps stage

All of this is very involved, in-general. I think I'll boil it down to an in-repo plugin. In which case I can muck with

PEX_INFO

myself. SO maybe no request, just highlighting a potential improvement