bitter-ability-32190
07/12/2022, 7:23 PMenough-analyst-54434
07/12/2022, 7:51 PM.deps/
dir.bitter-ability-32190
07/12/2022, 7:51 PMenough-analyst-54434
07/12/2022, 7:52 PM.deps/
has changed, you have a Pants problem, if not, you have a Pex problem.pex_binary
target depending on a python_distribution
? That'd do it.bitter-ability-32190
07/12/2022, 8:05 PM.pex
) has changed, how does it know to not try and re-create the image?docker images
• Add a comment to a first-party source
• re-build image
• re-run docker images
I see two <none>
images being build each time in addition to the tagged imageenough-analyst-54434
07/12/2022, 8:55 PMbitter-ability-32190
07/12/2022, 8:56 PMCOPY
then be uncached, because the PEX is changing?enough-analyst-54434
07/12/2022, 8:58 PMas
which just copies that portion of the input PEX, not the whole thing.as
--from
pair that gets you this fancyness. This trick did not always exist and is relatively new (maybe 5 years old, but not always around).bitter-ability-32190
07/12/2022, 9:02 PMCOPY
named deps
not get invalidated with the new contents of my-app.pex
?enough-analyst-54434
07/12/2022, 9:04 PMbitter-ability-32190
07/12/2022, 9:06 PMCOPY --from=deps /my-app /my-app
is "fast" becuse thats fast, then?deps
and srcs
only to produce the same deps
image, so the work involved when doing the final thing is re-used.deps
changing is much less frequent than srcs
changingenough-analyst-54434
07/12/2022, 9:07 PMbitter-ability-32190
07/12/2022, 9:09 PMenough-analyst-54434
07/12/2022, 9:10 PMbitter-ability-32190
07/12/2022, 9:12 PMCOPY
instructions higher upenough-analyst-54434
07/12/2022, 9:24 PMbitter-ability-32190
07/12/2022, 9:25 PMenough-analyst-54434
07/12/2022, 9:26 PMbitter-ability-32190
07/12/2022, 9:29 PM--compile
experimental_shell_command(
name="unpacked_deps",
command=f"PEX_TOOLS=1 python3.8 {'/'.join(('..' for _ in build_file_dir().parts))}/{'.'.join(build_file_dir().parts)}/binary.pex venv --scope=deps --compile app-deps",
dependencies=[":binary"],
outputs=["app-deps/lib"],
tools=["python3.8"],
)
enough-analyst-54434
07/13/2022, 1:57 AM--layout {loose,packed,zipapp}
option in pex_binary
you could always just configure loose and COPY loose.pex/.deps
for the deps
layer and the rest (loose.pex/{.bootstrap,__main__.py,PEX-INFO,<1st party>}
) for the src
layer.bitter-ability-32190
07/13/2022, 10:22 AMvenv
mode? Ideally the last image layer (I'm learning!) has the code prepped for immediate execution (while not having the files duplicated in the COPY
destination and the final destination).
I guess too, I don't care about what mode it runs in specifically just that it has near-native startup time (which venv
mode promises)experimental_shell_command
to use --scope=all
and then use multiple COPY
instructions for the relevant slices of the venvenough-analyst-54434
07/13/2022, 2:23 PM--layout
is orthogonal to the runtime execution mode; so if the loose
PEX is --venv
, when you run the loose PEX __main__.py
it will bootstrap a venv under PEX_ROOT 1st run.bitter-ability-32190
07/13/2022, 2:24 PMCOPY
isn't exclude-friendly. Ideally I have COPY path/to/lib/<not my 1stparty>
then COPY path/to/lib/<1stparty>
enough-analyst-54434
07/13/2022, 2:24 PMbitter-ability-32190
07/13/2022, 2:24 PMenough-analyst-54434
07/13/2022, 2:25 PMbitter-ability-32190
07/13/2022, 2:25 PMError expanding output globs: Failed to read link "/tmp/process-executionhCEYrB/mcd/techlabs/projects/aidt/asr_service/app-srcs/bin/python3.8": Absolute symlink: "/usr/bin/python3.8"
I think @witty-crayon-22786 fixed this, but it isn't in 2.12.enough-analyst-54434
07/13/2022, 2:55 PMbitter-ability-32190
07/13/2022, 2:58 PMenough-analyst-54434
07/13/2022, 3:00 PMbitter-ability-32190
07/13/2022, 3:49 PMCOPY
from the deps
stage still ran.
Admittedly the COPY
took 4 seconds, so honestly it doesn't make a difference timing-wise. But it is a datapointenough-analyst-54434
07/13/2022, 3:52 PMbitter-ability-32190
07/13/2022, 3:56 PM--scope=deps
took ~40 seconds with layout=zipapp
, but took ~150 seconds with layout=loose
.layout=packed
seems to match layout=zipapp
, I think if I only copy the PEX-specific bits and use layout=packed
we might be on to somethingFROM ... as deps
COPY my/binary.pex/__main__.py /bin/app.pex/__main__.py
COPY my/binary.pex/.bootstrap /bin/app.pex/.bootstrap
COPY my/binary.pex/PEX-INFO /bin/app.pex/PEX-INFO
COPY my/binary.pex/.deps /bin/app.pex/.deps
RUN PEX_TOOLS=1 python3.8 /bin/app.pex venv --scope=deps --compile /bin/app
plus
FROM ... as srcs
COPY my/binary.pex /bin/app.pex
RUN PEX_TOOLS=1 python3.8 /bin/app.pex venv --scope=srcs --compile /bin/app
if I edit a firstparty file inside the PEX and re-run it completely skips the deps
stage.deps
or the other deps
stage inputs change by editing a 1stparty file with meaningless (like a comment) changesdeps
stage running. Oddly it doesnt run the COPY
but does run the RUN
🤔enough-analyst-54434
07/13/2022, 4:13 PMbitter-ability-32190
07/13/2022, 4:14 PM"code_hash": "8c7b5100b86874cd4d41b5994e7ca8061ecf5403",
RIPenough-analyst-54434
07/13/2022, 4:36 PM--scope deps
for this reason. It only adds it when you complete things in --scope srcs
.bitter-ability-32190
07/13/2022, 4:37 PMenough-analyst-54434
07/13/2022, 4:38 PMbitter-ability-32190
07/13/2022, 4:39 PMenough-analyst-54434
07/13/2022, 4:51 PM~/pex/user_code/<hash>/
.bitter-ability-32190
07/13/2022, 5:04 PMAssertionError: Expected code_hash to be populated for Spread PEX directory /bin/app.pex.
FOr compiling deps 😞enough-analyst-54434
07/13/2022, 5:12 PMbitter-ability-32190
07/13/2022, 5:14 PMPEX-INFO
not staying the same after just touching 1stpartyPEX_INFO
myself. SO maybe no request, just highlighting a potential improvement