https://pantsbuild.org/ logo
#general
Title
# general
a

ambitious-student-81104

03/15/2022, 4:55 PM
Hi @curved-television-6568! I have some questions regarding building and releasing (deplying) PEX targets. Currently how we do this is we copy the source code into our Dockerfile, and our CD system builds the Dockerfile, and deploys the PEX built inside. This is apparently very inefficient as there is no cache and it always rebuilds it from ground up even when there is no change affecting the PEX. @happy-kitchen-89482 described to me how this will be resolved:
the idea is that there would a build-time container, and the pex would be built in that, and then copied into the deployment image, which is what the
docker_image
target describes
Is this build-time container gonna have all the source code needed and be re-used to build whatever we need to deploy?
h

hundreds-father-404

03/15/2022, 4:56 PM
Hey Horus!
Currently how we do this is we copy the source code into our Dockerfile, and our CD system builds the Dockerfile, and deploys the PEX built inside.
How are you building the PEX currently? E.g. are you running Pants inside the Docker image
a

ambitious-student-81104

03/15/2022, 10:59 PM
yes
h

hundreds-father-404

03/15/2022, 11:00 PM
Cool. Yes, Pants will build the
pex_binary
for you beforehand, using the normal Pants cache etc. Then it will copy the file into the Docker image for you. Are you already using the
platforms
field with the
pex_binary
target?
h

happy-kitchen-89482

03/15/2022, 11:00 PM
But if this is running in CI, which is a linux platform, then building locally and natively is fine, and https://github.com/pantsbuild/pants/issues/13682 is not critical
โž• 2
@ambitious-student-81104 in our other conversation I was thinking of what happens when running this on macOS, but if this is about CI then the existing functionality should just work.
โœ… 1
c

curved-television-6568

03/16/2022, 7:22 AM
Hi @ambitious-student-81104! ๐Ÿ™‚ In addition to what Eric and Benjy have already said, I think that in the mean time (until we have proper support for leveraging Docker for running arbitrary tasks) you can volume mount relevant cache directories into your build-time container in order to gain some efficiency back.
โœ… 1
Another thing to consider is the new PEX support for splitting the 3rdparty dependencies from the 1stparty application code when deploying the Pex to the production image, to reduce the number of layers being rebuilt in your image. https://pex.readthedocs.io/en/v2.1.72/recipes.html#pex-app-in-a-container
h

happy-kitchen-89482

03/16/2022, 3:21 PM
To clarify, since this is about running in CI, which is linux, then having a
docker_image
target that depends on a
pex_binary
target should work today: When you build the
docker_image
target Pants will first build the pex natively (which is fine, since that is still linux), and then embed it in the Docker image.
๐Ÿ‘ 1
a

ambitious-student-81104

03/16/2022, 4:17 PM
you can volume mount relevant cache directories into your build-time container in order to gain some efficiency back.
not sure how this will work. ci box starts afresh for each run, no? are you suggesting external storage?
c

curved-television-6568

03/16/2022, 6:34 PM
> you can volume mount relevant cache directories into your build-time container in order to gain some efficiency back.
not sure how this will work. ci box starts afresh for each run, no? are you suggesting external storage?
ah, right. yes in that case thereโ€™d need to be some remote syncing of caches during startup https://www.pantsbuild.org/docs/using-pants-in-ci#directories-to-cache
2 Views