https://pantsbuild.org/ logo
Join Slack
Powered by
Is there a way to speed up pex builds where the un...
# general
l
Is there a way to speed up pex builds where the underlying requirements aren't changing, vs just the source code? We have one pretty hefty dependency that's increasing the build times quite a bit when we're only making source code changes (
fiftyone
). Changing the layout to 
loose
gave us about a 2x speedup (this is just a CLI we expect users to run locally), but sources may change quite a bit over the course of development
b
Heya. For layout, we find 
layout="packed"
to be the fastest to build, because it doesn't require manipulating hundreds/thousands of small files. There's some other tricks too, give me a second...
Setting the 
--no-pre-install-wheels
PEX argument via https://www.pantsbuild.org/stable/reference/targets/pex_binary#extra_build_args can improve performance too. Install https://github.com/pex-tool/pex and run 
--help
to see more about what that does.
Two extra things that may work, but require more significant process change: • run a python source directly, e.g. 
pants run path/to/source.py
. This will separate the requirements and sources builds. • A manual version of the above: have separate pex_binarys, one with 
include_sources=True, include_requirements=False
and vice versa, and then run with something like 
PEX_PATH=path/to/requirements.pex path/to/sources.pex ...
https://docs.pex-tool.org/api/vars.html#PEX_PATH (I've not done this, so not 100% sure how to make it work)
w
> A manual version of the above: have separate pex_binarys I messed around a bit with this a while back - specifically with Docker builds, but the point likely holds across more than that if you can get your run command setup (maybe even a 
pants run
?) https://github.com/sureshjoshi/perfanity/tree/main The BUILD file of note would be something like: https://github.com/sureshjoshi/perfanity/blob/main/multipex/BUILD Again, noting that this is specifically for Docker, and not exactly what you're working on
h
This blog post has some info about separating source and requirements pexes
l
Switching to packed did the trick, thanks very much for the suggestion @broad-processor-92400!
Open in Slack
PreviousNext
Powered by