https://pantsbuild.org/ logo
Join Slack
Powered by
`pex_binary` - seems to be rebuilding all the requ...
# general
a
pex_binary
- seems to be rebuilding all the requirements, even on a tiny code change (not involving requirements). resulting in long build times. What would be the way to go about it?
h
That should not be happening, but we'd need a lot more details (ideally an example repo and the commands you're running)
And whether this is desktop or CI, and if CI, what caching you have set up there.
a
I run this locally, on my macos. This is the target
Copy code
pex_binary(
     name="launcher",
     dependencies=[
         "//:version_file",
         "./data:stimuli_data",
     ],
     entry_point="launcher.py",
 )
And that happens after adding a few new lines to 
launcher.py
(goes on for 70s)
b
You could try splitting the dependencies and sources into separate PEX files, as outlined here: https://blog.pantsbuild.org/optimizing-python-docker-deploys-using-pants/ This is what we do and it enables the dependencies PEX to be cached if the required libraries didn’t change.
a
Thanks @better-van-82973 - we're already using this method for the dockers we build. However, in the case above i'm trying to run locally. I'm not sure how to transfer the same method to local runs.
Don't know if relevant, but I see this with 
-ldebug
. Looks like pants thinks something in options chaged, event through it did not..
Copy code
20:55:16.15 [DEBUG] specs are: Specs(includes=RawSpecs(description_of_origin='CLI arguments', address_literals=(AddressLiteralSpec(path_component='stimulus', target_component='launcher', generated_component=None, parameters=FrozenDict({})),), file_literals=(), file_globs=(), dir_literals=(), dir_globs=(), recursi
ve_globs=(), ancestor_globs=(), unmatched_glob_behavior=<GlobMatchErrorBehavior.error: 'error'>, filter_by_global_options=True, from_change_detection=False), ignores=RawSpecs(description_of_origin='CLI arguments', address_literals=(), file_literals=(), file_globs=(), dir_literals=(), dir_globs=(), recursive_globs
=(), ancestor_globs=(), unmatched_glob_behavior=<GlobMatchErrorBehavior.error: 'error'>, filter_by_global_options=False, from_change_detection=False))
20:55:16.15 [DEBUG] changed_options are: ChangedOptions(since=None, diffspec=None, dependents=<DependentsOption.NONE: 'none'>)
20:55:16.15 [DEBUG] Launching 1 roots (poll=false).
20:55:16.15 [DEBUG] Dependency SessionValues of Some("@rule(pants.engine.internals.options_parsing.parse_options())") changed.
b
I think the logging output is a bit misleading : when it says “building … requirements” that also includes packaging up your own code. That is, the pex file needs to be recreated with requirements and launcher.py. Depending on how you’re using it, splitting into two PEXes, one with include_requirements=False and the other with include_sources=False may help (as mentioned above). Additionally seeing if you can use layout="packed" can make a huge difference, by not having to zip up large requirements
Also, if you’re running purely locally, you may be able to use 
pants run path/to/launcher.py
which is even faster than the packed layout (it conceptually combines the best bits of both optimisations). This may not work for how you need to invoke it, though
a
Thanks @broad-processor-92400! That makes sense now. Re separate pexs - I'm familiar with this method for building dockers, but how would I use it locally? Let's say I have 
requirements.pex
, 
app.pex
. What would be the command line? Re using 
pants run
- possible in our case, but adds some complication to our dev environment. I'd prefer the separate pexs solution if I can make it work.
b
I think the invocation would be something like 
PEX_PATH=requirements.pex app.pex
. That is, setting the PEX_PATH environment variable to point to the requirements file https://pex.readthedocs.io/en/latest/api/vars.html (I’m on mobile at the moment so no direct link, and I can’t experiment) That’s pretty fiddly, so I’d recommend trying the 
packed
layout first, and only putting effort into PEX_PATH and/or 
pants run
if that doesn’t work (and the packed layout likely still makes the two-pex option faster too)
a
perfect, this works! Thank you
2 Views
Open in Slack
PreviousNext
Powered by