https://pantsbuild.org/ logo
Join Slack
Powered by
I am seeing some performance issues with an integr...
# development
f
I am seeing some performance issues with an integration test of mine. The integration test invokes Pants via its Pex (and not via the 
pants.testutil.pants_integration_test
helpers to avoid a plugin resolution issue with there being a copy of 
packaging
on the test's 
PYTHONPATH
and 
pants.testutil.pants_integration_test
using the test's 
PYTHONPATH
even when 
hermetic=True
). 1. Any objection if I make some of the 
--python-*
options be read at bootstrap scheduler time so plugin resolution can use them? Plugin resolution does not appear to currently respect an updated 
--python-pip-version
nor an updated Pex version from the 
--pex-cli-*
options. As to why ... I am seeing a 25 second bootstrap time for plugin resolution in an integration test of mine for just an out-of-tree plugin available as a wheel locally. I'd like to see if newer Pex / Pip can reduce that resolution time. 2. I am also seeing plugin resolution occur twice in the same integration test to just get to a point where I can run the integration test. So the 25 second penalty is being paid twice. Unclear to me on first inspection other than maybe the creation of a new session here.
h
The bootstrapping sequence is pretty clunky due to the need to create a scheduler just powerful enough to resolve plugins, before anything else can proceed (including full options parsing, since that requires options registered by plugins)
It is unfortunate that the python backend has a special status within core pants, as it is used to resolve plguins
But it is what it is
So I guess we must either allow those options to be read (and then we may want to promote them to be global “bootstrap options”? or maybe we don’t have to?), or we create an alternative mechanism for plugin resolution via a vanilla venv and pip, or something
f
So I guess we must either allow those options to be read (and then we may want to promote them to be global “bootstrap options”? or maybe we don’t have to?), or we create an alternative mechanism for plugin resolution via a vanilla venv and pip, or something
What if Pants or the scie-pants just invokes the 
pip
in the Pants venv to install plugins?
h
Yeah, that would simplify the bootstrap sequence some. But then we wouldn’t be using the engine, so we’d have to have some invalidation story.
f
🐔 or 🥚
If the plugins change, should scie-pants just reinstall the Pants venv?
Maybe the engine needs a way to mark its venv as invalid and needing to be rebuilt?
Or if the code is in Pants itself, then a way of just re-executing itself to rebuild the plugin installation.
(the idea depends on what actually ends up putting the plugins into the venv)
There's also the issue with plugin resolution intentionally leaking paths from the PEX_ROOT and sourcing the dists from there.
I should probably document the current mess so we can competently talk about how and what to fix.
🙏 1
p
Or the plugins can go in a separate venv from pants itself with a 
.pth
file in the pants venv that injects the plugin venv. Or the other way around, a 
.pth
file in the plugin venv that injects the vanilla pants venv's paths. That should make rebuilding the custom plugins venv lighter weight.
f
❤️ 1
Thought: What if we just use 
PexCliProcess
directly (from a rule)? Instead of invoking the 
VenvPex
machinery for this bootstrap use case. That might allow us to have a "plugin resolution scheduler" with a much narrower scope (i.e., no need to load the Python backend there) as compared to the current "bootstrap scheduler."
Nor that matter load anything related to a normal Pants run. This would remove the current need for, for example, the 
BUILD
file parsing code to be present in the bootstrap scheduler (since referenced by the Python backend). And that code consumes 
BootstrapStatus
so it can avoid erroring during the bootstrap scheduler run. Simpler code if it does not need to know about 
BootstrapStatus
.
6 Views
Open in Slack
PreviousNext
Powered by