hundreds-father-404
03/27/2019, 3:57 PM@rule
resolution appears to be causing a serious startup performance hit at import time. Any thoughts appreciated on https://github.com/pantsbuild/pants/issues/7442average-vr-56795
03/27/2019, 4:34 PMaloof-angle-91616
03/27/2019, 5:14 PMast
issuesnakeviz: error: the file /Users/dmcclanahan/tools/pants/whole.prof is not a valid profile
and/or ValueError: unknown record type in log file
from gprof2dot
is it immediately clear what i'm doing wrong? i tried switching python
on my path to point to python 3 but that didn't change the outputwitty-crayon-22786
03/27/2019, 6:05 PMhundreds-father-404
03/27/2019, 6:06 PMpip3 install tuna
rather than pip
witty-crayon-22786
03/27/2019, 6:06 PMvenvs
with the different pythons for snakevizaloof-angle-91616
03/27/2019, 6:06 PMwitty-crayon-22786
03/27/2019, 6:06 PMaloof-angle-91616
03/27/2019, 6:06 PMhundreds-father-404
03/27/2019, 6:07 PMaloof-angle-91616
03/27/2019, 6:07 PMpip3 install tuna
worked, pex was yelling about encodingswitty-crayon-22786
03/27/2019, 6:09 PMvirtualenv -p /usr/local/bin/python3 ~/venvs/snakeviz-py3
, then activate and pip install snakeviz
. then, without the venv activated, you can do ~/venvs/snakeviz-py3/bin/snakeviz $prof
source ~/venvs/snakeviz-py3/bin/activate
, and exiting the shell, respectively)aloof-angle-91616
03/27/2019, 6:13 PMenough-analyst-54434
03/27/2019, 6:15 PMaloof-angle-91616
03/27/2019, 6:15 PMenough-analyst-54434
03/27/2019, 6:15 PMaloof-angle-91616
03/27/2019, 6:16 PMast.parse()
is slower that would be more annoyingwitty-crayon-22786
03/27/2019, 6:18 PMaloof-angle-91616
03/27/2019, 6:18 PMyield Get(...)
check and we can prune the branches a lotinspect.getmodule
and other methods which i think means we can optimize this if need be (and is a great place to look for a py3 perf difference) -- i will post a profile after this lunchdiff --git a/src/python/pants/bin/pants_loader.py b/src/python/pants/bin/pants_loader.py
index 91a4ca1..7510e6e 100644
--- a/src/python/pants/bin/pants_loader.py
+++ b/src/python/pants/bin/pants_loader.py
@@ -66,7 +66,7 @@ class PantsLoader(object):
Fix it by setting the LC_* and LANG environment settings. Example:
LC_ALL=en_US.UTF-8
LANG=en_US.UTF-8
- Or, bypass it by setting the below environment variable.
+ Or, bypass it by setting the below environment variable.
{}=1
Note: we cannot guarantee consistent behavior with this bypass enabled.
""".format(encoding, cls.ENCODING_IGNORE_ENV_VAR)
@@ -101,7 +101,9 @@ class PantsLoader(object):
def load_and_execute(entrypoint):
assert ':' in entrypoint, 'ERROR: entrypoint must be of the form `module.path:callable`'
module_path, func_name = entrypoint.split(':', 1)
- module = importlib.import_module(module_path)
+ from pants.util.contextutil import maybe_profiled
+ with maybe_profiled('wow.prof'):
+ module = importlib.import_module(module_path)
entrypoint_main = getattr(module, func_name)
assert callable(entrypoint_main), 'ERROR: entrypoint `{}` is not callable'.format(entrypoint)
entrypoint_main()
inspect.stack()
might not be necessary@rule
decorator (2.06 seconds in my run)inspect.stack()
seems to take all the runtime, it calls tons of other things, but calling inspect.getsource()
from the decorator itself is only 0.87% (and we suspected that might be culprit). it might be cached by python since inspect.stack()
is the first inspect
method used in that decoratorenough-analyst-54434
03/27/2019, 9:41 PMaloof-angle-91616
03/27/2019, 9:41 PMenough-analyst-54434
03/27/2019, 9:41 PMaloof-angle-91616
03/27/2019, 9:42 PMinspect.stack()
shaved 3 seconds off and i cannot find where rule parsing contributes to import time in the profile i have set up of that import lineenough-analyst-54434
03/27/2019, 10:31 PMaloof-angle-91616
03/27/2019, 10:34 PMhundreds-father-404
03/27/2019, 10:35 PM./pants2 -V
?aloof-angle-91616
03/27/2019, 10:35 PMhundreds-father-404
03/27/2019, 10:36 PMaloof-angle-91616
03/27/2019, 10:50 PMinspect.stack()
is like very very slow in py3 like this is p clear at least with our access pattern (which wasn't too weird)hundreds-father-404
03/27/2019, 10:52 PMaloof-angle-91616
03/27/2019, 10:52 PMhundreds-father-404
03/27/2019, 10:58 PMenough-analyst-54434
03/27/2019, 10:58 PM$ for py in python{2.7,3.{4,5,6,7}}; do echo -n "$py: " && $py -c "import timeit; print(timeit.timeit('import inspect; inspect.stack()', number=10000))"; done
python2.7: 2.63740515709
python3.4: 3.8067204039980425
python3.5: 4.471806422996451
python3.6: 4.527370049007004
python3.7: 3.3335486440046225
witty-crayon-22786
03/27/2019, 10:59 PMenough-analyst-54434
03/27/2019, 10:59 PMwitty-crayon-22786
03/27/2019, 10:59 PMaloof-angle-91616
03/27/2019, 11:46 PM