https://pantsbuild.org/ logo
#development
Title
# development
h

hundreds-father-404

04/06/2020, 6:03 PM
Hm,
./v2 fmt
is not terminating (or maybe just taking over a minute) in the
fmt
stage, meaning after all the formatters have run and write to the build root. It was also taking 100% of my CPU according to
top
. I didn’t notice this on Friday. Maybe the tokio upgrade has something to do with it?
cc @witty-crayon-22786
Also, when I ran
./v2 fmt src/python/pants/util/strutil.py
(no changes), it exited cleanly and quickly. So maybe the issue is when we modify the build root
w

witty-crayon-22786

04/06/2020, 6:03 PM
can bisect if you have a repro
h

hundreds-breakfast-49010

04/06/2020, 6:03 PM
hm I'm seeing this myself I think
h

hundreds-father-404

04/06/2020, 6:04 PM
I’ll bisect in a couple minutes and try to get the reproduction more precise
w

witty-crayon-22786

04/06/2020, 6:08 PM
py-spy shows it's spending time in materialize_directories... so yea, seems like it.
and since it's not more specific than that, can assume it is spending time in the rust code. so next step is to attach a native-code profiler. on linux that might be
perf
... on OSX, it's
Instruments
ok. so yea, definitely something inside
materialize_directories
, but the native profile doesn't indicate that any particular function in there is taking any longer... just possibly that it's looping more than it should be.
will bisect.
❤️ 2
h

hundreds-father-404

04/06/2020, 6:15 PM
Cool, thank you Stu!
w

witty-crayon-22786

04/06/2020, 6:32 PM
mm. yea, it looks like it was definitely github.com/pantsbuild/pants/pull/9467
will revert and report.
h

hundreds-father-404

04/06/2020, 6:34 PM
Huh. I wonder what caused it. Thank you for investigating and reverting
👍 1
they're pretty responsive. i'll report it upstream.
❤️ 1
h

hundreds-breakfast-49010

04/06/2020, 6:46 PM
I wonder why CI didn't catch this
h

hundreds-father-404

04/06/2020, 6:48 PM
I don’t know how CI would catch it. Things still work, they just take substantially longer. We already have major flakiness from timeouts, so we bias towards a really generous timeout, which means we wouldn’t catch it
w

witty-crayon-22786

04/06/2020, 6:49 PM
have you seen the run actually finish?
...oh. i guess you saw it finish for a smaller run?
h

hundreds-father-404

04/06/2020, 6:49 PM
Yes, when running on a single file. I got too impatient when running against 6 targets (—changed-since)
h

hundreds-breakfast-49010

04/06/2020, 6:50 PM
not locally. and I noticed that the linter job on https://github.com/pantsbuild/pants/pull/9310/checks?check_run_id=565041648 was failing when I'm pretty sure it had passed before, so I chalked it up to this issue
h

hundreds-father-404

04/06/2020, 6:51 PM
I don’t think this affects the lint goal because it never materializes directories. Locally, V2 lint is speedy for me
w

witty-crayon-22786

04/06/2020, 6:51 PM
yea, wouldn't affect lint.
h

hundreds-father-404

04/06/2020, 6:51 PM
That linter job never calls fmt, only lint
h

hundreds-breakfast-49010

04/06/2020, 6:51 PM
ah
w

witty-crayon-22786

04/06/2020, 6:52 PM
...oh. so that's why we wouldn't see it in CI.
there are tests of materialize directory though. hm.
h

hundreds-father-404

04/06/2020, 6:52 PM
Well, sort of. We do still run fmt in all the integration tests
w

witty-crayon-22786

04/06/2020, 6:52 PM
but yea, just slower.
👍 1
h

hundreds-father-404

04/06/2020, 7:47 PM
Huh.
./v2 test --debug
is really really slow to start the test in between the time of
--v2-ui
and when the output is supposed to render. I think this is the
InteractiveRunner
piece that took a perf hit
w

witty-crayon-22786

04/06/2020, 7:47 PM
possibly. use py-spy
h

hundreds-father-404

04/06/2020, 7:47 PM
./v2 test
behaved normally
w

witty-crayon-22786

04/06/2020, 7:48 PM
(InteractiveRunner uses
materialize_directory
, so it's possible. but easy to confirm with py-spy.)
👍 1
h

hundreds-father-404

04/06/2020, 8:43 PM
So it looks like
./v2 fmt
is actually overwriting the build root just as quickly as normal. The files are being updated properly. The issue is not terminating.
w

witty-crayon-22786

04/06/2020, 8:53 PM
ok. i wasn't able to get a simpler repro on https://github.com/pantsbuild/pants/issues/9476 , so I guess i'll need to resort to logging.
👍 1
e

early-needle-54791

04/06/2020, 9:06 PM
revert is ready to merge
💯 1
w

witty-crayon-22786

04/06/2020, 9:17 PM
thanks