pantsbuild #general

Is there a way to figure out where time is spent in a build? My compile step when no work is needed is slow (about 31 seconds). Is there a way to figure out where that time is going?

straight-waiter-86090

02/16/2017, 3:42 PM

Copy code

10:40:19 00:13   [compile]
10:40:19 00:13     [compile-jvm-prep-command]
10:40:19 00:13       [jvm_prep_command]
10:40:19 00:13     [scalafmt]
10:40:19 00:13     [compile-prep-command]
10:40:19 00:13     [compile]
10:40:19 00:13     [zinc]
10:40:50 00:44     [jvm-dep-check]
10:40:50 00:44   [complete]

fast-megabyte-33579

02/16/2017, 3:43 PM

does your

./pants server

server show any output for that step?

straight-waiter-86090

02/16/2017, 3:48 PM

@User uploaded a file:

Screen Shot 2017-02-16 at 10.47.40 AM.png▾

straight-waiter-86090

02/16/2017, 3:48 PM

not that I can see, if I’m looking at the right section.

fast-megabyte-33579

02/16/2017, 3:48 PM

yeah, bummer

straight-waiter-86090

02/16/2017, 3:49 PM

the self timings show time spent on compile and

messagegen

which is a custom task I have, also doing nothing in this case. both use rglobs to references all sources in those project directories.

straight-waiter-86090

02/16/2017, 3:50 PM

Copy code

[Self timings]
30.923	main:compile:zinc
11.212	main:gen:messagegen
0.917	main:gen:messagegen:execute
0.331	main:resolve:ivy
0.185	main:setup
0.064	main:jvm-platform-validate:jvm-platform-validate
0.028	main:gen
...

straight-waiter-86090

02/16/2017, 4:35 PM

if I run it with cProfiler it looks like

fileset.py:117(_do_rglob)

takes 27 cumulative seconds of the 39 total.

payload.py:109(_compute_fingerprint)

takes 22 cumulative seconds.

high-advantage-47550

02/16/2017, 5:28 PM

how do i enable the "strict ivy conflict manager”? is that the

scala-platform-strict-deps

flag?

rough-minister-58256

02/16/2017, 5:34 PM

@straight-waiter-86090

./pants -ldebug …

might be helpful

straight-waiter-86090

02/16/2017, 7:00 PM

I think part of the slowness in my case is I use

rglob(‘java/*.java’)

in most BUILD files. From what I can tell, rglob will traverse all directories, then match them against the pattern. If there are many other unrelated files (outside of

java/

) this is very slow. For example, if I run it in the root of my source tree, which has no

java/

directories at that level, it takes 1 minute 23 seconds and finds nothing.

witty-crayon-22786

02/16/2017, 7:08 PM

@high-advantage-47550 : in your ivy config

high-advantage-47550

02/16/2017, 8:51 PM

"""When True, limits compile time deps to those that are directly declared by a target.

high-advantage-47550

02/16/2017, 8:51 PM

thanks @witty-crayon-22786

witty-crayon-22786

02/16/2017, 8:55 PM

@straight-waiter-86090 : yea. we're very close to adding a daemon for filesystem operations.

witty-crayon-22786

02/16/2017, 9:23 PM

@limited-country-20626 : updated https://github.com/pantsbuild/pants/pull/4262 ... thanks!

witty-crayon-22786

02/16/2017, 9:25 PM

@straight-waiter-86090 : but, in general, big

rglobs

aren't a great idea anyway. more granular targets: http://www.pantsbuild.org/build_files.html#target-granularity

witty-crayon-22786

02/16/2017, 9:25 PM

(will help less with the amount of IO, and more with how much is invalidated per build)

witty-crayon-22786

02/16/2017, 9:26 PM

@straight-waiter-86090 : the daemon has a very nice strategy here though. so we would be interested in working with you to test it.

witty-crayon-22786

02/16/2017, 9:26 PM

it's currently in alpha, so before the end of the quarter we'll be moving to beta

straight-waiter-86090

02/16/2017, 9:26 PM

I can try testing it.

straight-waiter-86090

02/16/2017, 9:28 PM

what I just did was move my BUILD files down a level. instead of having them in

*/BUILD

they’re now in

*/java/BUILD

. this way the

rglobs

can be written as

rglobs(’*.java’)

instead of

rglobs(’java/*.java’)

straight-waiter-86090

02/16/2017, 9:29 PM

this cut the time from 45 seconds to 15 seconds.