https://pantsbuild.org/ logo
Powered by
#general
  • w

    witty-crayon-22786

    09/21/2015, 7:05 PM
    but yea
  • b

    bored-art-40741

    09/21/2015, 7:05 PM
    My dev box isn't saturated at all here
  • w

    witty-crayon-22786

    09/21/2015, 7:05 PM
    not ruling it out.
  • b

    bored-art-40741

    09/21/2015, 7:06 PM
    Though let me do it with a local run too
  • b

    bored-art-40741

    09/21/2015, 7:06 PM
    Benjy has a point though, this feels nonlinear
  • b

    bored-art-40741

    09/21/2015, 7:06 PM
    Maybe it's product population?
  • b

    bored-art-40741

    09/21/2015, 7:06 PM
    Not sure why that would be different from a noop build though
  • w

    witty-crayon-22786

    09/21/2015, 7:07 PM
    yea... we already know that is slow (but not why it would be slower)
  • b

    bored-art-40741

    09/21/2015, 7:08 PM
    This is definitely not blocking on work done in the workers
  • w

    witty-crayon-22786

    09/21/2015, 7:10 PM
    zinc hitting the remote cache:
    Copy code
    remote:

       disk0       cpu     load average
38.98 1978 75.29  29 14 57  2.97 2.49 2.35
23.70 2044 47.32  13 12 75  2.97 2.49 2.35
25.09 1704 41.76  19 14 68  2.97 2.49 2.35
       disk0       cpu     load average
 KB/t tps  MB/s  us sy id   1m   5m   15m
 7.84 548  4.19  11 15 74  2.89 2.48 2.34
13.57 987 13.07  18 19 62  2.89 2.48 2.34
13.29 819 10.63  19 16 65  2.89 2.48 2.34
15.04 680  9.99  16 14 70  2.89 2.48 2.34
14.61 808 11.52  17 16 68  2.89 2.48 2.34
13.87 616  8.35  19 20 60  3.06 2.52 2.36
14.25 1086 15.11  19 19 61  3.06 2.52 2.36
17.31 1183 19.99  26 22 51  3.06 2.52 2.36
16.24 923 14.63  18 20 62  3.06 2.52 2.36
14.88 1597 23.22  35 31 34  3.06 2.52 2.36
15.89 1552 24.09  23 28 49  2.98 2.52 2.36
14.58 579  8.24  16 14 70  2.98 2.52 2.36
14.60 1495 21.32  27 30 43  2.98 2.52 2.36
15.40 1590 23.92  24 31 45  2.98 2.52 2.36
14.70 2042 29.32  40 38 21  2.98 2.52 2.36
11.09 2507 27.15  30 49 22  3.22 2.57 2.38
32.87 632 20.28  18 23 59  3.22 2.57 2.38
12.50 1854 22.63  37 36 26  3.22 2.57 2.38
20.71 916 18.52  14 20 66  3.22 2.57 2.38
15.32 1225 18.32  54 29 17  3.22 2.57 2.38
       disk0       cpu     load average
 KB/t tps  MB/s  us sy id   1m   5m   15m
14.74 883 12.71  21 19 60  3.84 2.71 2.43
17.43 937 15.94  41 28 31  3.84 2.71 2.43
16.86 901 14.84  28 23 49  3.84 2.71 2.43
16.10 754 11.85  21 19 60  3.84 2.71 2.43
11.90 1267 14.73  21 22 56  3.84 2.71 2.43
24.39 458 10.91  23 14 63  3.77 2.72 2.43
10.11 2076 20.50  46 48  6  3.77 2.72 2.43
16.77 2315 37.92  34 49 18  3.77 2.72 2.43
17.16 1590 26.64  33 34 33  3.77 2.72 2.43
18.38 1558 27.96  43 28 29  3.77 2.72 2.43
10.95 2319 24.81  30 54 16  3.87 2.76 2.44
21.47 1595 33.45  42 39 20  3.87 2.76 2.44
12.36 2031 24.51  40 49 10  3.87 2.76 2.44
19.91 1838 35.75  39 36 24  3.87 2.76 2.44
16.45 1838 29.53  49 38 14  3.87 2.76 2.44
21.06 1628 33.47  53 35 12  4.12 2.83 2.47
16.52 1395 22.51  57 31 11  4.12 2.83 2.47
15.58 1323 20.13  56 35  9  4.12 2.83 2.47
13.53 1399 18.48  47 31 22  4.12 2.83 2.47
 9.58 2079 19.45  29 50 21  4.12 2.83 2.47
       disk0       cpu     load average
 KB/t tps  MB/s  us sy id   1m   5m   15m
10.31 2156 21.69  50 47  3  3.95 2.81 2.47
13.82 2144 28.94  44 49  7  3.95 2.81 2.47
 9.15 2824 25.24  34 60  5  3.95 2.81 2.47
16.11 1966 30.93  41 50 10  3.95 2.81 2.47
 9.77 2101 20.04  45 50  6  3.95 2.81 2.47
14.79 2117 30.58  44 47  9  4.76 3.00 2.54
 9.42 2308 21.22  37 56  7  4.76 3.00 2.54
13.58 1967 26.09  37 52 11  4.76 3.00 2.54
 8.56 2463 20.58  28 42 30  4.76 3.00 2.54
 7.83 2132 16.29  22 34 44  4.76 3.00 2.54
 6.70 2017 13.20  24 34 42  4.62 3.00 2.54
62.24 1154 70.11  45 29 26  4.62 3.00 2.54
30.00 1004 29.42  18 22 60  4.62 3.00 2.54
19.16 1173 21.96  30 23 47  4.62 3.00 2.54
 8.91 2096 18.23  16 31 53  4.62 3.00 2.54
45.12 565 24.90  12 13 75  4.41 2.98 2.53
15.07  43  0.63   9  6 85  4.41 2.98 2.53
 4.00   2  0.01   8  7 86  4.41 2.98 2.53
16.53 212  3.42  14  8 77  4.41 2.98 2.53
21.45 439  9.20  10 10 80  4.41 2.98 2.53
       disk0       cpu     load average
 KB/t tps  MB/s  us sy id   1m   5m   15m
15.93 489  7.61  12 13 75  4.21 2.96 2.53
53.83 1923 101.11  18 27 55  4.21 2.96 2.53
80.34 1321 103.67  34 22 44  4.21 2.96 2.53
25.31 2258 55.82  15 22 62  4.21 2.96 2.53
15.46 1324 20.00  14 18 69  4.21 2.96 2.53
13.83 1120 15.13  15 24 60  4.12 2.96 2.53
17.25 1269 21.37  11 18 71  4.12 2.96 2.53
22.10 817 17.63  28 20 52  4.12 2.96 2.53
17.65 1050 18.10  50 30 21  4.12 2.96 2.53
22.11 976 21.08  30 24 45  4.12 2.96 2.53
21.85 1233 26.31  32 21 47  4.11 2.98 2.54
27.95 520 14.18  34 11 55  4.11 2.98 2.54
 5.92  25  0.14  43  9 49  4.11 2.98 2.54
  • w

    witty-crayon-22786

    09/21/2015, 7:10 PM
    zinc hitting the local cache:
    Copy code
    local:

       disk0       cpu     load average
 KB/t tps  MB/s  us sy id   1m   5m   15m
10.36 117  1.18  14  6 80  3.26 3.42 2.91
12.51 3262 39.86  46 53  1  3.26 3.42 2.91
13.94 2812 38.29  48 52  1  3.26 3.42 2.91
13.00 3164 40.18  45 54  1  3.26 3.42 2.91
11.15 3029 32.98  49 51  0  3.48 3.46 2.93
13.07 3452 44.06  42 55  2  3.48 3.46 2.93
13.23 2523 32.60  53 46  1  3.48 3.46 2.93
12.59 3195 39.29  42 56  1  3.48 3.46 2.93
16.29 2689 42.77  59 40  1  3.48 3.46 2.93
 9.61 3300 30.97  39 60  1  4.00 3.57 2.97
10.97 3107 33.29  41 58  1  4.00 3.57 2.97
14.69 2960 42.48  51 48  1  4.00 3.57 2.97
16.77 2790 45.70  49 50  1  4.00 3.57 2.97
17.75 2649 45.90  53 46  1  4.00 3.57 2.97
 9.94 3196 31.02  38 60  1  4.00 3.58 2.97
18.13 3081 54.55  49 50  1  4.00 3.58 2.97
14.79 2456 35.48  55 44  1  4.00 3.58 2.97
14.93 3138 45.73  40 58  2  4.00 3.58 2.97
17.14 2886 48.31  50 49  1  4.00 3.58 2.97
12.19 3171 37.74  38 60  2  5.04 3.80 3.06
       disk0       cpu     load average
 KB/t tps  MB/s  us sy id   1m   5m   15m
17.94 3182 55.75  46 52  1  5.04 3.80 3.06
13.65 2850 37.99  44 56  1  5.04 3.80 3.06
16.28 2581 41.03  57 42  1  5.04 3.80 3.06
18.87 2742 50.54  60 40  0  5.04 3.80 3.06
17.80 2127 36.97  58 42  0  5.12 3.84 3.07
13.08 2425 30.98  59 41  0  5.12 3.84 3.07
10.30 2732 27.49  53 47  0  5.12 3.84 3.07
 8.88 2807 24.33  43 56  0  5.12 3.84 3.07
21.27 2790 57.95  50 49  0  5.12 3.84 3.07
18.75 2411 44.15  60 39  1  5.43 3.92 3.11
17.77 3058 53.05  52 44  4  5.43 3.92 3.11
 9.84 2727 26.21  42 56  2  5.43 3.92 3.11
11.45 2746 30.70  43 57  0  5.43 3.92 3.11
14.86 3111 45.15  39 59  1  5.43 3.92 3.11
13.44 2765 36.28  47 53  0  5.47 3.96 3.12
14.29 2428 33.89  58 42  0  5.47 3.96 3.12
11.07 2914 31.51  47 52  1  5.47 3.96 3.12
10.74 2997 31.43  37 60  2  5.47 3.96 3.12
12.61 3212 39.56  38 60  1  5.47 3.96 3.12
 9.07 3305 29.27  37 62  1  5.60 4.01 3.15
       disk0       cpu     load average
 KB/t tps  MB/s  us sy id   1m   5m   15m
10.91 3328 35.46  34 62  4  5.60 4.01 3.15
10.77 3197 33.63  39 59  1  5.60 4.01 3.15
10.78 3353 35.30  33 66  1  5.60 4.01 3.15
11.90 3357 39.02  47 53  0  5.60 4.01 3.15
13.86 2841 38.46  43 56  2  6.03 4.12 3.19
21.20 2825 58.50  50 48  2  6.03 4.12 3.19
24.29 2874 68.18  49 49  1  6.03 4.12 3.19
20.45 3222 64.33  44 53  2  6.03 4.12 3.19
11.64 3067 34.88  39 59  2  6.03 4.12 3.19
27.91 2434 66.34  48 46  5  6.83 4.32 3.27
16.95 2562 42.41  52 45  2  6.83 4.32 3.27
12.48 2467 30.07  44 56  0  6.83 4.32 3.27
12.99 2434 30.87  48 50  2  6.83 4.32 3.27
16.49 2345 37.78  51 46  3  6.83 4.32 3.27
13.99 3077 42.05  49 49  2  6.68 4.33 3.28
17.05 2490 41.45  46 51  3  6.68 4.33 3.28
12.64 3344 41.26  36 60  4  6.68 4.33 3.28
16.31 2691 42.87  47 48  4  6.68 4.33 3.28
19.24 2456 46.15  41 51  9  6.68 4.33 3.28
15.16 2935 43.47  40 53  6  6.63 4.36 3.29
       disk0       cpu     load average
 KB/t tps  MB/s  us sy id   1m   5m   15m
18.75 2380 43.57  46 48  6  6.63 4.36 3.29
14.96 3193 46.65  45 53  2  6.63 4.36 3.29
18.44 2412 43.44  51 45  4  6.63 4.36 3.29
15.58 3196 48.63  40 55  5  6.63 4.36 3.29
21.36 2376 49.56  55 40  5  6.50 4.37 3.30
15.80 2837 43.77  40 56  4  6.50 4.37 3.30
17.47 2226 37.98  60 38  1  6.50 4.37 3.30
10.96 3548 37.97  33 64  3  6.50 4.37 3.30
 9.75 3485 33.20  31 67  2  6.50 4.37 3.30
21.38 2551 53.26  33 51 16  6.29 4.36 3.31
22.70 2486 55.12  50 42  8  6.29 4.36 3.31
13.53 2552 33.72  57 42  1  6.29 4.36 3.31
15.52 2779 42.12  41 53  6  6.29 4.36 3.31
11.91 3104 36.10  43 54  3  6.29 4.36 3.31
  • b

    bored-art-40741

    09/21/2015, 7:11 PM
    Think it's IO?
  • w

    witty-crayon-22786

    09/21/2015, 7:11 PM
    (don't have network in there, sorry)
  • w

    witty-crayon-22786

    09/21/2015, 7:11 PM
    well, it's completely saturated when we're hitting locally
  • b

    bored-art-40741

    09/21/2015, 7:13 PM
    This repros with no remote involved
  • b

    bored-art-40741

    09/21/2015, 7:13 PM
    We're not blocking on decompress
  • b

    bored-art-40741

    09/21/2015, 7:13 PM
    IO doesn't seem to be taxed on the machine
  • w

    witty-crayon-22786

    09/21/2015, 7:14 PM
    hm.
  • w

    witty-crayon-22786

    09/21/2015, 7:14 PM
    fishy.
  • b

    bored-art-40741

    09/21/2015, 7:15 PM
    I have a profiling script that will rip into this
  • b

    bored-art-40741

    09/21/2015, 7:15 PM
    one sec
  • h

    happy-kitchen-89482

    09/21/2015, 7:16 PM
    Definitely reproing this on a dev box.
  • h

    happy-kitchen-89482

    09/21/2015, 7:16 PM
    It seems too consistent for this to be a server performance issue.
  • b

    bored-art-40741

    09/21/2015, 7:16 PM
    Repros with local only
  • b

    bored-art-40741

    09/21/2015, 7:16 PM
    Server is ruled out
  • b

    bored-art-40741

    09/21/2015, 7:18 PM
    Since this is local only, profiling will almost certainly reveal it... unless it's in a subprocess
  • b

    bored-art-40741

    09/21/2015, 7:18 PM
    But if it was in a subprocess we'd probably see it actually maxing the cores
  • b

    bored-art-40741

    09/21/2015, 7:18 PM
    So in 20 minutes I'll have a much clearer answer
  • b

    bored-art-40741

    09/21/2015, 7:35 PM
    Nope, it's entirely in the cache check
  • b

    bored-art-40741

    09/21/2015, 7:35 PM
    JvmCompile.check_artifact_cache
1...404405406...2420Latest