https://pantsbuild.org/ logo
Powered by
#development
  • w

    witty-crayon-22786

    12/01/2016, 12:22 AM
    point being: in cases where there are no sources, v2 is around 
    1.75x
    slower than v1 for cold
  • w

    witty-crayon-22786

    12/01/2016, 12:22 AM
    v2 is eager in terms of glob expansion and fingerprinting (because we want to cache that part), so a larger difference for larger sets of sources could make sense in twitter's repo
  • w

    witty-crayon-22786

    12/01/2016, 12:24 AM
    so, possible that there are "userspace" optimizations available, where rather than changing the engine heavily, we improve how globs are expanded
  • w

    witty-crayon-22786

    12/01/2016, 12:24 AM
    ...or add an intrinsic, potentially.
  • r

    rough-minister-58256

    12/01/2016, 12:25 AM
    or parallelize, somehow?
  • w

    witty-crayon-22786

    12/01/2016, 12:33 AM
    yes, definitely. mentioned offline: we're spending about 20% of our time in filesystem operations (under 
    engine.Engine._run
    )
  • w

    witty-crayon-22786

    12/01/2016, 12:33 AM
    about 75% of the time looks super suspicious and orderedset related. looking now.
  • r

    rough-minister-58256

    12/01/2016, 12:43 AM
    so the t.c OrderedSet impl is lifted from: http://code.activestate.com/recipes/576694/
  • r

    rough-minister-58256

    12/01/2016, 12:43 AM
    wherein the comments, someone alludes to gc issues with the impl: 
    I experienced memory problems (objects were not garbage-collected) when using this recipe, but the one based on weakref worked better.
  • r

    rough-minister-58256

    12/01/2016, 12:44 AM
    wonder if this is also responsible for at least part of the memory woes
  • r

    rough-minister-58256

    12/01/2016, 12:45 AM
  • r

    rough-minister-58256

    12/01/2016, 12:45 AM
    t.c ftw
  • w

    witty-crayon-22786

    12/01/2016, 12:45 AM
    heh
  • w

    witty-crayon-22786

    12/01/2016, 12:46 AM
    currently distracted getting our from-source workflow going
  • w

    witty-crayon-22786

    12/01/2016, 12:56 AM
  • r

    rough-minister-58256

    12/01/2016, 12:59 AM
    sa-weet
  • r

    rough-minister-58256

    12/01/2016, 12:59 AM
    🚢
  • w

    witty-crayon-22786

    12/01/2016, 2:02 AM
    ok, there is absolutely fishiness in the merging of lists, which i'm going to be focusing on tomorrow
  • w

    witty-crayon-22786

    12/01/2016, 2:02 AM
    50% of time spent in merging lists/sets during transitive walks
  • r

    rough-minister-58256

    12/01/2016, 2:02 AM
    nice
  • w

    witty-crayon-22786

    12/01/2016, 2:03 AM
    (removing OrderedSet helped take that down from 60% to 50%, but it's still outrageous)
  • w

    witty-crayon-22786

    12/01/2016, 2:03 AM
    out for now though.
  • w

    witty-crayon-22786

    12/01/2016, 8:19 PM
    @fancy-queen-20734 : i think i have a deeper fix for https://github.com/pantsbuild/pants/pull/4107
  • w

    witty-crayon-22786

    12/01/2016, 8:20 PM
    the time is all due to merging of HydratedTargets
  • w

    witty-crayon-22786

    12/01/2016, 8:20 PM
    working on optimizing that
  • w

    witty-crayon-22786

    12/01/2016, 8:22 PM
    er: "time is all due", the time in 
    extern_store_list
    at least
  • w

    witty-crayon-22786

    12/01/2016, 9:13 PM
  • f

    fancy-queen-20734

    12/01/2016, 9:36 PM
    yeah. that makes sense. I’d played with some things around that but didn’t get much improvement with the things I’d tried
  • w

    witty-crayon-22786

    12/02/2016, 12:04 AM
    so, after the above it gets fuzzier.
  • w

    witty-crayon-22786

    12/02/2016, 12:04 AM
    one thing that is clear is that there is a lot of time (and memory) tied up in object management
1...196197198...906Latest