https://pantsbuild.org/ logo
Powered by
#development
  • w

    witty-crayon-22786

    08/15/2016, 9:03 PM
    @billowy-beach-68498: it doesn't construct an AddressMapper... just declares the type
  • b

    billowy-beach-68498

    08/15/2016, 9:03 PM
    so type(address_mapper) is subclass of AddressMapper?
  • w

    witty-crayon-22786

    08/15/2016, 9:03 PM
    yea
  • w

    witty-crayon-22786

    08/15/2016, 9:03 PM
    isinstance(address_mapper, AddressMapper)
  • w

    witty-crayon-22786

    08/15/2016, 9:04 PM
    the point is to be explicit... we look for an exact type match when we go looking for tasks that can compute a thing
  • w

    witty-crayon-22786

    08/15/2016, 9:05 PM
    so if 
    address_mapper
    is actually a subtype of AddressMapper (let's say MyExtendedAddressMapper), if we just did 
    SelectLiteral(address_mapper, type(address_mapper))
    , tasks that declared a dependency on exactly an AddressMapper wouldn't find it
  • w

    witty-crayon-22786

    08/15/2016, 9:06 PM
    cc @billowy-beach-68498 : ^
  • b

    billowy-beach-68498

    08/15/2016, 9:07 PM
    ah very clear, i see. thanks!
  • r

    rough-minister-58256

    08/15/2016, 11:51 PM
    Copy code
    cold `./pants list ::`:
  tuple        9951708  +9942001
  set          7851229  +7845648
  StepRequest  3989446  +3989446
  Entry        2572224  +2572224
  dict         2305023  +2290572
  TaskNode     2287304  +2287304
  Return       1326623  +1326623
  Noop         1245601  +1245601
  list          902401   +890141
  Path          829217   +829216

warm `./pants list ::`:
  list            1344015   +441614
  dict            2698929   +393906
  PrimitiveField   456573   +228268
  set             7980165   +128934
  cell             109108    +54120
  Payload          104993    +52485
  tuple          10001832    +50124
  SourcesField     123210    +47967
  Scope             95369    +47680
  OrderedSet        94833    +47415

warm `./pants list ::`:
  list            1786018   +441936
  dict            3093212   +394283
  PrimitiveField   684841   +228268
  set             8109099   +128934
  cell             163228    +54120
  Payload          157478    +52485
  tuple          10051956    +50124
  SourcesField     171177    +47967
  Scope            143049    +47680
  OrderedSet       142248    +47415
  • r

    rough-minister-58256

    08/15/2016, 11:51 PM
    top 10 obj counts between daemon runs - w/ forced gc between them
  • r

    rough-minister-58256

    08/16/2016, 12:59 AM
    more detail w/ manual gc’ing and inverse deltas: https://gist.github.com/kwlzn/ec2ae705c9ff70aafb5b39d75a7ecec2
  • r

    rough-minister-58256

    08/16/2016, 1:01 AM
    (and no invalidation)
  • r

    rough-minister-58256

    08/16/2016, 1:06 AM
    seems a bit odd to me that we’re doubling the type counts of things like 
    PythonBinary
    run over run?
  • w

    witty-crayon-22786

    08/16/2016, 1:18 AM
    @rough-minister-58256: we re-instantiate all of the targets each time
  • w

    witty-crayon-22786

    08/16/2016, 1:18 AM
    they contain mutable stuff
  • w

    witty-crayon-22786

    08/16/2016, 1:19 AM
    so we're cloning out of the ProductGraph and into the BuildGraph
  • r

    rough-minister-58256

    08/16/2016, 1:19 AM
    yeah, thought so
  • w

    witty-crayon-22786

    08/16/2016, 1:19 AM
    (or, we can't trust that they don't contain mutable stuff, currently)
  • r

    rough-minister-58256

    08/16/2016, 1:19 AM
    why we’re hanging onto the old ones tho is puzzling
  • w

    witty-crayon-22786

    08/16/2016, 1:20 AM
    oh... are those live objects?
  • w

    witty-crayon-22786

    08/16/2016, 1:20 AM
    hm.
  • r

    rough-minister-58256

    08/16/2016, 1:20 AM
    yeah
  • w

    witty-crayon-22786

    08/16/2016, 1:21 AM
    unclear where those would be held, except in the BuildGraph
  • w

    witty-crayon-22786

    08/16/2016, 1:21 AM
    which gets discarded post-fork, presumably.
  • w

    witty-crayon-22786

    08/16/2016, 1:21 AM
    unless there is more memoization happening.
  • r

    rough-minister-58256

    08/16/2016, 1:22 AM
    yeh will have to do some ref walking
  • r

    rough-minister-58256

    08/16/2016, 1:23 AM
    so that seems like question #1 - next up is “why so many unreachable objects during gc”, then digging into the dict/list/tuple refs
  • r

    rough-minister-58256

    08/16/2016, 1:23 AM
    lmk if anyone has any other ideas on how/where to dig in here
  • r

    rough-minister-58256

    08/16/2016, 1:24 AM
    can re-run some of these analysis’ with the 
    Runnable
    RB patched in too
  • r

    rough-minister-58256

    08/16/2016, 1:24 AM
    assuming that might help - at least for the tracewise malloc view
1...140141142...906Latest