https://pantsbuild.org/ logo
Powered by
#development
  • r

    rough-minister-58256

    08/18/2016, 10:51 PM
    if the daemon’s memory space still has un-gc’d objects
  • r

    rough-minister-58256

    08/18/2016, 10:51 PM
    and it forks with that, those pollute the memory space of the forked runner
  • r

    rough-minister-58256

    08/18/2016, 10:52 PM
    even tho it will never use them
  • w

    witty-crayon-22786

    08/18/2016, 10:53 PM
    turtles all the way down!
    Copy code
    >>> type(type(None))
<type 'type'>
    ... except it's not pickleable =(
  • r

    rough-minister-58256

    08/18/2016, 10:55 PM
    heh
  • w

    witty-crayon-22786

    08/18/2016, 10:57 PM
  • w

    witty-crayon-22786

    08/18/2016, 10:57 PM
    er, sorry, that was too much
  • w

    witty-crayon-22786

    08/18/2016, 10:57 PM
    it's just that NoneType isn't pickleable.
  • b

    billowy-beach-68498

    08/18/2016, 10:57 PM
    @rough-minister-58256: ok got it. thanks!
  • w

    witty-crayon-22786

    08/18/2016, 10:57 PM
    (guessing that, ironically, 
    <type 'type'>
    is pickleable)
  • r

    rough-minister-58256

    08/18/2016, 10:58 PM
    hah
  • w

    witty-crayon-22786

    08/18/2016, 10:58 PM
    @fancy-queen-20734 : do we actually use the support for optional Select anywhere? iirc we don't
  • w

    witty-crayon-22786

    08/18/2016, 11:00 PM
    ...nope!
  • w

    witty-crayon-22786

    08/18/2016, 11:00 PM
    going to nuke that.
  • r

    rough-minister-58256

    08/18/2016, 11:06 PM
    seems like the LBG deferral definitely improves the run over run memory growth. appears like our average “compressed memory size” is way down too (rss@7G, compressed@300MB idle; ~2GB active)
  • r

    rough-minister-58256

    08/18/2016, 11:06 PM
    looking much better
  • r

    rough-minister-58256

    08/18/2016, 11:07 PM
    here’s what a post-daemon run delta pre and post gc looks like now:
  • r

    rough-minister-58256

    08/18/2016, 11:07 PM
    Copy code
    pre gc:
tuple                 9903507     +1876
list                   461999     +1775
dict                  1911954     +1517
set                   7722698      +403
ScopeInfo                1204      +402
Parser                   1203      +402
JvmTool                   370       +74
OptionHistoryRecord      5743       +73
OrderedDict             24411       +68
JarDependency            1063       +63
OptionHistory            3980       +24
ListValueComponent        276       +13
RelocateRule               56       +11
cell                      873        +4
SafeConfigParser            6        +2
_SingleFileConfig           6        +2
OptionTracker               3        +1
function                14936        +1
_ChainedConfig              3        +1
defaultdict                15        +1
OrderedSet                  4        +1
instancemethod            172        -1
OptionValueContainer       89        -1
RankedValue              1029        -5

found 6301 unreachable objects during gc

post gc:
function               14935        -1
OptionTracker              2        -1
_ChainedConfig             2        -1
defaultdict               14        -1
OrderedSet                 3        -1
SafeConfigParser           4        -2
_SingleFileConfig          4        -2
cell                     869        -4
ListValueComponent       245       -31
OrderedDict            24343       -68
OptionHistory           3807      -173
OptionHistoryRecord     5439      -304
ScopeInfo                804      -400
Parser                   803      -400
set                  7722297      -401
dict                 1910265     -1689
tuple                9901631     -1876
list                  460111     -1888
  • r

    rough-minister-58256

    08/18/2016, 11:08 PM
    so way cleaner deltas overall
  • w

    witty-crayon-22786

    08/18/2016, 11:08 PM
    not sure i understand the result... are you saying it looks stable run-over-run?
  • r

    rough-minister-58256

    08/18/2016, 11:09 PM
    yeah. pre gc shows the count of objects created during a synchronous pailgun run on the daemon-side (with gc disabled)
  • r

    rough-minister-58256

    08/18/2016, 11:09 PM
    then it runs a gc
  • r

    rough-minister-58256

    08/18/2016, 11:09 PM
    then shows the delta after gc
  • r

    rough-minister-58256

    08/18/2016, 11:10 PM
    so assuming a no-op list :: we’d expect this to be ~perfectly symmetrical
  • r

    rough-minister-58256

    08/18/2016, 11:11 PM
    if we weren’t massively leaking objects run over run - afaict seems we aren’t anymore
  • r

    rough-minister-58256

    08/18/2016, 11:11 PM
    we’re still using 7GB of memory tho
  • r

    rough-minister-58256

    08/18/2016, 11:12 PM
    appear to have some mild leakages but not nearly as bad as before
  • w

    witty-crayon-22786

    08/18/2016, 11:13 PM
    sonofa...
    Copy code
    Exception message: (u"Failed to pickle <type 'function'>: Can't pickle <type 'function'>: attribute lookup __builtin__.function failed", PicklingError("Can't pickle <type 'function'>: attribute lookup __builtin__.function failed",))
  • w

    witty-crayon-22786

    08/18/2016, 11:14 PM
    ...er, wait. i'm not doing that.
  • w

    witty-crayon-22786

    08/18/2016, 11:14 PM
    ...oh, yes I am. hm.
1...149150151...906Latest