https://pantsbuild.org/ logo
Powered by
#development
  • r

    rough-minister-58256

    03/22/2016, 9:51 PM
    like s/subject_key/subject, afaict
  • w

    witty-crayon-22786

    03/22/2016, 9:51 PM
    yea
  • w

    witty-crayon-22786

    03/22/2016, 9:51 PM
    that was premature.
  • f

    full-kilobyte-93429

    03/22/2016, 9:56 PM
    @witty-crayon-22786: i am doing your review now.. what do see the next step for this CA?
  • w

    witty-crayon-22786

    03/22/2016, 9:57 PM
    that's what i was saying above. the gist of https://github.com/pantsbuild/pants/issues/3066 is "we should only content address things we're going to cache or send out of process"
  • f

    full-kilobyte-93429

    03/22/2016, 9:57 PM
    yea, that's about right, will need that regardless
  • w

    witty-crayon-22786

    03/22/2016, 9:58 PM
    removing 
    Node.subject
    from content addressing seems like it makes sense though.
  • w

    witty-crayon-22786

    03/22/2016, 9:59 PM
    but there is probably a lot more... like lazily content addressing only when we're about to look something up in the Step cache, or change processes.
  • w

    witty-crayon-22786

    03/22/2016, 9:59 PM
    sorry... guess i got carried away with the initial change from 
    subject
    to 
    subject_key
    .
  • r

    rough-minister-58256

    03/22/2016, 10:01 PM
    yeah, having subject literals will also make things considerably easier to debug
  • r

    rough-minister-58256

    03/22/2016, 10:02 PM
    makes alot of sense to me
  • w

    witty-crayon-22786

    03/24/2016, 12:19 AM
    @full-kilobyte-93429: lmdb.MapFullError: mdb_txn_commit: MDB_MAP_FULL: Environment mapsize limit reached
  • w

    witty-crayon-22786

    03/24/2016, 12:19 AM
    a much, much larger max would be good
  • w

    witty-crayon-22786

    03/24/2016, 12:19 AM
    ...but also... hm
  • f

    full-kilobyte-93429

    03/24/2016, 12:19 AM
    256M isn't enough?
  • w

    witty-crayon-22786

    03/24/2016, 12:19 AM
    no.
  • w

    witty-crayon-22786

    03/24/2016, 12:19 AM
    but also, i wonder why i'm hitting it... that would mean 256MB for one run
  • w

    witty-crayon-22786

    03/24/2016, 12:20 AM
    lol
  • f

    full-kilobyte-93429

    03/24/2016, 12:20 AM
    last time i saw this was due to bug results more misses than necessary
  • f

    full-kilobyte-93429

    03/24/2016, 12:20 AM
    more likely a bug
  • f

    full-kilobyte-93429

    03/24/2016, 12:21 AM
    listing 
    3rdparty::
    is 6M
  • w

    witty-crayon-22786

    03/24/2016, 12:21 AM
    i'm listing all the things
  • w

    witty-crayon-22786

    03/24/2016, 12:22 AM
    or trying to
  • f

    full-kilobyte-93429

    03/24/2016, 12:22 AM
    in the first run? i guess
  • f

    full-kilobyte-93429

    03/24/2016, 12:44 AM
    one implication of s/subject_key/subject/ is subject needs to support ordering. now i fixed all tests saw improvement in both space (6.9M vs 6M) and time (30%-ish for cold cache and 10% for warmed up cache)
  • w

    witty-crayon-22786

    03/24/2016, 12:45 AM
    @full-kilobyte-93429: i don't think so... not if you content address for those cases
  • f

    full-kilobyte-93429

    03/24/2016, 6:31 AM
    one downside of CAing nodes in 
    _create_step
    means scheduler needs 
    storage
    again, the plus is is now we can get rid of that 
    Noop
    special case
  • w

    witty-crayon-22786

    03/24/2016, 5:47 PM
    yea, that would probably be good.
  • w

    witty-crayon-22786

    03/24/2016, 5:47 PM
    i don't think it's strictly true though... couldn't the engine apply the normalization?
  • f

    full-kilobyte-93429

    03/24/2016, 5:50 PM
    yea, should be since step_request is visible to engine
1...596061...906Latest