https://pantsbuild.org/ logo
Powered by
#development
  • w

    witty-crayon-22786

    01/16/2018, 6:50 PM
    ...er, to be clear. the thread would be doing more than extending leases.
  • w

    witty-crayon-22786

    01/16/2018, 6:52 PM
    i think you need both A and B to some degree: 1) something must be extending leases (i think this is cleanest implemented as part of B, because it avoids aliasing issues where something is in the graph for such a short time that you never bump its lease), and 2) you need something (probably a thread, ie A) looking for expired leases
  • w

    witty-crayon-22786

    01/16/2018, 6:53 PM
    argh, yea. so the comment i linked to is relatively useless, sorry. see above.
  • a

    average-vr-56795

    01/16/2018, 6:53 PM
    I’m imagining that we don’t want to be proactively looking for expired leases, but rather trying to GC if we try to write and it’s full, and potentially periodically trimming a bit
  • a

    average-vr-56795

    01/16/2018, 6:54 PM
    I’m thinking “occasional GC” rather than “Trying to get as close to refcount+drop as possible”
  • w

    witty-crayon-22786

    01/16/2018, 6:54 PM
    because it's hard to do periodically, or because that's actually superior?
  • w

    witty-crayon-22786

    01/16/2018, 6:54 PM
    (with regard to the daemon/forking/locking issues: would want to expose GC as a method, and then have python call in under the fork_lock, i think)
  • a

    average-vr-56795

    01/16/2018, 6:55 PM
    Because it’s superior; if we’re doing a refcount+drop style, we’ll probably end up re-reading files relatively often, which seems wasteful
  • a

    average-vr-56795

    01/16/2018, 6:56 PM
    I’m imaging realistically we only want to GC if we’re causing a nuisance for users because we’re using a lot of space
  • w

    witty-crayon-22786

    01/16/2018, 6:56 PM
    ok
  • w

    witty-crayon-22786

    01/16/2018, 6:56 PM
    sure.
  • a

    average-vr-56795

    01/16/2018, 6:56 PM
    (Or if we’re unable to execute because the store is full)
  • w

    witty-crayon-22786

    01/16/2018, 6:56 PM
    and "store is full" is cheap to check?
  • a

    average-vr-56795

    01/16/2018, 6:56 PM
    “store is full” is detected by a write to it failing with a known error
  • w

    witty-crayon-22786

    01/16/2018, 6:56 PM
    a hybrid would be to check "how full is the store" periodically, and run.
  • w

    witty-crayon-22786

    01/16/2018, 6:57 PM
    so the method exposed out of the native code would be "maybe_gc", and the daemon would call it periodically outside of the hot path
  • a

    average-vr-56795

    01/16/2018, 6:57 PM
    I think “how full is the store” is relatively expensive to check; it’s certainly not a primitive exposed by lmdb
  • a

    average-vr-56795

    01/16/2018, 6:58 PM
    So would end up being a whole B+Tree traversal. (Or I guess a hacky call to 
    du
    )
  • w

    witty-crayon-22786

    01/16/2018, 6:59 PM
    i think that if it is potentially expensive, then the background thread is a requirement
  • a

    average-vr-56795

    01/16/2018, 6:59 PM
    Background thread doing what?
  • w

    witty-crayon-22786

    01/16/2018, 6:59 PM
    checking for expired leases above our space utilization
  • w

    witty-crayon-22786

    01/16/2018, 7:00 PM
    i think there are two components: 1) something in the read/write path extending leases, 2) something outside the read/write path checking for expired leases
  • a

    average-vr-56795

    01/16/2018, 7:01 PM
    Yes 🙂 Let’s discuss those separately, because they’re nicely orthogonal. Let’s start with 2, as it’s where we currently are 🙂
  • w

    witty-crayon-22786

    01/16/2018, 7:01 PM
    k
  • a

    average-vr-56795

    01/16/2018, 7:02 PM
    Let’s say the store currently has X GB of files, and a maximum quota of Y GB. We have A GB of leased files, and N unleased files which take up B GB.
  • w

    witty-crayon-22786

    01/16/2018, 7:02 PM
    (one more clarification: having any sort of size target seems like a bit of a stretch goal to me)
  • w

    witty-crayon-22786

    01/16/2018, 7:02 PM
    (compared to just faithfully executing expired leases)
  • a

    average-vr-56795

    01/16/2018, 7:03 PM
    Is our target that N should tend to 0? Or that (Y - X) > something? Or something else?
  • w

    witty-crayon-22786

    01/16/2018, 7:04 PM
    i would be fine with N tends to 0.
  • a

    average-vr-56795

    01/16/2018, 7:04 PM
    In that case, we need to be pretty careful around when we lease. So I guess let’s start that discussion 🙂
1...426427428...906Latest