https://pantsbuild.org/ logo
Powered by
#development
  • h

    hundreds-breakfast-49010

    02/27/2020, 12:18 AM
    okay I think I got things typechecking that way
  • h

    hundreds-breakfast-49010

    02/27/2020, 12:19 AM
    there seems to be this new 
    IntoFuture
    trait that says it's implemented on 
    Result<T,E>
    and seems like it's meant to be the way to lift a result into a future, but I couldn't get it to typecheck
    a
    • 2
    • 2
  • a

    aloof-angle-91616

    02/27/2020, 12:19 AM
    ah
  • h

    hundreds-breakfast-49010

    02/27/2020, 12:35 AM
    why do we forbid the standard library 
    Mutex
    in favor of the parking_lot one?
    a
    • 2
    • 2
  • h

    hundreds-breakfast-49010

    02/27/2020, 2:15 AM
    is there a reason why 
    ProjectTree
    and 
    FileSystemProjectTree
    are two different classes?
    đź‘Ť 1
    a
    • 2
    • 13
  • h

    hundreds-breakfast-49010

    02/28/2020, 7:06 PM
  • a

    aloof-angle-91616

    02/28/2020, 9:34 PM
  • h

    happy-kitchen-89482

    03/03/2020, 1:56 AM
    Hi all, I'm making a standalone example repo specifically to demonstrate v2 Python support: https://github.com/pantsbuild/example-python just FYI
  • h

    hundreds-breakfast-49010

    03/04/2020, 12:01 AM
    I'd like to get some more pants' people's eyes on https://github.com/pantsbuild/pants/pull/9197, particularly @average-vr-56795 when he's back online UK time
  • h

    hundreds-breakfast-49010

    03/04/2020, 1:37 AM
    we're trying to debug some performance issues with running PEXs, and this is leading us into questions about graph and process_execution code that I'd like to get @witty-crayon-22786’s and maybe @average-vr-56795’s input on
    w
    h
    +3
    • 6
    • 116
  • h

    hundreds-breakfast-49010

    03/04/2020, 6:36 PM
    related to the discussion in the above thread from yesterday: we think that one major cause of the performance bottlenecks we're seeing, is many rules making multiple identical ExecuteProcessRequests simultaneously, and therefore being unable to take advantage of the cache
  • h

    hundreds-breakfast-49010

    03/04/2020, 6:38 PM
    I'm not sure what the engine currently does in this scenario - I think what this looks like from the graph perspective is there being many nodes representing the same EPR with 
    EntryState::Running
    , all put onto the futures scheduler at the same time
  • h

    hundreds-father-404

    03/04/2020, 6:38 PM
    Specifically, imagine you have a completely cold cache and run 
    ./v2 test
    on 50 targets with identical requirements and interpreter constraints. Because the cache is cold, we believe that will schedule 50 different EPRs to resolve the same Pex. Once one EPR finishes, it will write it to the cache, but it would be too late for the 49 other targets to leverage that result. The other 49 targets would already be doing the same work and re-writing to the cache. After those 50 targets run, then the cache would function normally. We’re trying to optimize for the use case of a completely cold cache
  • h

    hundreds-breakfast-49010

    03/04/2020, 6:39 PM
    because if the 
    EntryState
    was still 
    NotStarted
    , when we tried to schedule that node, it would successfully do a cache lookup and not execute anything
  • h

    hundreds-breakfast-49010

    03/04/2020, 6:39 PM
    but in the cold-cache case eric mentions, I think we would be seeing a bunch of 
    EntryState::Running
    nodes simultaneously
  • h

    hundreds-father-404

    03/04/2020, 6:40 PM
    At a high level, we’re thinking maybe add logic to the scheduler for processes (only processes) to check if any processes are already scheduled with the exact same inputs. If so, then don’t request a new process and somehow wait for the result of that first process
  • f

    fast-nail-55400

    03/04/2020, 6:47 PM
    would you be able to de-dupe requests by the action’s digest?
  • h

    hundreds-father-404

    03/04/2020, 6:48 PM
    would you be able to de-dupe requests by the action’s digest?
    I think we’d use the same functionality we use to determine the input key to the cache. I believe this is from the Python object’s 
    __eq__
    implementation. Generally, it’s comparing that every field of the object is equal. We already have that comparison code to determine if two ExecuteProcessRequests are identical. We’d only need to modify the scheduling code
  • h

    hundreds-breakfast-49010

    03/04/2020, 9:30 PM
    @witty-crayon-22786 am I correct in my understanding that in a cold cache scenario where lots of identical EPRs are being requested at the same time, we're not currently taking advantage of the cache? and that this can be fixed in principle by modifying how the graph code works?
  • h

    hundreds-father-404

    03/04/2020, 9:31 PM
    *We’re not currently leveraging the result of a process that was scheduled around the same time and did the same work, but finished before this process.
  • h

    hundreds-breakfast-49010

    03/04/2020, 10:11 PM
    I just noticed that the 
    capture_snapshots
    extern function is ignoring the global --pants-ignore list. is that deliberate?
    w
    • 2
    • 2
  • h

    hundreds-breakfast-49010

    03/04/2020, 10:11 PM
    (specifically it proxies to 
    capture_snapshot_from_arbitrary_root
    which creates a 
    PosixFs
    with an empty list of ignore files, regardless of what the global options to pants for this are)
  • h

    hundreds-breakfast-49010

    03/05/2020, 12:04 AM
    sorry to be pinging people again but I'd really like to get https://github.com/pantsbuild/pants/pull/9197 merged after some more pants people have looked at it. I added a new type abstraction on @average-vr-56795's suggestion to reflect the fact that many parts of pants won't care about the platform the result ran on; and @early-needle-54791’s input would be helpful
    e
    • 2
    • 2
  • w

    witty-crayon-22786

    03/05/2020, 2:59 PM
    No, we would absolutely take advantage of the cache
  • w

    witty-crayon-22786

    03/05/2020, 2:59 PM
    A Node has an identity in the graph: two "identical" EPRs are the same node in the graph.
    đź‘Ť 1
  • w

    witty-crayon-22786

    03/05/2020, 3:00 PM
    If you're seeing things that you think are identical running independently, then with >99% probability, they are not identical
  • w

    witty-crayon-22786

    03/05/2020, 3:01 PM
    The "NodeKey" eq/hash implementations that we use are what define identity
  • w

    witty-crayon-22786

    03/05/2020, 3:28 PM
    You can use --native-engine-visualize-to to render this for small graphs: it dumps the rule graph when it starts, but it also renders a "runtime" graph after each request that will show you exactly which Nodes are in the graph
    đź‘Ť 1
  • w

    witty-crayon-22786

    03/05/2020, 3:28 PM
    (but the output is very large, so you'd want to use it for your smallest possible case)
  • w

    witty-crayon-22786

    03/05/2020, 3:29 PM
    @hundreds-father-404: you saw mismatched identities before with the unordered set thing, iirc
    đź‘Ť 1
1...696697698...906Latest