https://pantsbuild.org/ logo
Powered by
#development
  • h

    happy-kitchen-89482

    01/04/2019, 7:39 AM
    And/or what is a much smaller cacheable set that gives us a better tradeoff
  • h

    happy-kitchen-89482

    01/04/2019, 7:45 AM
    For example, would it be true to say that the test shards that rely on the bootstrapped pants.pex don’t need to cache 
    src/rust/engine/target
    at all, since they’re not building any rust code?
    a
    a
    • 3
    • 2
  • a

    aloof-angle-91616

    01/04/2019, 10:44 AM
    i will look into answering the above^ now or tomorrow, but not licking that cookie yet
    a
    • 2
    • 2
  • a

    aloof-angle-91616

    01/04/2019, 10:46 AM
    i don't want to divert attention from the above, so please read benjy's message. i posted two PRs just now, one that makes 
    @rule
    parse errors much better and have the location and text of the offending 
    yield
    statement (https://github.com/pantsbuild/pants/pull/7023), and another that makes the visualized rule graph much, much nicer to read, i think (https://github.com/pantsbuild/pants/pull/7024).
  • h

    happy-kitchen-89482

    01/04/2019, 7:09 PM
    PS While I’m on a roll I’m making some more changes to the CI (consolidating the native engine builds down to one per platform), so it’s still in flux for now.
    👌 4
    a
    • 2
    • 1
  • a

    average-vr-56795

    01/04/2019, 10:01 PM
    https://internals.rust-lang.org/t/policies-around-default-cpu-architecture/9142 suggests that our OSX builds may be missing a bunch of useful instructions because of super conservative target platform defaults. Not sure how much this actually matters in practice, because we're mostly not numeric operation bound, but may be worth building with an SSE4 target and seeing if it makes a difference at all...
    ✨ 1
  • h

    happy-kitchen-89482

    01/06/2019, 1:59 AM
    OK, I need someone to help me out here: We currently build the native engine on linux in the bootstrap job when building the pex directly in the travis container, and in the “build linux native engine job”, where we build wheels in a docker container inside the travis container, presumably for maximum hermeticity and compatibility. So it seems to me that to consolidate these down to one build, that build needs to be in a docker container. I have attempted to run the pex build (by running 
    build-support/bin/ci.sh
    with no arguments) in the docker container, but am hitting this error:
    Copy code
    info: installing component 'rust-src'
info: downloading component 'clippy'
info: installing component 'clippy'
/travis/home/.cache/pants/rust/cargo/bin/cargo-ensure-installed: /lib64/libc.so.6: version `GLIBC_2.14' not found (required by /travis/home/.cache/pants/rust/cargo/bin/cargo-ensure-installed)
/travis/home/.cache/pants/rust/cargo/bin/cargo-ensure-installed: /lib64/libc.so.6: version `GLIBC_2.15' not found (required by /travis/home/.cache/pants/rust/cargo/bin/cargo-ensure-installed)
/travis/home/.cache/pants/rust/cargo/bin/cargo-ensure-installed: /lib64/libc.so.6: version `GLIBC_2.18' not found (required by /travis/home/.cache/pants/rust/cargo/bin/cargo-ensure-installed)
Failed to build native engine.
+ die 'Failed to bootstrap pants.'
+ (( 1 > 0 ))
+ log '\n\x1b[31mFailed to bootstrap pants.\x1b[0m'
+ echo -e '\n\x1b[31mFailed to bootstrap pants.\x1b[0m'
Failed to bootstrap pants.
    a
    a
    • 3
    • 53
  • h

    happy-kitchen-89482

    01/06/2019, 1:59 AM
  • h

    happy-kitchen-89482

    01/06/2019, 2:01 AM
    Any insight, rust folk? I’m assuming that the right thing to do is build the pex in the docker container, then have the “build linux native engine” job use the prebuilt native_engine.so extracted from the pex when building the wheel. Does that make sense as a strategy?
    a
    • 2
    • 4
  • a

    aloof-angle-91616

    01/06/2019, 2:03 AM
    looking
  • h

    happy-kitchen-89482

    01/06/2019, 2:06 AM
    Running 
    release.sh -n
    in that same docker image works, by which I mean successfully bootstraps the native engine
  • h

    happy-kitchen-89482

    01/06/2019, 2:06 AM
    but for some reason 
    ci.sh
    doesn’t work
  • a

    aloof-angle-91616

    01/06/2019, 6:54 PM
    @average-vr-56795 from this comment -- is there a benefit to using musl vs providing glibc? if it's more likely to be packageable / runnable on arbitrary linux environments i'm all for it -- i'm just not sure how that fits into the packaging process right now: https://pantsbuild.slack.com/archives/C0D7TNJHL/p1546773452021400?thread_ts=1546739970.015700&cid=C0D7TNJHL
    a
    h
    • 3
    • 23
  • h

    happy-kitchen-89482

    01/06/2019, 9:35 PM
    qq about the cargo cache (in 
    ~/.cache/pants/rust/
    )
  • h

    happy-kitchen-89482

    01/06/2019, 9:35 PM
    We cache this between CI runs
  • h

    happy-kitchen-89482

    01/06/2019, 9:35 PM
    But we prune out 
    ~/.cache/pants/rust/rustup
    first
  • h

    happy-kitchen-89482

    01/06/2019, 9:35 PM
    Which leaves the directory 
    ~/.cache/pants/rust/cargo
  • h

    happy-kitchen-89482

    01/06/2019, 9:36 PM
    And some symlinks, e.g., on my system:
  • h

    happy-kitchen-89482

    01/06/2019, 9:36 PM
    Copy code
    gonzo:[~/src/pants][ci_consolidation]$ ls -la ~/.cache/pants/rust/
total 0
drwxr-xr-x   6 benjyw  staff  192 Dec 25 23:22 .
drwxr-xr-x  14 benjyw  staff  448 Nov 27 13:24 ..
drwxr-xr-x   7 benjyw  staff  224 Nov 30 11:51 cargo
lrwxr-xr-x   1 benjyw  staff   86 Nov 15 06:42 cargo-1.30.0-b245396faf5f51df0f6ce3875a6285e4bb7fc563 -> /Users/benjyw/.cache/pants/rust/rustup/toolchains/1.30.0-x86_64-apple-darwin/bin/cargo
lrwxr-xr-x   1 benjyw  staff   54 Dec 25 23:22 cargo-1.31.1-b245396faf5f51df0f6ce3875a6285e4bb7fc563 -> rustup/toolchains/1.31.1-x86_64-apple-darwin/bin/cargo
drwxr-xr-x   7 benjyw  staff  224 Nov 15 06:42 rustup
  • h

    happy-kitchen-89482

    01/06/2019, 9:36 PM
    My question is whether we need to cache those symlinks.
  • h

    happy-kitchen-89482

    01/06/2019, 9:37 PM
    If so, then we have to continue to cache all of 
    ~/.cache/pants/rust/
    and prune out 
    rustup
  • a

    average-vr-56795

    01/06/2019, 9:37 PM
    If the targets exist, we should cache them. If the targets don’t exist, it doesn’t matter whether we do or don’t
  • h

    happy-kitchen-89482

    01/06/2019, 9:37 PM
    If not then we can just cache 
    ~/.cache/pants/rust/cargo
    and be done with it
  • a

    aloof-angle-91616

    01/06/2019, 9:37 PM
    would working on https://github.com/pantsbuild/pants/pull/6891 today (rust/cargo ruleset) be at all useful for this?
    a
    • 2
    • 3
  • a

    average-vr-56795

    01/06/2019, 9:37 PM
    The stuff in rustup is just downloaded from the internet, so no real point in caching it, so might as well just cache the cargo dir
  • h

    happy-kitchen-89482

    01/06/2019, 9:37 PM
    Is there value in caching the 
    cargo
    subdir without the sibling symlinks
  • h

    happy-kitchen-89482

    01/06/2019, 9:38 PM
    Oh good point, those symlinks point into 
    rustup
    which we delete
  • h

    happy-kitchen-89482

    01/06/2019, 9:38 PM
    ha, ok, so obviously we don’t need the symlinks. Good.
  • h

    happy-kitchen-89482

    01/06/2019, 9:38 PM
    I’m just trying to shrink our absurdly large caches.
  • h

    happy-kitchen-89482

    01/06/2019, 9:39 PM
    and simplify the cache-related stanzas in our .travis.yml
1...603604605...906Latest