https://pantsbuild.org/ logo
Powered by
#development
  • r

    red-balloon-89377

    11/12/2018, 8:05 PM
    thanks!
  • w

    witty-crayon-22786

    11/12/2018, 8:41 PM
  • w

    witty-crayon-22786

    11/12/2018, 8:41 PM
    generally i don't look at the perf impact of rust version bumps, but that will be one to watch out for
  • r

    red-balloon-89377

    11/12/2018, 8:43 PM
    It will be interesting to see if we see the hit, might be worth testing with jemalloc and with default a couple of times just to see if there is a significant slowdown with the system alloc
  • w

    wide-energy-11069

    11/13/2018, 7:47 PM
    qq about clap: I have
    Copy code
    let untar_opts = &App::new("tar_api")
    .subcommand(
      SubCommand::with_name("untar")
    ).get_matches();

  let tar_file = untar_opts
    .value_of("file")
    .map(PathBuf::from)
    .unwrap();

  let dest = untar_opts
    .value_of("dest")
    .map(PathBuf::from)
    .unwrap();
  • w

    wide-energy-11069

    11/13/2018, 7:48 PM
    Copy code
    $ cargo run --bin tar-cli untar --file=/Users/yic/.cache/pants/buildcache/435112070d95ee9e68888acaa47cc019e37477e2/wilyns.thrift.src.main.thrift.thrift-scala/eca0524003263ae610f3d8d9d61ffdd834845a26-JavaThriftLibraryFingerprintStrategy.0c5015cdf521_f8a312d9d0c3.tgz --dest /tmp/x
    Finished dev [unoptimized + debuginfo] target(s) in 0.85s                                                                                                                                             
     Running `/Users/yic/workspace/pants/src/rust/engine/target/debug/tar-cli untar --file=/Users/yic/.cache/pants/buildcache/435112070d95ee9e68888acaa47cc019e37477e2/wilyns.thrift.src.main.thrift.thrift-scala/eca0524003263ae610f3d8d9d61ffdd834845a26-JavaThriftLibraryFingerprintStrategy.0c5015cdf521_f8a312d9d0c3.tgz --dest /tmp/x`
error: Found argument '--file' which wasn't expected, or isn't valid in this context
  • w

    wide-energy-11069

    11/13/2018, 7:48 PM
    what should be the cli format?
  • w

    wide-energy-11069

    11/13/2018, 8:04 PM
    Progress:
    Copy code
    let untar_opts = &App::new("tar_api")
    .subcommand(
      SubCommand::with_name("untar")
        .arg(Arg::with_name("file").required(true).takes_value(
          true,
        ))
        .arg(Arg::with_name("dest").required(true).takes_value(
          true,
        ))
    ).get_matches();
    seems better
  • w

    wide-energy-11069

    11/13/2018, 8:04 PM
    but
    Copy code
    untar_opts
    .value_of("file")
    gives None
    e
    • 2
    • 3
  • w

    witty-crayon-22786

    11/13/2018, 8:34 PM
    @wide-energy-11069: haven't added feedback to the review yet, but: this doesn't need to be subprocessed, does it?
  • w

    witty-crayon-22786

    11/13/2018, 8:35 PM
    i'm not sure it needs to be its own crate even
  • w

    wide-energy-11069

    11/13/2018, 8:35 PM
    just creating an cli tool to lower the iteration cycle on experiments
  • w

    wide-energy-11069

    11/13/2018, 8:36 PM
    no plan to commit it
  • w

    witty-crayon-22786

    11/13/2018, 8:36 PM
    k
  • w

    wide-energy-11069

    11/13/2018, 11:07 PM
    looks like I’d have to try implementing zipping in rust, so the interface would take a list of str (paths). is there an example taking python list of str across rust boundary?
  • w

    witty-crayon-22786

    11/13/2018, 11:19 PM
    @wide-energy-11069: yes: take a look at the synchronous snapshot method
  • w

    witty-crayon-22786

    11/13/2018, 11:20 PM
  • w

    witty-crayon-22786

    11/13/2018, 11:22 PM
    (and a caller on the python side)
  • w

    wide-energy-11069

    11/13/2018, 11:24 PM
    great, thanks!
  • w

    witty-crayon-22786

    11/13/2018, 11:27 PM
    @wide-energy-11069: oh, hm. i guess we also have this interface, which is probably easier: https://github.com/pantsbuild/pants/blob/master/src/rust/engine/src/lib.rs#L501-L502
  • w

    witty-crayon-22786

    11/13/2018, 11:28 PM
    one more thing though: you might be able to use 
    PathGlobs
    as your interface here.
  • w

    witty-crayon-22786

    11/13/2018, 11:29 PM
    basically: "capture a tarball for these PathGlobs relative to this root".
  • w

    witty-crayon-22786

    11/13/2018, 11:29 PM
    which is very close to the method i originally linked.
  • w

    wide-energy-11069

    11/13/2018, 11:31 PM
    I mean to replace https://github.com/pantsbuild/pants/blob/master/src/python/pants/cache/artifact.py#L104-L109, within the scope, all the paths are already known
  • w

    wide-energy-11069

    11/13/2018, 11:31 PM
    hence a list of str, which should be simpler than 
    PathGlobs
    ?
  • w

    witty-crayon-22786

    11/13/2018, 11:31 PM
    yea.
  • w

    witty-crayon-22786

    11/13/2018, 11:32 PM
    also, the snapshot method is more complicated because it supports batch snapshotting, which isn't necessary here.
  • w

    wide-energy-11069

    11/13/2018, 11:32 PM
    gotcha
  • w

    witty-crayon-22786

    11/13/2018, 11:32 PM
    (...as far as i know)
  • w

    wide-energy-11069

    11/13/2018, 11:48 PM
    ok, glad I did the sanity test. hardcoded the following:
    Copy code
    pub fn compress() -> Result<(), std::io::Error> {
  let tar_gz = File::create("archive.tar.gz")?;
  let enc = GzEncoder::new(tar_gz, Compression::default());
  let mut tar = tar::Builder::new(enc);
  let mut f = File::open("/tmp/good/gen/scrooge/297e53822ddf/wilyns.thrift.src.main.thrift.thrift-scala/c8959dfdc97a/com/twitter/wilyns/thriftscala/LookupRequest.scala").unwrap();

  tar.append_file("gen/scrooge/297e53822ddf/wilyns.thrift.src.main.thrift.thrift-scala/c8959dfdc97a/com/twitter/wilyns/thriftscala/LookupRequest.scala",
                  &mut f)?;
  Ok(())
}
    generates 
    archive.tar.gz"
    . Then 
    tar -xvf archive.tar.gz
    produces the correct 
    LookupRequest.scala
    , whereas using rust to untar it giving 
    LookupRequest.scala
    as a directory.
1...594595596...906Latest