https://pantsbuild.org/ logo
Powered by
#general
  • h

    happy-kitchen-89482

    09/21/2015, 11:54 PM
    No, same compression
  • h

    happy-kitchen-89482

    09/21/2015, 11:54 PM
    ~83%
  • r

    rough-minister-58256

    09/21/2015, 11:54 PM
    hypothesis: the same files are being added to the tarfile multiple times
  • h

    happy-kitchen-89482

    09/21/2015, 11:54 PM
    But interestingly, it thinks the uncompressed size should be much bigger than it actually is.
  • h

    happy-kitchen-89482

    09/21/2015, 11:55 PM
    @rough-minister-58256: let me see if the uncompressed size it reports is an exact multiple of the real size.
  • r

    rough-minister-58256

    09/21/2015, 11:56 PM
    Copy code
    >>> import tarfile
>>> with tarfile.open('/tmp/xyz.tar.gz', 'w:gz') as t:
...   t.add('/tmp/pants.out')
... 

[illuminati tmp]$ ls -alh /tmp/xyz.tar.gz
-rw-r--r--  1 kwilson  wheel    36K Sep 21 16:54 /tmp/xyz.tar.gz

>>> with tarfile.open('/tmp/xyz.tar.gz', 'w:gz') as t:
...   t.add('/tmp/pants.out')
...   t.add('/tmp/pants.out')
...   t.add('/tmp/pants.out')
...   t.add('/tmp/pants.out')
... 

[illuminati tmp]$ ls -alh /tmp/xyz.tar.gz
-rw-r--r--  1 kwilson  wheel   141K Sep 21 16:54 /tmp/xyz.tar.gz
  • r

    rough-minister-58256

    09/21/2015, 11:56 PM
    strange behavior, but perhaps supported by the tar format
  • h

    happy-kitchen-89482

    09/21/2015, 11:57 PM
    what does it report when you gunzip --list in each case?
  • r

    rough-minister-58256

    09/21/2015, 11:58 PM
    Copy code
    [illuminati tmp]$ gunzip --list /tmp/xyz.tar.gz 
  compressed uncompressed  ratio uncompressed_name
       36603       337920  89.1% /tmp/xyz.tar
[illuminati tmp]$ gunzip --list /tmp/xyz.tar.gz 
  compressed uncompressed  ratio uncompressed_name
      144764      1331200  89.1% /tmp/xyz.tar
  • h

    happy-kitchen-89482

    09/21/2015, 11:59 PM
    So it's not an exact multiple
  • h

    happy-kitchen-89482

    09/21/2015, 11:59 PM
    but maybe we're only adding some of the files multiple times
  • r

    rough-minister-58256

    09/21/2015, 11:59 PM
    a 
    tar tzvf <tar>
    does show 4 copies of pants.out tho - so presumably that’d be a quick test
  • h

    happy-kitchen-89482

    09/21/2015, 11:59 PM
    let's see
  • r

    rough-minister-58256

    09/22/2015, 12:00 AM
    tar tzvf <tar> |awk '{print $NF}' | sort | uniq -c
  • h

    happy-kitchen-89482

    09/22/2015, 12:00 AM
    I don't see any
  • r

    rough-minister-58256

    09/22/2015, 12:00 AM
    hrmm!
  • h

    happy-kitchen-89482

    09/22/2015, 12:01 AM
    yeah, was just creating a |cut|sort|uniq cmd
  • h

    happy-kitchen-89482

    09/22/2015, 12:01 AM
    will use yours
  • h

    happy-kitchen-89482

    09/22/2015, 12:01 AM
    Well, what do you know...
  • h

    happy-kitchen-89482

    09/22/2015, 12:01 AM
    we're adding each class file 5 times
  • r

    rough-minister-58256

    09/22/2015, 12:01 AM
    aww yiss
  • m

    mammoth-pharmacist-54215

    09/22/2015, 12:01 AM
    5x the fun simple smile
  • h

    happy-kitchen-89482

    09/22/2015, 12:02 AM
    Copy code
    fozzie:[/tmp/cachetest]$ tar tvfz artifact2.tgz | awk '{print $NF}' | sort | uniq -c
   1 compile/jvm/zinc/isolated-analysis/src.jvm.com.foursquare.common.zookeeper.zookeeper.analysis.portable
   1 compile/jvm/zinc/isolated-classes/src.jvm.com.foursquare.common.zookeeper.zookeeper/com/
   2 compile/jvm/zinc/isolated-classes/src.jvm.com.foursquare.common.zookeeper.zookeeper/com/foursquare/
   3 compile/jvm/zinc/isolated-classes/src.jvm.com.foursquare.common.zookeeper.zookeeper/com/foursquare/common/
   4 compile/jvm/zinc/isolated-classes/src.jvm.com.foursquare.common.zookeeper.zookeeper/com/foursquare/common/zookeeper/
   5 compile/jvm/zinc/isolated-classes/src.jvm.com.foursquare.common.zookeeper.zookeeper/com/foursquare/common/zookeeper/ConcreteZooKeeperCacheService$$anon$1.class
   5 compile/jvm/zinc/isolated-classes/src.jvm.com.foursquare.common.zookeeper.zookeeper/com/foursquare/common/zookeeper/ConcreteZooKeeperCacheService$$anonfun$1.class
   5 compile/jvm/zinc/isolated-classes/src.jvm.com.foursquare.common.zookeeper.zookeeper/com/foursquare/common/zookeeper/ConcreteZooKeeperCacheService$$anonfun$2.class
  • h

    happy-kitchen-89482

    09/22/2015, 12:02 AM
    Suspicious
  • h

    happy-kitchen-89482

    09/22/2015, 12:02 AM
    we add the root dir once
  • h

    happy-kitchen-89482

    09/22/2015, 12:02 AM
    then its child twice
  • h

    happy-kitchen-89482

    09/22/2015, 12:02 AM
    and that thing's child 3 times...
  • h

    happy-kitchen-89482

    09/22/2015, 12:02 AM
    etc etc
  • h

    happy-kitchen-89482

    09/22/2015, 12:02 AM
    I think we might be doing an unnecessary walk
  • h

    happy-kitchen-89482

    09/22/2015, 12:02 AM
    and then adding each dir recursively
1...415416417...2420Latest