10k? way more than that, no?

that's one build

Maybe a bit more

It's ~4k targets times whatever number of task produce small artifacts for those targets

(smaller than that in practice, since most tasks don't operate over all targets)

But it's a good point--finding a way to mess up HDFS would be very bad

Considering that the write throughput is trivial, having the "backend" of the cache just be a group of sharded nginx servers would also work fine

Alternatively we could just use separate caches for Zinc artifacts vs everything

Since zinc artifacts are 99+% of the cache throughput

not in terms of requests

volume, yes

Volume is the issue

er, no... not in the HDFS case

Right, which is why zinc would go through HDFS and everything else would go to our existing cache

Which can handle lots of requests fine, but which is volume limited

That would limit the files on HDFS to only large(ish) zinc artifacts, and only a few thousand a day at most new files

And zinc artifact retention is the easiest one

Question: anyone ever seen a -9 exit code from jarjar, when trying to bootstrap Zinc (in a new repo)? Googling doesn't come up with much. I have a playground repo on github (personal pet project, not related to Square work), and just updated it to use pants (works great on my laptop). But it blows up trying to bootstrap its tools in travis. On @ancient-stone-50795's advice, I copied some stuff from pantsbuild/pants' travis config, but that didn't help. Here's a link to the failed build. I turned up debug output, but there's nothing useful. Nothing written to stdout or stderr. It just exits with -9. TravisCI's docs suggest the error would be different if the build ran out of resources. https://travis-ci.org/jhump/bluegosling/builds/111907440

I'd guess that you exceeded the heap

We don't have any specific settings that I can find, but I know this was an issue for us in the past

Oh, this is a travis failure?

Does it fail locally?

@straight-river-55032: kill -9 can indicate that the OS memory killer is at work.

@bored-art-40741: does not fail locally, only in travis

@straight-river-55032: did you try adding JVM tuning to pants.ini?

[DEFAULT]
# TODO: Still needed until we migrate jvm tools to subsystems.
jvm_options: ["-Xmx1g", "-XX:MaxPermSize=256m"]
[jvm]
options: ["-Xmx1g", "-XX:MaxPermSize=256m"]

huh I wonder what the container sizes are

(in travis)

I don't know why jarjar would want more than 1G of heap, but its what we have set in Pants' own repo

@ancient-stone-50795, ok, I'll try that. Travis CI doc indicates a VM can use up to 3g. Seems odd that jarjar would try to use more than that?

If nailgun is in use (I think I pointed you to the

pants.travis-ci.ini

from the pants repo that turns it off) , you may have many active jvms just lying around. It may just be the straw that broke the camel's back.