pantsbuild #general

Channels

welcome

github-notifications

announce

straight-river-55032

02/26/2016, 2:26 AM

I did already disable nailgun.

straight-river-55032

02/26/2016, 2:26 AM

That did it! (the heap limits)

straight-river-55032

02/26/2016, 2:26 AM

Just got past that step. Thanks!

ancient-stone-50795

02/26/2016, 2:27 AM

that's a bit nuts

bored-art-40741

02/26/2016, 2:35 AM

ah yeah this is ours, now that I know what to look for

bored-art-40741

02/26/2016, 2:35 AM

Copy code

[jar-tool]
jvm_options: ['-Xms2g','-Xmx2g', '-Djava.io.tmpdir=%(pants_workdir)s']

bored-art-40741

02/26/2016, 2:36 AM

We easily blew the heap on our jar tool invocations

bored-art-40741

02/26/2016, 2:38 AM

The Xms is probably unnecessary, but the Xmx is critical

straight-river-55032

02/26/2016, 2:43 AM

BTW, where do the bootstrapped tools and the venv get cached? I've added some cache statements in travis (copying the ones from pantsbuild/pants, as a starting point), but it still repeats all of the bootstrap steps each build, taking an extra ~90 seconds. I'm thinking I want to ask it to save more stuff from build to build...

straight-river-55032

02/26/2016, 2:44 AM

I'm currently saving "$HOME/.cache/pants", "$HOME/.ivy2/pants", and "$HOME/.npm" (total scientist dog here...)

ancient-stone-50795

02/26/2016, 2:45 AM

The venv in the pants repo is under build-support/

straight-river-55032

02/26/2016, 2:46 AM

Those are from pants project's own travis CI settings. Methinks I may just want ".cache" and ".pants.d" instead, or in addition?

ancient-stone-50795

02/26/2016, 2:46 AM

But if you are using the

pants

setup script I think it puts it in

PANTS_HOME="${PANTS_HOME:-${HOME}/.cache/pants/setup}"

ancient-stone-50795

02/26/2016, 2:46 AM

So your caching should get it. You don't want to cache .pants.d in general. That's going to grow without bound.

straight-river-55032

02/26/2016, 2:46 AM

ancient-stone-50795

02/26/2016, 2:46 AM

And besides, if its a real CI, you want to build from a clean-all state

ancient-stone-50795

02/26/2016, 2:49 AM

I think some auto-pruning of .pants.d is now being done. https://rbcommons.com/s/twitter/r/3444/, this recently landed

bored-art-40741

02/26/2016, 2:50 AM

Which bootstrap steps specifically?

bored-art-40741

02/26/2016, 2:50 AM

Can you link the output?

straight-river-55032

02/26/2016, 4:06 AM

Sorry for delayed response. It spends ~1 minute every time in

[shade-zinc]

before it starts actually compiling: https://travis-ci.org/jhump/bluegosling/builds It looks like the actual virtual envs (e.g. pants bootstrapping itself) is properly cached. It has more "bootstrap" steps printed out in travis, but this is the one that takes a while. (On my laptop, it skips all of these and gets right into compiling code in ~1 second.)

bored-art-40741

02/26/2016, 4:10 AM

Ah, I think that's a combo of making sure your jvm tool shading is cached, and making sure travis respects that cache

bored-art-40741

02/26/2016, 4:10 AM

e.g. internally we have

bored-art-40741

02/26/2016, 4:10 AM

Copy code

[cache.bootstrap.bootstrap-jvm-tools]
write_to: %(local_and_remote_artifact_caches)s
read_from: %(local_and_remote_artifact_caches)s

bored-art-40741

02/26/2016, 4:10 AM

Though I'm not sure how that translates exactly to travis configs

straight-river-55032

02/26/2016, 4:23 AM

K. I think that's enough for me to go on. Thanks.

red-cpu-25099

02/26/2016, 4:04 PM

I'm trying to figure out how caching works in case of

bootstrap_jvm_tools

. I can see where write to cache is occur: https://github.com/pantsbuild/pants/blob/369da95dc6f4dd7b3d67b7b2cc9fb82b32457eb0/src/python/pants/backend/jvm/tasks/bootstrap_jvm_tools.py#L274, but how read from cache is working?

bored-art-40741

02/26/2016, 4:48 PM

self.invalidated

above

bored-art-40741

02/26/2016, 4:48 PM

https://github.com/pantsbuild/pants/blob/369da95dc6f4dd7b3d67b7b2cc9fb82b32457eb0/src/python/pants/backend/jvm/tasks/bootstrap_jvm_tools.py#L231

bored-art-40741

02/26/2016, 4:49 PM

self.invalidated

hits the cache, the target will be in

invalidation_check.valid_vts

rather than invalid

red-cpu-25099

02/26/2016, 5:02 PM

ouch, and this will be only one vts which corresponds to set of all targets ?