pantsbuild #general

so far we've mostly relied on uncoordinated cache hits, so each new chunk starting (sometimes up to 20 chunks) was a chance to fetch things that other workers had finished building

witty-crayon-22786

08/19/2015, 4:01 PM

other than a lot of additional output, would there be any other negative effects to splitting the

self.invalidated

block into a block per target?

witty-crayon-22786

08/19/2015, 4:03 PM

...i suppose we'd have to schedule the entire context in execution_graph, rather than only scheduling portions that we missed for

bored-art-40741

08/19/2015, 4:07 PM

That defeats the benefit of the global strategy, doesn't it?

witty-crayon-22786

08/19/2015, 4:08 PM

sorry, i guess i was implicitly talking about the isolated strategy

witty-crayon-22786

08/19/2015, 4:09 PM

...which still has a single

invalidated

block per chunk

bored-art-40741

08/19/2015, 4:11 PM

But yeah, where possible being more granular with

invalidated

calls is I think a good default

witty-crayon-22786

08/19/2015, 4:12 PM

@bored-art-40741: would it be feasible to do double checking?

witty-crayon-22786

08/19/2015, 4:12 PM

one large invalidated block, and then a check later once the work is ready to start?

happy-kitchen-89482

08/19/2015, 4:34 PM

We can trim the output, if chattiness is the issue.

witty-crayon-22786

08/19/2015, 4:39 PM

here's the summary of the internal ticket i created... maybe it should be an external ticket.

witty-crayon-22786

08/19/2015, 4:39 PM

Copy code

Now that we have a single large chunk arriving in ZincCompile, the {{self.invalidated}} block for compilation will do a single upfront check for all cache entries (thousands potentially), and then begin compiling them, without ever checking the cache again.

There is one advantage to having the check up front: when we hit the cache, we schedule a smaller set of tasks in the execution graph. But otherwise, not checking the cache lazily is a disadvantage.

Two potential approaches to fixing the issue:
# split the invalidation block into the jobs - this might be the cleanest way to solve the problem
# double-check the cache - if we were to re-check the cache inside execution_graph jobs, immediately before attempting to compile, it might give us the best of both worlds

In both cases, we'll have to be cautious of the effects of checking the cache from a background proc. In theory the cache code is all thread safe though.

witty-crayon-22786

08/19/2015, 4:56 PM

...hm. it looks like the invalidation check bit would require a significant refactor, because that portion currently lives in the base class

witty-crayon-22786

08/19/2015, 4:56 PM

so definitely easier once global has been removed.

enough-analyst-54434

08/19/2015, 5:14 PM

Folks - I'd like to do an pre-release today ahead of my duties releasing on Friday. If there are no objections I'll commence at 11:30am pacific.

witty-crayon-22786

08/19/2015, 5:15 PM

fine with me.

dry-analyst-73584

08/19/2015, 5:41 PM

Hi! I’m applying for one of the Outreachy internships this fall and thought I’d start learning my way around pants working on this: https://github.com/pantsbuild/pants/issues/793

enough-analyst-54434

08/19/2015, 5:46 PM

Excellent! You've probably found this already, but in case not - your dev setup is all described here: http://pantsbuild.github.io/howto_contribute.html

enough-analyst-54434

08/19/2015, 5:48 PM

Is this you?: https://github.com/di0spyr0s - if so I can add you to the contributors role so you can self-assign the githib issue and add the in-progress label

dry-analyst-73584

08/19/2015, 6:05 PM

Yes! That's me simple smile

dry-analyst-73584

08/19/2015, 6:06 PM

And that would be awesome

enough-analyst-54434

08/19/2015, 6:07 PM

K - sent you an invite

enough-analyst-54434

08/19/2015, 6:07 PM

I can assign the issue to you once you accept the invite. The github role is a read-only role, so you won;t be able to do much except be assigned, but there you go

dry-analyst-73584

08/19/2015, 6:10 PM

Thanks!

rough-minister-58256

08/19/2015, 6:55 PM

can anyone think of a good reason to keep the current .setup()-based initialization model for GoalRunner as used here: https://github.com/pantsbuild/pants/blob/master/src/python/pants/bin/pants_exe.py#L79