pantsbuild #general

so I have a build that quickly goes into some sort of memory leak. It begins paging memory into a python process and the CPU stops working. The python process grows to eat all available memory ( I have stopped it as high as 12+ gbs).

limited-country-20626

09/27/2015, 8:32 PM

Cracking open the heap reveals 90%+ of memory is a list of tuples containing canceled jobs that look like:

Copy code

(u'compile(test/jvm/com/foursquare/test_alpha/test:test)', u'Canceled', None)
(u'compile(test/jvm/com/foursquare/test_alpha/test:test)', u'Canceled', None)
(u'compile(test/jvm/com/foursquare/test_beta/test:test)', u'Canceled', None)

limited-country-20626

09/27/2015, 8:32 PM

with repeated jobs as shown above.

limited-country-20626

09/27/2015, 8:33 PM

The job starts as normal but then slows dramatically. The CPU stops spinning and memory just pages. Every once and awhile, a job will get kicked out and actually compile (sometimes after five minutes on no CPU activity).

limited-country-20626

09/27/2015, 8:36 PM

Oh, and this is off of pants master so we are all caught up with the current HEAD.

limited-country-20626

09/27/2015, 8:43 PM

Might be a problem with the queuing in the execution_graph. Any thoughts, @witty-crayon-22786 or maybe @fancy-queen-20734?

limited-country-20626

09/27/2015, 9:46 PM

Oh yeah, one last data point. The list of jobs is excerpted from a file with just a list of those tuples and the file is 6 gbs large 😱

most-cricket-96651

09/28/2015, 3:42 PM

Hey Patrick: Yeah, I'd love to pitch. You're at 4sq, right? email me at dbentley@twitter.com and we can set something up?

fancy-queen-20734

09/28/2015, 4:09 PM

@limited-country-20626: wow. that’s terrible.

fancy-queen-20734

09/28/2015, 4:44 PM

@limited-country-20626: The execution graph doesn’t do cycle detection right now, maybe you have a cycle? That could explain the behavior because cancelations are propagated down the DAG—so if there’s a cycle, I think it would spin on it.

limited-country-20626

09/28/2015, 4:49 PM

That is my guess too. I absolutely could have a cycle...this was a mechanical search and replace, basically as big of a bg as we could produce. I am working on a hack to not repeatedly add jobs as an experiment.

fancy-queen-20734

09/28/2015, 4:51 PM

The failure branch of handling results could check the status table and not add dependees if the status is done already

fancy-queen-20734

09/28/2015, 4:51 PM

that would be a simple fix I think—that plus warning on that case, since it implies a cycle

limited-country-20626

09/28/2015, 5:38 PM

thanks @fancy-queen-20734 I will give that a look tonight.

happy-kitchen-89482

09/28/2015, 5:44 PM

I'll be in our NY office tomorrow and Wednesday, so if @most-cricket-96651 is able to do this on Wednesday, I could lend an ear.

most-cricket-96651

09/28/2015, 5:47 PM

OK, Benjy, can you send me & Patrick an email so we can coordinate offline?

bored-art-40741

09/28/2015, 5:53 PM

@fancy-queen-20734: A cycle in the target graph? How could it get that far, though? We have to topo sort the target graph first, and that has cycle detection

bored-art-40741

09/28/2015, 5:54 PM

@happy-kitchen-89482 @most-cricket-96651: I will be in the office starting Wednesday this week

mammoth-pharmacist-54215

09/28/2015, 6:15 PM

I was a bit bored this weekend and cleaned up arg_splitter.py a little. There is no bug for this that I know of… should I create one for or just put up a RB without a bug associated with it?

ancient-stone-50795

09/28/2015, 6:22 PM

@mammoth-pharmacist-54215: Usually, I create a PR in github and that is the bug ID I use

mammoth-pharmacist-54215

09/28/2015, 6:23 PM

Sounds good, will do

bright-dinner-81701

09/28/2015, 6:23 PM

hey @witty-crayon-22786 i saw your comment