#general
- w
witty-crayon-22786
10/20/2015, 7:53 PM^ in this case, because we now have so much source, the entire internal classpath ends up in a monolithic jar that is larger than all the 3rdparty deps by far - w
witty-crayon-22786
10/20/2015, 7:53 PM@freezing-motorcycle-14222: ^ - f
freezing-motorcycle-14222
10/20/2015, 7:53 PMYeah - f
freezing-motorcycle-14222
10/20/2015, 7:54 PMPreviously the JARS from science were taken via 3rdparty so they were granular, but now that science and birdcage are together the local repository JAR has exploded in size since we are getting so few things via 3rdparty now - w
witty-crayon-22786
10/20/2015, 7:55 PM@freezing-motorcycle-14222: but yea: patch welcome. - f
freezing-motorcycle-14222
10/20/2015, 7:55 PMAwesome. This will make our lives so much easier - b
bored-art-40741
10/20/2015, 8:29 PM@freezing-motorcycle-14222: We've been particularly interested in generating a "classpath" jar that we deploy to all of our hadoop machines to the filesystem - b
bored-art-40741
10/20/2015, 8:29 PMThen when bundling a hadoop target that hasn't messed with the classpath (which is extremely rare), we would put a classpath entry in the generated jar manifest telling it where to find its third party classpath on the filesystem - b
bored-art-40741
10/20/2015, 8:30 PMI was hoping to talk to @full-kilobyte-93429 and @witty-crayon-22786 about the feasibility of this, given that they're working on similar solutions to get around classpath length issues - w
witty-crayon-22786
10/20/2015, 8:30 PM@bored-art-40741: hadoop has a thing they, perhaps confusing, call the "distributed cache". - w
witty-crayon-22786
10/20/2015, 8:31 PMit is basically only used for classpath jars, afaik. - w
witty-crayon-22786
10/20/2015, 8:31 PMbut it is content addressable, and works automatically if you invoke hadoop with stable jars. - w
witty-crayon-22786
10/20/2015, 8:31 PMso i don't think you actually need to do anything there other than have stable jars. - b
bored-art-40741
10/20/2015, 8:31 PMstable jars? - b
bored-art-40741
10/20/2015, 8:31 PMAre there any docs on doing this in practice? - w
witty-crayon-22786
10/20/2015, 8:31 PM - w
witty-crayon-22786
10/20/2015, 8:32 PMi think people use it in practice. - b
bored-art-40741
10/20/2015, 8:32 PMhmmm - w
witty-crayon-22786
10/20/2015, 8:32 PMi mean, i used it in 2006 - b
bored-art-40741
10/20/2015, 8:32 PMI'm familiar with the distributed cache, I was not familiar with using it to add things to the classpath - b
bored-art-40741
10/20/2015, 8:32 PMHowever I'd prefer to have this be opaque and handled by pants - w
witty-crayon-22786
10/20/2015, 8:33 PMif you specify the classpath to MR correctly, it uses the distributed cache automatically - e
enough-analyst-54434
10/20/2015, 8:33 PM"However I'd prefer to have this be opaque and handled by pants" <- ? - e
enough-analyst-54434
10/20/2015, 8:33 PMWe could implement a goal that emitted a bundle with a built-in script for submission of jobs? - e
enough-analyst-54434
10/20/2015, 8:33 PMWhat do you mean by this Patrick? - e
enough-analyst-54434
10/20/2015, 8:34 PMThe script thing would be cool, would need some new data like name node info, etc - b
bored-art-40741
10/20/2015, 8:35 PMThat's one way to do it, though we already have a lot of code around submission of jobs - b
bored-art-40741
10/20/2015, 8:35 PMI guess it makes sense that we could integrate this into that code - e
enough-analyst-54434
10/20/2015, 8:35 PMTwitter had extenstions in scald.rb for this at one point - b
bored-art-40741
10/20/2015, 8:35 PMpants could just dump a metadata file into the resulting jar that the job submission entry point inspects