pantsbuild #general

Channels

welcome

github-notifications

announce

witty-crayon-22786

07/26/2017, 7:55 PM

@bright-apartment-76721 : it's a thing that's possible to generate from the ivy report, but we don't currently have anything that does.

witty-crayon-22786

07/26/2017, 7:56 PM

ie, the XML file that is generated by

./pants resolve.ivy --open $target

contains enough information to generate a bunch of targets that would lock the deps.

witty-crayon-22786

07/26/2017, 7:57 PM

in practice, locking happens incrementally by explicitly declaring more and more deps in 3rdparty as there is a need to pin them.

witty-crayon-22786

07/26/2017, 8:16 PM

hm, but i guess i should say: "in practice, at Twitter"

witty-crayon-22786

07/26/2017, 8:16 PM

square uses the dependency management stuff, which is a whole other strategy: http://www.pantsbuild.org/3rdparty_jvm.html#managing-transitive-dependencies

fast-nail-55400

07/26/2017, 8:23 PM

and as another data point: at foursquare we are beginning to experiment with that dependency management stuff (in order to get ourselves off the

pom-resolve

resolver we wrote a ways back)

bright-apartment-76721

07/26/2017, 10:29 PM

interesting, thanks!

happy-kitchen-89482

07/26/2017, 10:51 PM

@fast-nail-55400 out of curiosity, what are the drawbacks of pom-resolve, that are making you move away from it?

rough-minister-58256

07/26/2017, 10:53 PM

curious as well - iirc, there was a point where we were musing about going the opposite direction (replacing ivy with pom-resolve).

witty-crayon-22786

07/26/2017, 10:54 PM

(i can think of one off the top my head: requires exactly one version of any dep... we have 3 versions of elasticsearch, etc)

fast-nail-55400

07/26/2017, 11:08 PM

hadoop/spark

fast-nail-55400

07/26/2017, 11:08 PM

wonderful stuff such as us using Netty 4.1's HTTP/2 support in order to speak to Apple's APNS gateway

fast-nail-55400

07/26/2017, 11:08 PM

but netty 4.0 being incompatible with netty 4.1

fast-nail-55400

07/26/2017, 11:09 PM

and spark being built against netty 4.0

limited-country-20626

07/26/2017, 11:09 PM

The main problem, honestly, is supporting it ourselves. If we had made the push to get it into Pants master we would probably be able to stay

limited-country-20626

07/26/2017, 11:09 PM

But I wrote a mixin that forces ivy to have a consistent resolve

limited-country-20626

07/26/2017, 11:10 PM

so we are kind of getting the advantages of both

fast-nail-55400

07/26/2017, 11:10 PM

although the pain of supporting it ourselves was "okay" up to the moment we needed to have >1 classpath resolution

fast-nail-55400

07/26/2017, 11:10 PM

and hadoop/spark is the forcing function there

limited-country-20626

07/26/2017, 11:11 PM

We would not be able to force a consistent resolve w/o having had pom-resolve enforce it at the start

fast-nail-55400

07/26/2017, 11:11 PM

we also need to move away from it so that we can have separate classpaths for our datacenter hadoop cluster and any AWS EMR clusters we spin up

limited-country-20626

07/26/2017, 11:11 PM

not to mention that we want to open source our docker plugin

limited-country-20626

07/26/2017, 11:12 PM

and making it be pluggable between ivy and pom-resolve was complex enough to make that work fairly challenging

witty-crayon-22786

07/26/2017, 11:14 PM

that sounds like good progress.

most-wire-83210

07/27/2017, 12:31 AM

@rough-minister-58256 @enough-analyst-54434 - thanks for the help

most-wire-83210

07/27/2017, 12:31 AM

I am able to move to new version of PEX running in bazel rules

most-wire-83210

07/27/2017, 12:31 AM

https://github.com/streamlio/bazel_rules_pex

most-wire-83210

07/27/2017, 12:31 AM

the latest release 0.3.6 works in heron and tested in the code base

most-wire-83210

07/27/2017, 12:32 AM

now adding the platform support to bazel pex rules to generate multi platform pex

rough-minister-58256

07/27/2017, 12:44 AM

sweet