#general
- w
witty-crayon-22786
06/05/2017, 6:25 PMsure. - s
straight-waiter-86090
06/05/2017, 6:26 PMgood to know that this will be something the daemon will address in the future. - w
witty-crayon-22786
06/05/2017, 6:26 PMour current situation internally is that we have v2 enabled on user machines, and the daemon disabled on laptops, but will shortly have it* enabled in ci. - w
witty-crayon-22786
06/05/2017, 6:29 PMbut because of the alpha status, you need to
between runs in CIclean-all - s
straight-waiter-86090
06/05/2017, 6:32 PMquestion about caches - does your setup have developers pulling from a central cache (like in the cloud), or do you push to caches that are local to developer offices? - w
witty-crayon-22786
06/05/2017, 6:32 PMour cache lives in our datacenters... so sortof like using s3 - w
witty-crayon-22786
06/05/2017, 6:34 PMour UK office has a local cache that proxies to the datacenter, just because their latency isn't great otherwise - w
witty-crayon-22786
06/05/2017, 6:35 PMso https://github.com/pantsbuild/pants/pull/4589 is definitely interesting as a general case - s
straight-waiter-86090
06/05/2017, 6:36 PMthat’s interesting. I was wondering if lower latency would help. Sometimes it’s 30 second plus or so for the zinc cache hits to be retrieved from our data center. - w
witty-crayon-22786
06/05/2017, 6:36 PMsince our CI machines will continue to run in our datacenters, we're not likely to use s3. but for anyone running CI in ec2, it would make sense - w
witty-crayon-22786
06/05/2017, 6:36 PM@straight-waiter-86090 : should see whether that is latency or throughput though - b
bright-apartment-76721
06/05/2017, 7:06 PM@enough-analyst-54434 mind if I run a quick sketch of an idea for unifying external binary support by you before I submit a formal issue? - b
bright-apartment-76721
06/05/2017, 7:10 PMor anyone else who knows pants internals for that matter 🙂 - b
bright-apartment-76721
06/05/2017, 7:18 PMbasically, I'd like to abstract over all the binaries that get pulled in by the different subsystems, which all follow roughly the same pattern of registering some options for where to look for it, then calling binary_util.select_binary with that stuff, and instead declare the binary when registering the subsystem - b
bright-apartment-76721
06/05/2017, 7:18 PMand then the underlying machinery takes care of registering the supportdir and similar options for you - b
bright-apartment-76721
06/05/2017, 7:19 PMthis would allow me to provide a more generic "I already have this binary so please don't bother trying to download it" mechanism and the centralization also allows a more reliable "offline" mode - b
bright-apartment-76721
06/05/2017, 7:21 PMas well as reducing fairly mechanical (as far as I've been able to see) code duplication floating around the various subsystems - w
witty-crayon-22786
06/05/2017, 9:15 PM@bright-apartment-76721 : so, to be clear: there is already an abstraction of sorts, in that they all use the
Subsystem to get a handle to the base urls and then fetch things:BinaryUtilCopy codesrc/python/pants//backend/codegen/protobuf/java/protobuf_gen.py:from pants.binaries.binary_util import BinaryUtil src/python/pants//backend/codegen/protobuf/java/protobuf_gen.py: return super(ProtobufGen, cls).subsystem_dependencies() + (BinaryUtil.Factory,) - b
bright-apartment-76721
06/05/2017, 9:15 PMyeah, I saw that - w
witty-crayon-22786
06/05/2017, 9:15 PMand that is what supports fetching from a
base urlfile: - b
bright-apartment-76721
06/05/2017, 9:15 PMbut there's still a lot of duplication around the actual arguments and the fecthing of the binary itself - b
bright-apartment-76721
06/05/2017, 9:16 PMI mostly looked into this because @enough-analyst-54434 said the current abstraction wasn't enough - b
bright-apartment-76721
06/05/2017, 9:16 PMthings like register('--supportdir', advanced=True, default='bin/native-engine', - b
bright-apartment-76721
06/05/2017, 9:16 PMthey all have a register('--supportdir' - b
bright-apartment-76721
06/05/2017, 9:16 PMand all need to remember to do that, etc. - w
witty-crayon-22786
06/05/2017, 9:16 PMsure - b
bright-apartment-76721
06/05/2017, 9:16 PMfurthermore there's no statically compilable list of things that these things might want to fetch - b
bright-apartment-76721
06/05/2017, 9:17 PMwhat I'm saying is basically to take it a step further - w
witty-crayon-22786
06/05/2017, 9:17 PMhm - w
witty-crayon-22786
06/05/2017, 9:19 PMnearby to "statically compilable", you can get a list by doing something like:$ ./pants options | grep '\.supportdir'