also someone mentioned that you can use wasm for approximately equivalent performance to C, but safely in an os kernel because of aforementioned sandboxing, and that kinda blew my mind a bit

#thankyoubasedbazel https://twitter.com/jmmv/status/1091445960524341248?s=21

when i fix up the local process caching PR i’m gonna debut a way to allow dynamic selection of execution strategies during a task run — going to have to involve the concept of split or parallelizable tasks, probably starting out with scalafmt examples although i should ship the graal PR first. this is probably going to be mostly a rust edit. i think this is probably a good way to incrementally move to v2ifying the jvm stack without changing the way any tasks work immediately, just providing tasks the advantage when moving manual

subprocess

invocations into scheduler requests of cacheability and also funky fresh scheduling for parallelizable tasks (and that is the way we can approach moving the zinc/rsc execution graph into v2)

this would also make it easier to test and iterate when working on improving rsc compile performance, but i would ship the embarrassingly parallel scalafmt version first before attacking the dynamic execution graph in zinc compilations

and if i did this then my talk at CGO about using graal native-image for scalafmt perf would be like 300x cooler

also i replied to that tweet because oscar quote tweeted it and i wanted to cunningly try to divert the attention of scala folks, too bad it was yesterday https://twitter.com/posco/status/1091955750568513536?s=21

@aloof-angle-91616 not sure what you mean by split/parallelizable tasks?

"dynamic" scheduling goes by another name... speculative execution. and it would be a great idea to implement speculative execution.

but i think that the nailgun question is still a larger one. speculative execution would make many cases where remote nailgun is necessary unnecessary (because you could race local nailgun against remote no-nailgun). but it's not clear that it would be enough.

I’m pretty skeptical of racing a local super-fast thing with a remote impractically-slow thing… I suspect that in 90% of cases it will end up with you doing everything locally, and repeatedly creating and then cancelling remote things…

oh ok see i just dislike their use of the term "dynamic scheduling" then i just meant choosing execution strategy at runtime

the phrase "speculative execution" comes from (i think) "The Tail at Scale"... to reduce the cost, you generally don't start your second copy of something until it has run in your first choice location for some percentile of the expected runtime

so in this case that might mean "run locally for 500ms, then launch a remote copy to race against"

dark horse here is also local process execution caching -- if we know a process execution is splittable we can cache it at a finer granularity

at the top level: https://pantsbuild.slack.com/archives/C0D7TNJHL/p1549306797028800?thread_ts=1549269379.018000&cid=C0D7TNJHL

Hi folks, I think this old PR is finally ready for review, if you have a moment: https://github.com/pantsbuild/pants/pull/6817 I’m particularly interested in feelings about the logging format.

Also I love our new Travis CI 😄

Could ai get a kind pants commiter to restart this shard please? https://travis-ci.org/pantsbuild/pants/jobs/490201663

It's a known flaky test

ye

Turns out it’s a legitimate failure, repros locally on Pyhon2 and 3

Will investigate for a bit

from a downstream conversation with @average-vr-56795 about lmdb gc with pantsd active (also noting that currently we don't do any gc without pantsd active, which i think i'm fine with)

dwagnerhall [00:53]

FWIW the LMDB garbage collection we do is also kind of crappy because it doesn’t necessarily compact… Not sure of a good strategy for doing some background garbage collection without blocking user action…

dmcclanahan [Today at 00:54]

? is blocking user action necessary for compacting?

9 replies

dwagnerhall [8 hours ago]

Garbage collecting at all requires an exclusive lock on the store, so pants will be unable to do anything

dwagnerhall [8 hours ago]

Compacting requires copying the entries we want to keep to a new directory, deleting the existing one, and renaming over the top, which then requires resetting the Store in any process which has it open, which basically is a pantsd restart if you’re using pantsd

dmcclanahan [8 hours ago]

can we shard with a partition key of some sort to avoid a global store lock? i do not know what renaming is referring to unless you just mean moving the file that keeps the data and then resetting to point to the new file

dwagnerhall [8 hours ago]

We already shard 16 ways, but fundamentally, garbage collecting 1/16 of the store is still going to stop pants from being able to do anything while that 1/16 is locked

dwagnerhall [8 hours ago]

And yes, the renaming is for the backing file that keeps the data

dwagnerhall [8 hours ago]

The lmdb format doesn’t allow for compacting when you delete, so the way we “compact” is by iterating over the entries to decide which ones we want to keep, then copying those to a new dir, and deleting the old one

dmcclanahan [8 hours ago]

i don't immediately have a solution for that

dmcclanahan [6 minutes ago]

also forgot that we shard exactly like that already i remember the ShardedLmdb code was so slick

dmcclanahan [1 minute ago]

ok so i’m confused as to how the method you describe requires blocking pants — are we ever deleting anything from the db except for garbage collection? can we do the copying without blocking and only block on the rename? and it seems intrinsically like there should be a way to reset the store without restarting pantsd, not that i know what i’m talking about...

i'm going to let the

TypedCollection

PR simmer for a bit in case anyone has any more input https://github.com/pantsbuild/pants/pull/7115

also pulled out the suggestions on https://github.com/pantsbuild/pants/pull/7171 into the PR comments and may attack that this weekend: https://pantsbuild.slack.com/archives/C0D7TNJHL/p1548678786134200?thread_ts=1548655292.133600&cid=C0D7TNJHL

also @average-vr-56795 i believe you tweeted about this but i would be extremely comfortable with adding a clippy lint for

_

cases in

match

expressions, i would love to have an excuse to learn how to do that unless there are too many legitimate uses of

_

in the codebase somehow

@aloof-angle-91616 How important are defaults to enums? I’m not sure why they’re implemented…

https://github.com/RustPython/RustPython I haven't looked at all closely, but an interesting project

https://twitter.com/bazelbuild/status/1096134985550098432?s=21 since pants doesn’t cross compile, a lot of this doesn’t directly apply, and i still wish bazel and other tools would start trying to maintain distributions of things like compilers (and compiler flags) instead of letting every user/rule writer figure that out each time (gcc is necessary for graal native-image, for example, and adding that was extraordinarily easy in pants by just adding a subsystem dependency, eliminating a ridiculous amount of work that would have been required to make it remotable). it’s nice that this is a framework to allow something like that, but having a build engineer maintain critical infrastructure like a compiler and linker seems so valuable to me (but bazel also isn’t going to open source their c++ #include scanning for a while either so i’m not holding my breath)

...weren't you involved in writing something like that @average-vr-56795 ?