and also, the typed datatype collections PR (https://github.com/pantsbuild/pants/pull/7115) doesn't do any engine stuff and is green so that would be cool to start on if anyone (not just the specific reviewers I asked) has time to review (unless you want to dive into the union types one, which would also be very cool and great). no rush on any of this, questions are encouraged

i think i'm gonna swing back to the rule graph viz one later today https://github.com/pantsbuild/pants/pull/7024

ok i'm actually definitely going to look at persistent subprocesses in v2 a la https://github.com/pantsbuild/pants/issues/7079 because that is immediately important (to me)

i just accidentally pinged you stu and deleted the message but i'm still working on this just fine for now sorry

got it

i forgot to say i’ve been cooking up a scalafmt console_rule which is invoked using graal native-image and the only things missing are getting classpath entries for jvm tools (going to use a tiny coursier wrapper for that) and capturing a snapshot for the graal binary (haven’t thought about that yet, building fs_util right now to work around it). i don’t understand what hermetic zinc compile is doing with its jdk_home so am going to hack around that too, ideally i want to be running a

yield [Get(FallibleExecuteProcessResult, ...)]

with a specified degree of parallelism over all input files (as in the native image PR i currently have up) in a couple hours

also is there a reason we're formatting

bazel_protos

in its

<http://build.rs|build.rs>

?

looks like

ExecuteProcessRequest

has issues with symlinked executables

also

env={'PATH': './bin'}

isn't working for some reason although relative

PATH

entries work on the command line

(or i might have uploaded the wrong snapshot, might be more likely)

working

> ./pants --v2 --no-v1 lint.scalafmt --jvm-options="[]" src/scala::
Looking for unformatted files...
Exception in thread "main" org.scalafmt.Error$MisformattedFile: /home/cosmicexplorer/tools/pants/src/scala/org/pantsbuild/zinc/extractor/Extractor.scala is mis-formatted. 
	at java.lang.Throwable.<init>(Throwable.java:265)
	at java.lang.Exception.<init>(Exception.java:66)
	at org.scalafmt.Error.<init>(Error.scala:15)
	at org.scalafmt.Error$MisformattedFile.<init>(Error.scala:66)
	at org.scalafmt.cli.InputMethod$FileContents.write(InputMethod.scala:51)
	at org.scalafmt.cli.Cli$.org$scalafmt$cli$Cli$$handleFile(Cli.scala:121)
	at org.scalafmt.cli.Cli$$anonfun$run$1.apply(Cli.scala:180)
	at org.scalafmt.cli.Cli$$anonfun$run$1.apply(Cli.scala:177)
	at scala.collection.Iterator$class.foreach(Iterator.scala:891)
	at scala.collection.AbstractIterator.foreach(Iterator.scala:1334)
	at scala.collection.parallel.ParIterableLike$Foreach.leaf(ParIterableLike.scala:972)
	at scala.collection.parallel.Task$$anonfun$tryLeaf$1.apply$mcV$sp(Tasks.scala:49)
	at scala.collection.parallel.Task$$anonfun$tryLeaf$1.apply(Tasks.scala:48)
scala.collection.parallel.AdaptiveWorkStealingForkJoinTasks$WrappedTask.compute(Tasks.scala:443)
        ... many more lines
	at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
	at com.oracle.svm.core.thread.JavaThreads.threadStartRoutine(JavaThreads.java:480)
	at com.oracle.svm.core.posix.thread.PosixJavaThreads.pthreadStartRoutine(PosixJavaThreads.java:198)

Compilation exited abnormally with code 1 at Mon Jan 21 20:20:41

going to play around a bit with why

native-image

can't find my gcc that i very kindly placed on its PATH (it's only working when i provide the absolute path to the gcc install)

i also don't know how to cache the result of a process execution (e.g. a generated image) in a persistent way. since this already involves pulling in

PathGlobsAndRoot

to

hackily_snapshot()

the graal

NativeTool

(which still can't be done from an

@console_rule

because it requires a

context._scheduler

) i feel like an "inverse" (ish) which writes a specific file or directory to a specified subdir of

get_pants_cachedir()

and then checks if it exists on successive runs could also be a fantastic and easy and fun way to bridge v1 and v2 tasks. i just thought of this and i think it's a great idea that will make all of this a very small amount of code. looking into that now

and i need to make the coursier resolve rule that'd be so slick

also,

optionable_rule

is fantastic

@average-vr-56795 i developed the idea about teeing workunit output in a structured way that i was mentioning earlier today a little more: https://github.com/pantsbuild/pants/issues/7071#issuecomment-456303950. @witty-crayon-22786 this is relevant to people who want to get output out of pants for ci (as per discussion last week). @red-balloon-89377 would also love to hear your thoughts on the option described in that comment. i also think this (the

--workunit-stdout

dict option described in the comment) wouldn't take that long to implement, and after the

@union_rule

stuff goes through and i synthesize all the thoughts above about storing rule output persistently (such as native images, in

~/.cache/pants

, so we don't keep reproducing them) i think the only remaining blocker to using v2 as a first-class citizen is developing the output story ("obviously"), and i think itemizing workunit output like this fits in really well with the whole v2/pantsd idea of "pants is always running, and you can request big or small things from it, and the tokio executor/etc makes things parallel" instead of "big monolithic pants process that you have to wait around to complete"

also it should go without saying i think zipkin tracing, profiling, and logging are deeply integrated, and thinking about being able to jump from a zipkin trace of a workunit to the log corresponding to it is kind of what inspired this thought in the first place (i'll put that in the ticket too)

had an EVEN BETTER IDEA in https://github.com/pantsbuild/pants/issues/7071#issuecomment-456309997 which can be summarized as "tee the stdout from zinc compiles to a file named `output.txt`":

> ./pants --compile-zinc-workunit-output="+{'stdout': {'output.txt': ['COMPILE']}}" test ::

and the

stdout

corresponds to

workunit.output('stdout')

, but we can extend that to other output types to allow for non-hierarchical output labelling for an asynchronous v2 UI!

V2 related python 3 failure that is causing a target to have to be on blacklist, if anyone has time to look https://github.com/pantsbuild/pants/issues/7131. I’m not sure where to begin

if anyone has thoughts on how to make a

SimpleCodegenTask

type of thing in v2 (we may not even need a specific wrapper) lmk https://github.com/pantsbuild/pants/issues/7137

@aloof-angle-91616 I haven’t read enough to know yet, but… How much simpler would https://github.com/pantsbuild/pants/pull/7115 be if we forced API consumers to make things the right type to start with, rather than supporting

wrapper_type

?

there's a webassembly meetup at mozilla right now that i am about to go to if anyone wants to swing by

6:30-9pm

you all should definitely have been there and i definitely should have invited you earlier

i have lots of ideas about stuff we’re doing right now i might bounce some of them to even more and/or less code

left a comment with a specific thought on (one of) the grand unified abstraction(s) for v1 and v2 tasks to allow using the local filesystem e.g. pants cachedir as the local cache for specific process executions, too tired to flesh it out all the way, will tomorrow https://github.com/pantsbuild/pants/issues/6898#issuecomment-457524192

update: i'm doing @average-vr-56795's very detailed plan on #6898 instead of whatever i was thinking, thanks a lot for that

having a separate database for it "obviously" addresses the concern i had in my head about accidentally sharing local resources from one host to another without having to figure out platforms yet so that's great

....really liking the

CommandRunner

proposal though

mostly because factoring out the common code necessary to stick it into

fs

seems like an adventure we don't need to have, still looking