Ergonomic, hermetic, user-friendly software build system for Python, Go, Java, Kotlin, Protobuf, Scala, Shell. Pants lets you fearlessly scale up your codebase!

Pants

re `Optional[Foo]`: it depends whether that is a wrapper type or a marker type.

i think that it is a marker type, and if so, the Engine doesn't have explicit support for it

We were returning None and not getting an exception

I mean, you don't explicitly unpack an Optional, it's not like a scala `Option`

<https://github.com/pantsbuild/pants/blob/95715d817f1b36d90263bda72ab5dabc7527e164/src/rust/engine/src/nodes.rs#L1076-L1086>

we explode if a function `@rule` returns the wrong type, but not if a generator does

so I guess in this scenario we'd need to have a sentinel value that is of the `Foo` type.

not sure if i'm misunderstanding the use case

as in, i have, this week, when implementing `@rule`s, have converted ones returning `Optional[T]` into `UnionRule`s

especially after <https://github.com/pantsbuild/pants/pull/8542>, but i've been thinking more about that

returning `Optional[T]` is really interesting

i think i would prefer the engine in general to be able to support things that make sense in normal python

so yea: opportunity for a tiny rust patch if you'd like!

or explicit support for `Optional[X]` as a marker, which might be reasonable

I've run into cases where I wish I could easily transfer a type to rust and have it manifest in rust as an Option&lt;T&gt;

a type we grab out of a `Value` in rust with one of the `externs::*` functions, for instance

i'm pretty sure that's already the case, you can check if the value is equal to `externs::none` i think

not sure what the right way to handle this is

it sounds like you might want the opposite thing - a rust Option&lt;T&gt; that shows up in python as None or some value?

Well, I want to be able to use python's Optional[T] as a rule param type

How exactly it's represented in Rust is not something I've given thought to

worst case we can just wrap it in a dataclass with an `Optional[T]` param

it's either a marker or a wrapper. if it's a marker, then it's represented as None.

this statement is wrong, sorry: ~a marker and a wrapper need not be separate~ -- a marker and a wrapper are still separate ideas, even if `@union`s can be used as rule params: <https://github.com/pantsbuild/pants/pull/8542>

In rust, a duration is designed to hide its implementation details from you. Among other things, that means you can’t serialize it.
concrete_time gives you a `Duration` that exposes its implementation details so it can be serialized.
`concrete_time::Duration` implements `From` and `Into` for the standard duration so you’re encouraged to use that in most places and only use the concrete_time one if you need the serialization.

Also, it has a `TimeSpan` since that’s useful for us

The `TimeSpan` also gives you some convenience conversion functions like to the protobuf time representation as these are useful for us.

in the v2 engine, when we're not running pantsd, do we generally expect the `scheduler_execute` FFI function to be run exactly once?

it seems like without the v2 flags `scheduluer_execute` gets called a bunch of times by python code, but with the v2 flags it gets called one time (in my test runs)

which is what you'd expect if `scheduler_execute` is the thing that kicks off the engine's tracing through all the rules that can be executed, but I want to make sure I'm not missing some case where it might get called multiple times in one pants run

you can call scheduler_execute as many times as you'd like

there is exactly one call that represents "the running of `@console_rule`s", but there are lots of calls for other reasons

the context for this question is that I'm trying to figure out where the best place is to add code that can asynchronously poll `WorkUnitStore`

so that we can report build statistics while the build is still going on

rather than polling, i'd suggest changing the recording to eagerly send things somewhere

and I was thinking that spawning a new thread and/or creating a new Future in `Scheduler::execute` is a reasonable place to do that, mostly because it's the place that the loop to write to the display already exists

a middle ground would be putting events in a queue next to where we currently add them

the implementation I have right now makes `WorkUnitStore` have a queue of new workunits

and then I want to have code that just periodically checks for new items in teh queue and pops them off

so I need code that 1) can mutate the `WorkUnitStore` of a `Session` and 2) runs asynchronously from the engine

and if I can do that then I can introduce some interface in python maybe that will hook into that async thread/future and make HTTP rquests or do whatever else somoene wants to do with that data live

if it's in a queue, you can consume the queue from anywhere

if you want the consuming code to be python, you could start a thread on startup of a RunTracker, and join it on shutdown

afaik, (and Benjy/Alex might know more) the RunTracker lives for the length of one pants run

a rough equivalent on the rust side is the Session

I wonder how many v1 assumptions it makes

a `Sesssion` has an inner `WorkUnitStore`

so maybe a session could spawn a thread that polls its own `WorkUnitStore`, and then gets killed in the session's drop method?

Sounds like a reasonable plan :slightly_smiling_face:

so one thing that I'm not sure about, because I'm not as familiar with rust futures as I'd like to be, is what the tradeoffs are between spawning a thread (with `std::thread::spawn`) and spawning a future on the runtime

I'm not 100% sure it's possible to spawn a future in one method, keep around a handle to it, and kill it on drop time, where that future contains an infinite loop

and if you can do that, why pick a future instead of a thread?

what are you planning on doing with the events? some sort of pluggable publish logic?

i ask, because that "what" probably influences the "where"

toolchain wants to use it to hit an http api, but the details of that are toolchain-specific, so we probably want to implement that functionality as a plugin

that can generically get a stream of events from the engine

an alternative would be to make the API public, and have the implementation be specific.

(and maybe in the future zipkin can also be implemented this way, instead of being a special case in the engine)

rust-based plugins are "probably" not happening anytime soon, so i'm guessing that the extension logic would be implemented in python

so I think that implies that it's better not to try to create a public "pants" http API (at least right now) and instead create a generic events stream within pants that arbitrary code can use, including code that hits an API

I think we're fine with doing the http in python

it just needs to be capable of interacting with the engine asynchronously

...then the harness for plugins needs to be able to do various things

but the plugin code may just be a callback for each event.

a callback per-event is probably fine, although I wonder if it's better to allow python to ask for batched events every 500 ms (or whatever time), and have rust do that batching

we don't want to make one http request for every event so we have to buffer events somewhere