I’m looking at tackling invalidation of options re...
# development
h
I’m looking at tackling invalidation of options read from
@fromfile
. See https://github.com/pantsbuild/pants/issues/10360 for context. (thread)
Options parsing (and fromfile handling) is now entirely in Rust. So I’m not sure how to delegate reading the @fromfile to the scheduler.
One way is to pass references to Python’s underlying Scheduler and Session through to the native OptionsParser. But that means that the
options
crate would have to depend on the top-level engine crate, which in turn depends on basically everything, so I’m not sure that’s even possible.
I suppose it could be hidden behind an impl of a
FromFileReader
trait of some kind
But even then, can one use a Scheduler/Session to call an intrinsic directly in Rust?
(Reason to do so is to get the filewatching and subsequent invalidation of the fromfile)
Alternatively, now that options parsing is in Rust, and fast, maybe we don’t memoize options values via Python
memoized_property
and just recompute every time we read options?
Anyway, thoughts? Particularly from @witty-crayon-22786 ?
f
Maybe we need some sort of "invalidatable value" abstraction exposed from Rust to Python? It could be something like the "watch" channel in Tokio. https://docs.rs/tokio/latest/tokio/sync/watch/index.html
instead of
@memoized_property
, you would
.flat_map
(or
.and_then
for Rust name for it) the value to derive a new value
and depending on the context, the engine could have some way of knowing when one of the values changed
w
@happy-kitchen-89482: I like your approach, but to avoid a cycle, you would add a trait like "FileProvider" or something, with the shape of the method you were passing in
and then, yea: absolute file intrinsic
h
Thanks @witty-crayon-22786. That's what I figured re a trait to break the cycle, but I'm still not sure if it's straightforward to call an intrinsic on a scheduler directly in Rust ? I'll dig into it though
w
You can: see for example how the
watch
crate invalidates graph nodes via an interface
specifically: i think that you'll create the instance of the interface using the
Context
which is passed to some separate options parsing intrinsic
f
@happy-kitchen-89482: My (now abandoned) attempt to add engine invalidation for system paths touches on some of those APIs and might be useful for general context. https://github.com/pantsbuild/pants/pull/21233