Hm, for

--pantsd-max-memory-usage

, two questions: * We default to 1 GiB - instead, should we detect total size of memory on machine and change default dynamically? * the option takes bytes, rather than kb. That makes it harder to reason about, and

ulimit

uses kb. I think we should prob even be using mb

or have the option still be bytes, but accept a string with a unit like (M or G)

Oh, good idea! Thanks Tom! Now to see if we can get that playing nicely with TOML

how much should be set aside for running tooling?

Unclear what the max should be, but I'd anticipate from 0.5-2gb Something that complicates this is I believe

--pantsd-max-memory-usage

includes memory used for the current run + memory from prior memoization

maybe the memoization cache max size should have its own tuning option?

I think that would be ideal, although I'm not sure how technically feasible it is. Right now, we use

psutil

to inspect the Pantsd process's memory consumption - I don't know how to get more granular than that

is the memorization cache a single data structure?

Iiuc, it makes no distinguishment between memoization from the current run vs prior runs. Note that

--no-pantsd

does memoize within the current run, only the process dies after the run completes. All Pantsd does is keeps that process long-living

but it is a single data structure? i.e., the code could keep a sum of the size of all nodes and apply LRU to discard ones that go over the configured max memory size

but it is [pantsd memoization] a single data structure?

hm @average-vr-56795, do you know the answer to that?

Yes and no... It's a one of these: https://github.com/pantsbuild/pants/blob/aa8e55c6ef610fa1ca1ac5fd8dc40fcb2d278ad0/src/rust/engine/graph/src/lib.rs#L69-L72 which fundamentally ends up being a

HashMap

of `NodeKey`s (https://github.com/pantsbuild/pants/blob/aa8e55c6ef610fa1ca1ac5fd8dc40fcb2d278ad0/src/rust/engine/src/nodes.rs#L1142-L1154) and a bunch of graph metadata. But there's a lot of structure to the contents (because it's a graph, and nodes refer to each other), so we couldn't just pop out the most recently used ones (well, we could, but we should probably do so through a dependency-aware traversal if we wanted to)... There's also an element of versioning in each entry...

Hm, that sounds very not-in-scope here. Thank you for explaining that! I'm thinking our guidance should maybe be something then like: - if 2GiB or less, disable Pantsd - between 2gib - 8gib, either a step-wise function or a percentage of total ram that we compute? - >8gib, default to 1.5 or 2gib This feels hand-wavy, I don't know how to choose sensible defaults here

It would be really interesting to know what the things using all the memory are - there are potentially optimisations we could make around that

agreed, @fast-nail-55400 is looking into profiling. Perhaps I should wait to tweak defaults until we understand that

For instance if process execution results are taking the space, we could replace process execution nodes with digests that point into the local process execution cache, so that loading them is slightly slower (we need to hit an LMDB), but they take a trivial amount of memory

I did a memory profile yesterday. It looked like the winner for most allocations was engine::Value

Sounds good - happy to take a look if you have things to share!

I think

engine::Value

is an

Arc

around any FFI value, so it's not particularly clear what that refers to. Could be for example a

FrozenDict

representing

env

///
/// We wrap PyObject (which cannot be cloned without acquiring the GIL) in an Arc in order to avoid
/// accessing the Gil in many cases.
///
#[derive(Clone)]
pub struct Value(Arc<PyObject>);

I was going to look into if it makes sense to remove that abstraction with PyO3, which claims better support for references of Python objects, whereas Rust-CPython takes ownershi