https://pantsbuild.org/ logo
#development
Title
# development
h

hundreds-father-404

05/24/2021, 11:46 PM
Hm, for
--pantsd-max-memory-usage
, two questions: * We default to 1 GiB - instead, should we detect total size of memory on machine and change default dynamically? * the option takes bytes, rather than kb. That makes it harder to reason about, and
ulimit
uses kb. I think we should prob even be using mb
Thoughts on this both? For #2, we would need to come up with a new option name 🤔
f

fast-nail-55400

05/25/2021, 12:02 AM
or have the option still be bytes, but accept a string with a unit like (M or G)
so
--pants-max-memory-usage=512MB
for example
h

hundreds-father-404

05/25/2021, 12:04 AM
Oh, good idea! Thanks Tom! Now to see if we can get that playing nicely with TOML
I'm thinking we deprecate using only a number and expect it to have a suffix of
{gb,mb,kb,b}
(capitalization doesn't matter)
although lol confusion on binary vs decimal https://en.wikipedia.org/wiki/Gigabyte
@polite-garden-50641 looking at dev ops config, looks like best practice is to use GiB, MiB, KiB, and B?
For the default, any opinion on a stepwise function vs % of total memory? Strawman numbers: - <= 2gb => 0.25gb for Pantsd - <= 4gb => 0.5gb - 4 < m <= 8 => 1gb - 8+ => 1.5gb
f

fast-nail-55400

05/26/2021, 1:53 PM
how much should be set aside for running tooling?
h

hundreds-father-404

05/27/2021, 5:12 PM
Unclear what the max should be, but I'd anticipate from 0.5-2gb Something that complicates this is I believe
--pantsd-max-memory-usage
includes memory used for the current run + memory from prior memoization
f

fast-nail-55400

05/27/2021, 5:22 PM
maybe the memoization cache max size should have its own tuning option?
h

hundreds-father-404

05/27/2021, 5:26 PM
I think that would be ideal, although I'm not sure how technically feasible it is. Right now, we use
psutil
to inspect the Pantsd process's memory consumption - I don't know how to get more granular than that
f

fast-nail-55400

05/27/2021, 5:27 PM
is the memorization cache a single data structure?
h

hundreds-father-404

05/27/2021, 5:28 PM
Iiuc, it makes no distinguishment between memoization from the current run vs prior runs. Note that
--no-pantsd
does memoize within the current run, only the process dies after the run completes. All Pantsd does is keeps that process long-living
f

fast-nail-55400

05/27/2021, 5:29 PM
but it is a single data structure? i.e., the code could keep a sum of the size of all nodes and apply LRU to discard ones that go over the configured max memory size
h

hundreds-father-404

05/27/2021, 5:30 PM
but it is [pantsd memoization] a single data structure?
hm @average-vr-56795, do you know the answer to that?
a

average-vr-56795

05/27/2021, 6:41 PM
Yes and no... It's a one of these: https://github.com/pantsbuild/pants/blob/aa8e55c6ef610fa1ca1ac5fd8dc40fcb2d278ad0/src/rust/engine/graph/src/lib.rs#L69-L72 which fundamentally ends up being a
HashMap
of `NodeKey`s (https://github.com/pantsbuild/pants/blob/aa8e55c6ef610fa1ca1ac5fd8dc40fcb2d278ad0/src/rust/engine/src/nodes.rs#L1142-L1154) and a bunch of graph metadata. But there's a lot of structure to the contents (because it's a graph, and nodes refer to each other), so we couldn't just pop out the most recently used ones (well, we could, but we should probably do so through a dependency-aware traversal if we wanted to)... There's also an element of versioning in each entry...
👍 1
So - we could garbage-collect the graph, but it would be more of a "work out some heuristics for a graph traversal to work out what to pop" kind of problem than a "keep track of which ones are least recently used" kind of problem
(In particular, I'm pretty sure if we do want to pop things, we need to do so in a topologically sorted order, top-down, because the way the graph structure we use works, it uses "index in a list" to refer to nodes, and if we remove things which are depended on we may end up with dangling references to things which don't exist, and I think the way that may manifest itself is they may sometimes become references to other things in the list)
h

hundreds-father-404

05/27/2021, 6:44 PM
Hm, that sounds very not-in-scope here. Thank you for explaining that! I'm thinking our guidance should maybe be something then like: - if 2GiB or less, disable Pantsd - between 2gib - 8gib, either a step-wise function or a percentage of total ram that we compute? - >8gib, default to 1.5 or 2gib This feels hand-wavy, I don't know how to choose sensible defaults here
a

average-vr-56795

05/27/2021, 6:45 PM
It would be really interesting to know what the things using all the memory are - there are potentially optimisations we could make around that
e.g. right now we don't store file contents in the graph, we shove them into an LMDB and store their digest in the graph
h

hundreds-father-404

05/27/2021, 6:46 PM
agreed, @fast-nail-55400 is looking into profiling. Perhaps I should wait to tweak defaults until we understand that
a

average-vr-56795

05/27/2021, 6:46 PM
For instance if process execution results are taking the space, we could replace process execution nodes with digests that point into the local process execution cache, so that loading them is slightly slower (we need to hit an LMDB), but they take a trivial amount of memory
1
To guide that kind of optimisation, knowing how much memory we're using for each of these types: https://github.com/pantsbuild/pants/blob/aa8e55c6ef610fa1ca1ac5fd8dc40fcb2d278ad0/src/rust/engine/src/nodes.rs#L1142-L1154 would be pretty interesting (though I suspect none of them are taking up particularly much space)
💯 1
f

fast-nail-55400

05/27/2021, 6:47 PM
I did a memory profile yesterday. It looked like the winner for most allocations was engine::Value
but that is off one run, will have some more info later today when I write up how to run memory profile and stuff
a

average-vr-56795

05/27/2021, 6:48 PM
Sounds good - happy to take a look if you have things to share!
My understanding is that an
engine::Value
is just a pointer, right? It points to some memory owned by the Python runtime?
h

hundreds-father-404

05/27/2021, 6:55 PM
I think
engine::Value
is an
Arc
around any FFI value, so it's not particularly clear what that refers to. Could be for example a
FrozenDict
representing
env
Copy code
///
/// We wrap PyObject (which cannot be cloned without acquiring the GIL) in an Arc in order to avoid
/// accessing the Gil in many cases.
///
#[derive(Clone)]
pub struct Value(Arc<PyObject>);
I was going to look into if it makes sense to remove that abstraction with PyO3, which claims better support for references of Python objects, whereas Rust-CPython takes ownershi