@witty-crayon-22786 would you expect the memory profile to keep expannding like this? https://gist.github.com/Eric-Arellano/cf886685f990eb5520c1a1d97a56dcc6

i don't know how that tool works

It profiles where memory blocks are.

size

is the average size of a memory block, and count is the # of blocks allocated Should the memory block for the

native.py

handles you just referenced continue to substantially grow both in size and in count throughout the rule, or should any memory be freed up along the way?

handles will grow and shrink, because not all handles end up "held" as Node values in the Graph: some are just temporary

I think the issue is the intermediate memory, though. That more memory keeps getting used as the rule goes on, and with enough concurrent rules, too much memory is used before any gets freed up

...or if you're logging things, inserting a log statement at the end of https://github.com/pantsbuild/pants/blob/fd3ff0c129fae0e75928d1ede69a48c1c98cd9ae/src/python/pants/init/engine_initializer.py#L172-L197

things created in the bodies of

@rules

won’t persist past the end of the run

Agreed. I think the issue is that the rule body takes too much memory.

test

is using 2.42 more megabytes than

list

during the rule just for the memory allocated with `native.py`’s

to_value()

i don't agree. but i don't know how to convince you.

how do you profile what’s in that? You mentioned

__repr__

being in the outputted graph but I didn’t understand how you got to that insight

when i ran

--native-engine-visualize-to

for what should have been a relatively small graph, i could see in py-spy that it was spending a crazy amount of time in

__repr__

(minutes)

Okay I think I agree that one of my hypotheses is proven wrong on memory. I was hypothesizing that if we break up

run_python_test()

into multiple helper functions / rules, the memory usage would decrease because

run_python_test()

would have less variables in its scope that it has to keep in scope throughout the lifetime of the function. Maybe I’ve been reading too much about how Rust works and this doesn’t apply in Python, but was thinking along the lines of a function keeps its local variables on the stack then drops them after If this were true, then I would expect the new

inject_init()

rule to result in the memory usage increasing less than before between the init setup and what comes right before. But, it actually slightly increases from before. I’m a bit at a loss on this memory investigation. Going to take a break and come back in an hour or so. Will reread everything you suggested

sounds good. would look at the content of

self._handles

, one way or the other

would look at the content of

self._handles

, one way or the other

It’s all just

CDataOwnGC

, which wasn’t super helpful. I then had the idea to print the

obj

before it gets converted from CFFI into a

handle

. Wasn’t very helpful, but now I’m thinking to instead use

sys.getsizeof( obj)

, which can show us what’s so huge.

given a handle, you can get the thing it points to

Size of everything sent through

from_value

that’s

> 200

bytes: https://gist.github.com/Eric-Arellano/cf886685f990eb5520c1a1d97a56dcc6#file-size-of-objects-received That’s a whole lot of

datatypes

being passed around. And not sure why we ever materialize files in either direction

i'm suggesting only looking at the handles after running

Also I found that the

__init__.py

rule does make a difference, but I think primarily because it uses Daniel’s new builtin rule to go from

Digest -> Snapshot

so we avoid

Digest -> FIlesContent

ie

objgraph.show_most_common_types(objects=[self.from_value(h) for h in self._handles])

how would you know in

native.py

when it’s over and you should inspect

self._handles

? Maybe put it in

raise_or_return(self, pyresult)

, as I think that gets called near the end

i linked above a spot to insert the call

Number of references after running `./pants list tests/python/pants_test/util:strutil`:

CDataOwn  985
str       60
function  50
partial   9
tuple     6
NoneType  5
ABCMeta   5
generator 4
Digest    3
int       3
None

After `./pants --no-v1 --v2 test tests/python/pants_test/util:strutil`:

CDataOwn  1320
str       182
function  50
int       40
tuple     38
Digest    34
NoneType  32
generator 28
bytes     9
Snapshot  9
None

Hm. CDataOwn is still the handle, right? Did you unwrap it with

from_value

...?

No, the handle is

CDataOwnGC

This is the snippet I added to `run_console_rules()`:

from pants.engine.native import Native
import objgraph
native_context = Native().context    print(objgraph.show_most_common_types(objects=[native_context.from_value(h) for h in native_context._handles]))

... that's unexpected

fyi what’s leftover in

ExternContext._handles

after the run: https://gist.github.com/Eric-Arellano/cf886685f990eb5520c1a1d97a56dcc6#file-leftover-in-handles The file content looks to be the culprit to me again. I ran the same on

./pants list

and found that it too has materialized file content, but the key difference is that it’s

O(1)

for space. For

./pants test

, it’s

O(n)

where n is # of source files in the closure

Interesting.