https://pantsbuild.org/ logo
#development
Title
# development
h

hundreds-father-404

05/15/2019, 8:46 PM
@witty-crayon-22786 would you expect the memory profile to keep expannding like this? https://gist.github.com/Eric-Arellano/cf886685f990eb5520c1a1d97a56dcc6
w

witty-crayon-22786

05/15/2019, 8:46 PM
i don't know how that tool works
so not sure.
h

hundreds-father-404

05/15/2019, 8:48 PM
It profiles where memory blocks are.
size
is the average size of a memory block, and count is the # of blocks allocated Should the memory block for the
native.py
handles you just referenced continue to substantially grow both in size and in count throughout the rule, or should any memory be freed up along the way?
w

witty-crayon-22786

05/15/2019, 8:50 PM
handles will grow and shrink, because not all handles end up "held" as Node values in the Graph: some are just temporary
so while the intermediate memory usage is interesting, you probably want to be looking at the final memory usage
one way to do that is to
--enable-pantsd
, and look at the memory usage of the process after running things
h

hundreds-father-404

05/15/2019, 8:52 PM
I think the issue is the intermediate memory, though. That more memory keeps getting used as the rule goes on, and with enough concurrent rules, too much memory is used before any gets freed up
w

witty-crayon-22786

05/15/2019, 8:53 PM
@hundreds-father-404: i don't think so.
the graph memoizes things, intentionally.
so it's very, very likely that if the wrong things are ended up memoized, that that is the reason
things created in the bodies of
@rules
won't persist past the end of the run
but if you run with
--enable-pantsd
, you'll see that this persists.
h

hundreds-father-404

05/15/2019, 8:58 PM
things created in the bodies of
@rules
won’t persist past the end of the run
Agreed. I think the issue is that the rule body takes too much memory.
test
is using 2.42 more megabytes than
list
during the rule just for the memory allocated with `native.py`’s
to_value()
w

witty-crayon-22786

05/15/2019, 8:59 PM
i don't agree. but i don't know how to convince you.
so see what tools are available to experiment and see!
...my point about what persists being more important is that if you fix what persists after the run, then you will likely fix as a sideeffect anything created during it.
👍 1
and what persists is very easy to see... it's the stuff in the
handles
map.
h

hundreds-father-404

05/15/2019, 9:01 PM
how do you profile what’s in that? You mentioned
__repr__
being in the outputted graph but I didn’t understand how you got to that insight
w

witty-crayon-22786

05/15/2019, 9:02 PM
when i ran
--native-engine-visualize-to
for what should have been a relatively small graph, i could see in py-spy that it was spending a crazy amount of time in
__repr__
(minutes)
the visualization of the Graph is approximately the
repr
of everything that it holds (and in a stable state, should be ~equal to the content of the
_handles
map)
so, maybe either 1) figuring out what is huge by rendering the graph and spotting it, or 2) "analyzing" the handles to see what is heavy
Patrick previously used
objgraph
... possible that running an
objgraph
analysis on the handles map to find "large" things would work.
maybe: https://mg.pov.lt/objgraph/#reference-counting-bugs
objgraph.show_most_common_types(objects=self._handles)
...?
👍 1
(not because i suspect a reference counting bug: but just to get a summary of what we're holding onto)
gonna log off for a bit
h

hundreds-father-404

05/16/2019, 12:36 AM
Okay I think I agree that one of my hypotheses is proven wrong on memory. I was hypothesizing that if we break up
run_python_test()
into multiple helper functions / rules, the memory usage would decrease because
run_python_test()
would have less variables in its scope that it has to keep in scope throughout the lifetime of the function. Maybe I’ve been reading too much about how Rust works and this doesn’t apply in Python, but was thinking along the lines of a function keeps its local variables on the stack then drops them after If this were true, then I would expect the new
inject_init()
rule to result in the memory usage increasing less than before between the init setup and what comes right before. But, it actually slightly increases from before. I’m a bit at a loss on this memory investigation. Going to take a break and come back in an hour or so. Will reread everything you suggested
w

witty-crayon-22786

05/16/2019, 12:45 AM
sounds good. would look at the content of
self._handles
, one way or the other
during a run, the coroutines/generators are themselves inside the handles map as well as the produced values... after the run, they'll be gone, and it should contain only the produced values that are held in the Graph
(and
objgraph
should show that)
h

hundreds-father-404

05/16/2019, 12:48 AM
would look at the content of
self._handles
, one way or the other
It’s all just
CDataOwnGC
, which wasn’t super helpful. I then had the idea to print the
obj
before it gets converted from CFFI into a
handle
. Wasn’t very helpful, but now I’m thinking to instead use
sys.getsizeof( obj)
, which can show us what’s so huge.
By far the biggest objects are text files being materialized and the captured stdout. Gist of everything passed through `native.py`’s
to_value()
that’s larger than 100 bytes: https://gist.github.com/Eric-Arellano/cf886685f990eb5520c1a1d97a56dcc6#file-size-of-objects-send Is this expected? We are materializing more now that we resolve all transitive deps
w

witty-crayon-22786

05/16/2019, 1:04 AM
given a handle, you can get the thing it points to
...so i think before calling objgraph, could look-up all of the handles
hm... that link shows a bunch of 100-200 byte objects... is that what you meant to link?
...oh, and the file content of the tests themselves... that's sortof unexpected.
👍 1
not a smoking gun yet, because i think that even if we pulled all of the transitive code in, it still wouldn't add up to that much.
but yea
would recommend looking at the objects in the handles map itself after a run:
context.from_value
does the opposite of
to_value
👍 1
h

hundreds-father-404

05/16/2019, 1:20 AM
Size of everything sent through
from_value
that’s
> 200
bytes: https://gist.github.com/Eric-Arellano/cf886685f990eb5520c1a1d97a56dcc6#file-size-of-objects-received That’s a whole lot of
datatypes
being passed around. And not sure why we ever materialize files in either direction
w

witty-crayon-22786

05/16/2019, 1:21 AM
i'm suggesting only looking at the handles after running
👍 1
h

hundreds-father-404

05/16/2019, 1:21 AM
Also I found that the
__init__.py
rule does make a difference, but I think primarily because it uses Daniel’s new builtin rule to go from
Digest -> Snapshot
so we avoid
Digest -> FIlesContent
w

witty-crayon-22786

05/16/2019, 1:21 AM
ie
objgraph.show_most_common_types(objects=[self.from_value(h) for h in self._handles])
👍 1
or whatever other analysis
mm, yea: avoiding the FileContent is significant, for sure.
h

hundreds-father-404

05/16/2019, 1:23 AM
how would you know in
native.py
when it’s over and you should inspect
self._handles
? Maybe put it in
raise_or_return(self, pyresult)
, as I think that gets called near the end
w

witty-crayon-22786

05/16/2019, 1:26 AM
i linked above a spot to insert the call
(can just reach in there and grab it, afaik)
h

hundreds-father-404

05/16/2019, 2:16 AM
Number of references after running `./pants list tests/python/pants_test/util:strutil`:
Copy code
CDataOwn  985
str       60
function  50
partial   9
tuple     6
NoneType  5
ABCMeta   5
generator 4
Digest    3
int       3
None
After `./pants --no-v1 --v2 test tests/python/pants_test/util:strutil`:
Copy code
CDataOwn  1320
str       182
function  50
int       40
tuple     38
Digest    34
NoneType  32
generator 28
bytes     9
Snapshot  9
None
w

witty-crayon-22786

05/16/2019, 2:20 AM
Hm. CDataOwn is still the handle, right? Did you unwrap it with
from_value
...?
h

hundreds-father-404

05/16/2019, 2:21 AM
No, the handle is
CDataOwnGC
This is the snippet I added to `run_console_rules()`:
Copy code
from pants.engine.native import Native
import objgraph
native_context = Native().context    print(objgraph.show_most_common_types(objects=[native_context.from_value(h) for h in native_context._handles]))
w

witty-crayon-22786

05/16/2019, 2:22 AM
... that's unexpected
Afaik,
type(from_value(...))
should ~always match one of the Params or return values of a @rule
But you might consider using one of the other objgraph analysis methods... there are a lot
h

hundreds-father-404

05/16/2019, 2:25 AM
fyi what’s leftover in
ExternContext._handles
after the run: https://gist.github.com/Eric-Arellano/cf886685f990eb5520c1a1d97a56dcc6#file-leftover-in-handles The file content looks to be the culprit to me again. I ran the same on
./pants list
and found that it too has materialized file content, but the key difference is that it’s
O(1)
for space. For
./pants test
, it’s
O(n)
where n is # of source files in the closure
w

witty-crayon-22786

05/16/2019, 2:27 AM
Interesting.
I think you're looking at the right data now... maybe try scaling it up slowly and seeing if the distribution changes? Run with 1, then 2, etc?
💯 1