https://pantsbuild.org/ logo
Join Slack
Powered by
A huge chunk of the memory in Pantsd is just the d...
# development
f
A huge chunk of the memory in Pantsd is just the dataclass objects that act as nodes in the rule graph. Have we ever tried using 
slots=True
on these to decrease memory consumption?
a
Yes
b
New in version 3.10.
I looked into it once. I think there were a few hotspots. Other than that I think we could on upgrade 🙂
Honestly, we'd likely define our own decorator for dataclasses (or base class), and leverage 
dataclass_transform
so we could set it universally
f
Might be an easy-ish win for larger repos
I'm trying to come up with a list of things that would make working with large repos with Pants more ergonomic, and then consider ranking them along some kinda risk/impact/effort scale
Didn't realize that would take upgrading to 3.10 though. How far away from that are we?
b
Ideally, less than a days worth of work, with 
scie-pants
in the mix. We'd probably jump to 3.11 for the perf bump too.
I need a pet project for my upcoming trip. Maybe I'll poke at it 🙂
f
if you want another pet project, i'm pretty sure porting graph.py to rust would be fun too
b
My last pet project, porting the dep inference to Rust, had me doing compilation on my laptop. Plz not again
😅 1
f
You think that's bad? I started my GC work on GitHub Codespaces
Waiting for a 2 core 4 GB potato node to compile is...not fun 😩
b
🥔
h
Yeah, with scie-pants we can upgrade python at will
b
I think the work will be in whacking all the "moles" where the repo uses a certain IC
h
I'm looking for a pet Rust project, so maybe porting graph.py is it
❤️ 4
f
Maybe even more useful than porting graph.py per say might be porting the smallest rules in 
internals/*.py
to rust... it sucks to have to take on the overhead of acquiring the GIL just to run
Copy code
@rule
def extract_subproject_roots(global_options: GlobalOptions) -> SubprojectRoots:
    return SubprojectRoots(global_options.subproject_roots)
but a flame graph of rules invoked when resolving transitive targets (as mentioned in some other thread) would probably be the ideal way to see what actually gets called most
There might be some benefit in leaving some of the rules in graph.py in Python. One of the ways I think Pants might need to work for larger repos is by minimizing the number of times you need to resolve all dependencies for all targets, and that might take a bit more experimentation to get right, and that experimentation will be harder to do in Rust
f
I believe upgrading an internal Python version will also have a positive impact on plugins authors as they would be able to use 3.10 / 3.11 syntax they are likely already writing in their repository's code; and of course the Pants repo code itself would benefit as well. @bitter-ability-32190 please let me know if you'd like to pair up on internal Python upgrade - it feels a bit too much to do on my own in terms of scope, but would be super happy to work on that together, if you'd like
b
That's very kind Alexey. I think I got it, butt I'll let you know
Happy to share this data from the Pants repo:
Copy code
joshuacannon@JCANNON-WIT:~/work/pants$ /home/joshuacannon/.cache/pants/pants_dev_deps/f83c94980ecbf5333c98482b31a064c7bbe69ab2.venv/bin/mprof peak mprofile_*
mprofile_311_noslots_alldeps.dat        1507.441 MiB
mprofile_311_slots_alldeps.dat  1441.770 MiB
mprofile_39_noslots_alldeps.dat 1516.316 MiB
f
somehow our repo is marginally worse with slots 🤯
Copy code
❯ mprof peak 311*.dat
311-slots_rss.dat	4137.328 MiB
311_rss.dat	4104.812 MiB
🫠 2
b
Are you sure those aren't swapped 😭 ?
f
yeah i'm sure, it's also consistent at least across n=2
Copy code
❯ mprof peak 311*.dat
311-noslots_pss.dat	3948.714 MiB
311-slots_pss.dat	4086.278 MiB
311-slots_rss.dat	4137.328 MiB
311_rss.dat	4104.812 MiB
I used 
psutil_pss
to see if there was a big diff between pss and rss
example command was
Copy code
PANTS_SOURCE=../../pantsbuild/pants mprof run --backend psutil_pss --output 311-noslots_pss.dat pants --no-pantsd dependencies ::
Honestly baffled by this, I thought this would be an easy (if small) win
b
I, too, am baffled
Open in Slack
PreviousNext
Powered by