https://pantsbuild.org/ logo
Join Slack
Powered by
out of curiosity, how long does `pants --no-pantsd...
# development
f
out of curiosity, how long does 
pants --no-pantsd --no-local-cache --filter-target-type=python_source,python_test dependencies ::
take for you all on the Pants repo itself?
👀 2
b
Should this also be run with 
--no-pantsd
? Also Maybe recommend 
--filter-target-type=python_source
?
f
switching 
--no-local-cache
on and off forces pantsd restarts anyways
b
Yes, but If I run that twice... Probably good to be explicit
f
updated
✅ 1
for me it's around 1m20s - 1m30s
b
So to be clear, this measuring how long a cold-cache run takes, considering you'll be spawning a process-per-file for dependency inference.
f
Yes exactly
c
Made a few runs, seems pretty stable around this time:
Copy code
❯ time pants --no-pantsd --no-local-cache --filter-target-type=python_source,python_test dependencies :: >/dev/null
11:47:18.92 [INFO] Starting: Resolving plugins: hdrhistogram, toolchain.pants.plugin==0.27.0
11:47:22.87 [INFO] Completed: Resolving plugins: hdrhistogram, toolchain.pants.plugin==0.27.0
pants --no-pantsd --no-local-cache  dependencies :: > /dev/null  59.24s user 63.71s system 268% cpu 45.780 total

zsh  ~/src/github/kaos/pants    main $  11:48 
❯ git last
commit 5afb9a01173baf5b25acf4bc1517754abdf76b7f (HEAD -> main, upstream/main, origin/main, origin/HEAD)
f
as a naive comparison
Copy code
time for f in $(cat python-files.txt); PYTHONPATH=src/python python src/python/pants/backend/python/dependency_inference/scripts/main.py $f; end
only takes 50s. I realize that this does strictly less than pants dep inference but it's also the kinda comparison that I'm going to get from cranky devs
b
Copy code
real    1m6.371s
user    1m5.151s
sys     1m51.295s
w
as mentioned on the ticket, about 50% of the startup time in the 
pantsbuild/pants
repo is rule graph solving, because we have “all” the rules enabled
b
Solving or parsing (or a mix of the two)?
w
solving.
b
😕
w
which is the key motivation for https://github.com/pantsbuild/pants/discussions/18905, honestly. i don’t want to revisit that algorithm until we have a way to significantly simplify it, and that would do it.
f
okay thanks... this is reassuring to me that you're aware of these issues and working on them
I think the answer for us may to just raise the pantsd memory limit and live with that. We still don't use Pants to run tests so we can just 
pkill pantsd
before tests to clear up space.
b
Reminder that "you" is "us" 😉
f
Idk until I actually get down and start helping here I feel it’s more of a formality that I’m part of this in the same way as some more active contributors are
What could I do to help?
b
How's your Rust? 😅
f
I’ve read the book, that’s about it
Maybe I could learn while trying to tackle one of these issues
If Pantsd is so critical to perf, then maybe finding a way to optimize what it actually caches would be an interesting project
b
FWIW I've only learned rust through changes in this project 🙂
f
Yeah seems like the right way to learn
Porting target or field set to native sounds at least self contained
(Maybe not if the code calls a bunch of other Python code)
w
yea… 90% of the challenge with Python->Rust porting is finding the right boundary to port. because you want to release the GIL for long periods, and so if you have to re-acquire it from Rust, then you lose some of the benefit (not all of it though, because “Rust with the GIL acquired” is still faster than “Python with the GIL acquired”
f
I mean is there a good place to start here? I’d be interested in diving in, particularly with making pantsd smarter about what it holds in memory, but that sounds like a tall order
I’ve implemented some LRU stuff in other languages. Is there a practical reason why pantsd doesn’t do LRU but just dies when it outgrows it’s memory limit?
If I could be pointed in the right direction here I’d take a look
w
no, no practical reason! see https://github.com/pantsbuild/pants/issues/7675 about doing that.
i’d be happy to write up some details about how to implement it, if you’re interested. it would be about 95% Rust, but i’m happy to overexplain
f
Yeah please do. I’ve never done anything significant in Rust but this seems like a good as place as any to dive in
w
k: will do before the end of the day. thanks!
f
You can explain on that issue, I’m subscribed now (probably better there than in slack)
👍 1
w
here you go: https://github.com/pantsbuild/pants/issues/7675#issuecomment-1537037248 : feel free to ping any of us with questions, but no pressure obviously: if you don’t have time that’s fine!
b
(your bullet point number two uses 
./pants check
when it should be 
./cargo check
Actually everywhere you use 
./pants
should be 
./cargo
😂
😓 1
w
fixed! thanks.
b
That's just years of muscle memory 😉
Open in Slack
PreviousNext
Powered by