https://pantsbuild.org/ logo
#development
Title
# development
f

flat-zoo-31952

05/05/2023, 3:15 PM
out of curiosity, how long does
pants --no-pantsd --no-local-cache --filter-target-type=python_source,python_test dependencies ::
take for you all on the Pants repo itself?
👀 2
b

bitter-ability-32190

05/05/2023, 3:15 PM
Should this also be run with
--no-pantsd
? Also Maybe recommend
--filter-target-type=python_source
?
f

flat-zoo-31952

05/05/2023, 3:16 PM
switching
--no-local-cache
on and off forces pantsd restarts anyways
b

bitter-ability-32190

05/05/2023, 3:17 PM
Yes, but If I run that twice... Probably good to be explicit
f

flat-zoo-31952

05/05/2023, 3:18 PM
updated
1
for me it's around 1m20s - 1m30s
b

bitter-ability-32190

05/05/2023, 3:25 PM
So to be clear, this measuring how long a cold-cache run takes, considering you'll be spawning a process-per-file for dependency inference.
f

flat-zoo-31952

05/05/2023, 3:25 PM
Yes exactly
c

curved-television-6568

05/05/2023, 3:49 PM
Made a few runs, seems pretty stable around this time:
Copy code
❯ time pants --no-pantsd --no-local-cache --filter-target-type=python_source,python_test dependencies :: >/dev/null
11:47:18.92 [INFO] Starting: Resolving plugins: hdrhistogram, toolchain.pants.plugin==0.27.0
11:47:22.87 [INFO] Completed: Resolving plugins: hdrhistogram, toolchain.pants.plugin==0.27.0
pants --no-pantsd --no-local-cache  dependencies :: > /dev/null  59.24s user 63.71s system 268% cpu 45.780 total

zsh  ~/src/github/kaos/pants    main $  11:48 
❯ git last
commit 5afb9a01173baf5b25acf4bc1517754abdf76b7f (HEAD -> main, upstream/main, origin/main, origin/HEAD)
f

flat-zoo-31952

05/05/2023, 5:01 PM
as a naive comparison
Copy code
time for f in $(cat python-files.txt); PYTHONPATH=src/python python src/python/pants/backend/python/dependency_inference/scripts/main.py $f; end
only takes 50s. I realize that this does strictly less than pants dep inference but it's also the kinda comparison that I'm going to get from cranky devs
b

bitter-ability-32190

05/05/2023, 5:23 PM
Copy code
real    1m6.371s
user    1m5.151s
sys     1m51.295s
w

witty-crayon-22786

05/05/2023, 6:30 PM
as mentioned on the ticket, about 50% of the startup time in the
pantsbuild/pants
repo is rule graph solving, because we have “all” the rules enabled
b

bitter-ability-32190

05/05/2023, 6:34 PM
Solving or parsing (or a mix of the two)?
w

witty-crayon-22786

05/05/2023, 6:34 PM
solving.
b

bitter-ability-32190

05/05/2023, 6:34 PM
😕
w

witty-crayon-22786

05/05/2023, 6:35 PM
which is the key motivation for https://github.com/pantsbuild/pants/discussions/18905, honestly. i don’t want to revisit that algorithm until we have a way to significantly simplify it, and that would do it.
f

flat-zoo-31952

05/05/2023, 7:01 PM
okay thanks... this is reassuring to me that you're aware of these issues and working on them
I think the answer for us may to just raise the pantsd memory limit and live with that. We still don't use Pants to run tests so we can just
pkill pantsd
before tests to clear up space.
b

bitter-ability-32190

05/05/2023, 7:03 PM
Reminder that "you" is "us" 😉
f

flat-zoo-31952

05/05/2023, 7:48 PM
Idk until I actually get down and start helping here I feel it’s more of a formality that I’m part of this in the same way as some more active contributors are
What could I do to help?
b

bitter-ability-32190

05/05/2023, 8:03 PM
How's your Rust? 😅
f

flat-zoo-31952

05/05/2023, 8:04 PM
I’ve read the book, that’s about it
Maybe I could learn while trying to tackle one of these issues
If Pantsd is so critical to perf, then maybe finding a way to optimize what it actually caches would be an interesting project
b

bitter-ability-32190

05/05/2023, 8:10 PM
FWIW I've only learned rust through changes in this project 🙂
f

flat-zoo-31952

05/05/2023, 8:14 PM
Yeah seems like the right way to learn
Porting target or field set to native sounds at least self contained
(Maybe not if the code calls a bunch of other Python code)
w

witty-crayon-22786

05/05/2023, 8:40 PM
yea… 90% of the challenge with Python->Rust porting is finding the right boundary to port. because you want to release the GIL for long periods, and so if you have to re-acquire it from Rust, then you lose some of the benefit (not all of it though, because “Rust with the GIL acquired” is still faster than “Python with the GIL acquired”
f

flat-zoo-31952

05/05/2023, 8:54 PM
I mean is there a good place to start here? I’d be interested in diving in, particularly with making pantsd smarter about what it holds in memory, but that sounds like a tall order
I’ve implemented some LRU stuff in other languages. Is there a practical reason why pantsd doesn’t do LRU but just dies when it outgrows it’s memory limit?
If I could be pointed in the right direction here I’d take a look
w

witty-crayon-22786

05/05/2023, 8:58 PM
no, no practical reason! see https://github.com/pantsbuild/pants/issues/7675 about doing that.
i’d be happy to write up some details about how to implement it, if you’re interested. it would be about 95% Rust, but i’m happy to overexplain
f

flat-zoo-31952

05/05/2023, 9:03 PM
Yeah please do. I’ve never done anything significant in Rust but this seems like a good as place as any to dive in
w

witty-crayon-22786

05/05/2023, 9:04 PM
k: will do before the end of the day. thanks!
f

flat-zoo-31952

05/05/2023, 9:14 PM
You can explain on that issue, I’m subscribed now (probably better there than in slack)
👍 1
w

witty-crayon-22786

05/06/2023, 3:56 AM
here you go: https://github.com/pantsbuild/pants/issues/7675#issuecomment-1537037248 : feel free to ping any of us with questions, but no pressure obviously: if you don’t have time that’s fine!
b

bitter-ability-32190

05/06/2023, 11:29 AM
(your bullet point number two uses
./pants check
when it should be
./cargo check
Actually everywhere you use
./pants
should be
./cargo
😂
😓 1
w

witty-crayon-22786

05/07/2023, 8:26 PM
fixed! thanks.
b

bitter-ability-32190

05/07/2023, 9:01 PM
That's just years of muscle memory 😉