https://pantsbuild.org/ logo
p

proud-dentist-22844

03/15/2023, 6:32 PM
@curved-television-6568: Thank you for
generate-lockfiles --diff
! (I added the option in pants.toml) This is so nice!
❤️ 4
h

high-yak-85899

03/15/2023, 6:39 PM
woah! Which release is this available in?
p

proud-dentist-22844

03/15/2023, 6:39 PM
2.16.0a0
h

happy-kitchen-89482

03/15/2023, 6:53 PM
It's super great
I think we should turn it on by default
p

proud-dentist-22844

03/15/2023, 6:54 PM
I agree, though it does seem to make lockfile generation take significantly longer (but totally worth it).
It takes ~4 min to lock in the st2 repo. Adding diff adds another minute or two. (or at least it normally goes to around 260 seconds, and I saw it go over 320 or so when generating the diff).
e

enough-analyst-54434

03/15/2023, 6:56 PM
Hrm. That's too bad. The
pex3 lock update ...
which Pants does not use, has supported printing this - less prettily - since day 1. It adds ~no extra overhead.
p

proud-dentist-22844

03/15/2023, 6:57 PM
We really need to iron down the UX for dependecy management in pants beyond just
generate-lockfiles
c

curved-television-6568

03/15/2023, 7:03 PM
Nice. But I’m surprised it adds so much overhead…. did you try locking the same requirements both with and without diff?
p

proud-dentist-22844

03/15/2023, 7:03 PM
I can investigate further...
🙏 1
c

curved-television-6568

03/15/2023, 7:06 PM
@enough-analyst-54434 would’ve been nice if pants made better use of the pex features. Another benefit of this approach though will be that it is agnostic to the lock tool being used, so we can get the same information also for go and java etc..
p

proud-dentist-22844

03/15/2023, 7:14 PM
Yeah. I way over estimated: with diff:
./pants generate-lockfiles --resolve=st2 1.78s user 0.11s system 0% cpu 3:44.55 total
without:
./pants generate-lockfiles --resolve=st2 1.41s user 0.09s system 0% cpu 3:16.25 total
Just goes to show that I really have no concept of time and I blink at the wrong times.
And that's only one run, so 🤷
c

curved-television-6568

03/15/2023, 7:16 PM
thanks for checking. I’ll keep an eye on this too anyhow
h

happy-kitchen-89482

03/15/2023, 8:28 PM
Should this be re-implemented via pex? (we want
lock update
anyway for incremental lockfile generation - officially now the community's highest priority feature request)
c

curved-television-6568

03/15/2023, 8:57 PM
if done in pex we need to redo this for go/js/java/etc too… but, we may have some way to communicate the diff data if there turns out to be a perf issue..
where as now what is missing is “only” the diff information that is missing for those other eco systems..
(we want
lock update
anyway for incremental lockfile generation..
this should not affect the diff logic at all
h

happy-kitchen-89482

03/15/2023, 9:49 PM
Oh right, multiple languages... never mind then
but yeah, possibly to communicate the diff data
or, we should profile that code to see if we can speed it up, seems like it should be possible
I can't offhand think why this would be computationally intensive, and it's not doing any network requests?
c

curved-television-6568

03/15/2023, 10:38 PM
exactly, I would expect it to be rather performant as it merely parses the lockfile json and traverses it, basically. There are a bunch of set operations over it, but still would expect it to be reasonable..
p

proud-dentist-22844

03/15/2023, 11:45 PM
This lockfile is 259K in size. most of my other lockfiles are in the 10s of K in size.
h

happy-kitchen-89482

03/16/2023, 12:16 AM
So there's probably some accidentally egregious algorithm in there that could be discovered via profiling