https://pantsbuild.org/ logo
Join Slack
Powered by
I've been poking around <https://github.com/pantsb...
# development
v
I've been poking around https://github.com/pantsbuild/pants/issues/13390 (named_caches for go modules), and gotten something minimal working in my own fork. So I wanted to share back the plan / progress.
motivation: pants was running out of memory / running very slow building go projects with a large import set, so I'm moving the gopath into named caches to see if that addresses speed and / or memory pressure. approach: 
append_only_cache
for the go process + follow on changes in 
analyze_third_party_module
to examine sources from the cache instead of digests. next: first any concerns with the overall direction. Since go builds fast because of heavy local caching, and go supports parallel module download per https://github.com/golang/go/issues/26794 that's enough to say it should work, but second opinions welcome there. Second suggestions for further testing (eg. I'm sure this has downstream impact for remote build agents, but I'd be looking for how to double check those)
👀 1
h
This may be a major performance bottleneck on repeated builds with minor changes, and is clearly something we should do, but AFAICT would not affect clean (no cache) build times where everything has to be downloaded anyway. It seems like the other big source of performance issues on large import set Go builds is https://github.com/pantsbuild/pants/issues/20274
v
I can do a few more passes when clearing the cache, but between fewer downloads and not passing around the digests for downloaded modules, I am seeing much more success on my sample repository from 20274
h
That is good empirical data!
4 Views
Open in Slack
PreviousNext
Powered by