https://pantsbuild.org/ logo
Join Slack
Powered by
Hi Folks - hopefully not too random of a question,...
# general
a
Hi Folks - hopefully not too random of a question, but I notice that, in CI, the vast majority of our time building comes from downloading & building our 3rd party dependencies - those don't change that often - and I'm wondering if there's a way to use pants to generate a wheelhouse or something that we could use to speed up installation? Is there some way I can either get pants to do that for me - or export my resolve in some way that pip would understand it? or are other people experiencing this and have other solutions? i might just be barking up the wrong tree
w
Shouldn't that all get cached?
a
I find that if this is running on CI there are a lot of cases for the cache to be invalidated
w
But, downloading the 3rd party deps?
https://www.pantsbuild.org/stable/docs/using-pants/using-pants-in-ci Also that, but that's just back to regular caching. You can also try to get more granular caching - but I've personally found that to be mixed quality. There are also mechanisms to change how everything gets bundled together, to try to make everything as cache friendly as possible. This is usually for Docker, but it might help give a vibe https://www.pantsbuild.org/blog/2022/08/02/optimizing-python-docker-deploys-using-pants
a
yah - so I am following the instructions in that blog post which is building the two pex files, then the two docker images, and then copying the resulting files into a new docker image But... it seems like I can't rely on docker caching there - and if the pants cache is invalidated it needs to regenerate the pex with all the dependencies. and since we do this in CI it's hard to keep a coherent enough cache to keep that from happening?
what really takes a long time is building wheels for some things that don't have binary wheels available 😞
w
So, I don't think I understand this line:
and if the pants cache is invalidated it needs to regenerate the pex with all the dependencies.
Why is the pants cache getting invalidated in regular CI usage?
The caches are all gone here, because of Github - but I was testing using this, and the cache stays stable, unless you're changing your lockfiles and stuff often - in which case, you do want to re-build cache. https://github.com/sureshjoshi/perfanity/blob/main/.github/workflows/ci.yml Maybe there is also a world where you use multiple resolves? I'm not sure offhandedly, but maybe that would at least allow a more granular cache? I haven't done it, so kinda just making things up.
a
honestly I'm not 100% sure on it - i'll look into it some more (and it was a while since I tried this before and I might be mistaken) and it might have been that the caches were... too big? we have quite a large project
w
Ahh, interesting. GH has a limit, and I recall that being almost an issue for a company I worked with
How big are we talking here?
a
uh... 109 dependencies being installed Edit - not sure I'm supposed to share this so I've removed (but it's pretty big)
w
Okay, I can see how this might be more of an issue. There is also the question of how inherently coupled/decoupled your files/projects/subprojects all are. The way I typically setup my monorepos, it’s just a single repo for a whole bunch of somewhat independent projects - and I still have each project be the interface for consumption. However, in other projects, I’ve seen that files anywhere can access any other file - and that usually becomes a lot harder to optimize
a
😞 this is unfortunately quite hard to optimize
w
Hmm, damn. That sucks. Also, I feel like I’ve strayed you far from your original question. Sorry
a
yah i mean - I haven't really tried it on CI in a while so maybe it'll work? (and we did switch from circle to github actions)
but it could be a win if we could, using pants, only build all required wheels once and then persist them (idk, in an image somewhere, or github cache, or a pip repository)
(i guess another question about caching - but are there limits locally? I changed a parameter on a docker_image target just now and it's recalculating a bunch of transitive dependencies)
(that's maybe an unanswerable question sorry)
w
Yeah, hard to say without working on it. There might be some logging about what it's doing in .pants.d
Oh, I almost forgot - but one thing I've found speeds up "the process" is tightening up interpreter_constraints. I don't know how much that helps here, but when searching for and resolving packages, the difference between "3.11.* and 3.11.5" is night and day
a
Oh that’s super interesting - I increased daemon memory limit and cache shards, and added some ignore files and will tighten the constraints and check it out tomorrow
4 Views
Open in Slack
PreviousNext
Powered by