https://pantsbuild.org/ logo
Join Slack
Powered by
I've been thinking along the lines of having a pan...
# development
a
I've been thinking along the lines of having a pants feature that computes an optimal shard size for my CI. When using pants test --changed-since=origin/${{ github.base_ref }} --changed-dependents=transitive. I'm never really sure if I will be running 10 or 100 tests in my CI. If I'm able to dynamically set shard size based on number of targets in combination with ARC on a K8S cluster I could really optimise my usage. Essentially this would help me avoid the cases where sharding may actually not be helpful (too few targets) and also having to update my shards every few months when number of test cases increase. Feel free to drop thoughts, can try to have something out next week for this
Of course I can always add a post processing step after I get all my targets with pants --changed-since --dependents. Then add the logic for how to shard but would this be better off as a feature offered by pants natively?
I think that might actually be better since pants already offers all the primitives I need. It won't really know my runner spin-up cost, ARC provisioning latency, $/min, cache topology and expecting it to learn all that would introduce coupling of unrelated items
h
How does this relate to the 
--shard=k/N
feature? That will partition the tests into roughly equal shards, so are you actually trying to assess the desired total number of shards?
c
I've long wanted to (vertically) scale based on 
--changed-since
, but I'm not sure exactly what additional Pants primitives would help that versus rolling up my sleaves on the dynamic CI config.
2 Views
Open in Slack
PreviousNext
Powered by