We made our Pants CI pipeline mandatory at my clie...
# general
We made our Pants CI pipeline mandatory at my client, and so developers start having concerns that our Pants pipeline is slower than the regular CI pipeline. The crux is that our regular CI spawns a test job per library of our monorepo being modified, whereas the Pants pipeline simply runs
./pants --changed-since=origin/main --changed-dependees=transitive test
. And because GitHub default runners have pretty low performance, the Pants CI is on the average 4 to 5 times slower, because it uses a single machine. I wonder where I should be headed to speed this up. For the moment we don't cache Pants' state and use
, because downloading the cache was quite slow. Ways forward that I see: 1. Use persistent runners, keep Pants' state 2. Use remote executors 3. Use a remote cache Any experience to share on what would yield the biggest speedup?
You may want to look into sharding the tests. See https://www.pantsbuild.org/docs/reference-test#shard
Thanks @curved-television-6568, exactly the kind of things I was looking for 🙏
👍 1
In 2.15 you can also run tests in larger batches than one-per-file: https://www.pantsbuild.org/v2.15/docs/python-test-goal#batching-and-parallelism
👍 1
And also (shamless plug): you can try out remote caching at https://www.toolchain.com/ 🙂
@happy-kitchen-89482> yes remote caching is an option 🙂 I wanted to have opinions on the balance between providing a quick enhancement and a more long-term solution.