https://pantsbuild.org/ logo
#general
Title
# general
l

lemon-oxygen-72498

01/31/2023, 3:21 PM
We made our Pants CI pipeline mandatory at my client, and so developers start having concerns that our Pants pipeline is slower than the regular CI pipeline. The crux is that our regular CI spawns a test job per library of our monorepo being modified, whereas the Pants pipeline simply runs
./pants --changed-since=origin/main --changed-dependees=transitive test
. And because GitHub default runners have pretty low performance, the Pants CI is on the average 4 to 5 times slower, because it uses a single machine. I wonder where I should be headed to speed this up. For the moment we don't cache Pants' state and use
--changed-since=origin/main
, because downloading the cache was quite slow. Ways forward that I see: 1. Use persistent runners, keep Pants' state 2. Use remote executors 3. Use a remote cache Any experience to share on what would yield the biggest speedup?
c

curved-television-6568

01/31/2023, 3:49 PM
You may want to look into sharding the tests. See https://www.pantsbuild.org/docs/reference-test#shard
l

lemon-oxygen-72498

01/31/2023, 3:50 PM
Thanks @curved-television-6568, exactly the kind of things I was looking for 🙏
👍 1
h

happy-kitchen-89482

01/31/2023, 4:11 PM
In 2.15 you can also run tests in larger batches than one-per-file: https://www.pantsbuild.org/v2.15/docs/python-test-goal#batching-and-parallelism
👍 1
And also (shamless plug): you can try out remote caching at https://www.toolchain.com/ 🙂
l

lemon-oxygen-72498

02/01/2023, 8:37 AM
@happy-kitchen-89482> yes remote caching is an option 🙂 I wanted to have opinions on the balance between providing a quick enhancement and a more long-term solution.
3 Views