#21255 scala reproducibility is broken somwhere between 2.14.2 to 2.20.2
Issue created by
somdoron
Describe the bug
We are using buildbuddy as a remote cache for our CI environment, we recently upgraded from 2.14.2 to 2.20.2, since the upgrade the build time is much longer. Triggering the same build twice mostly use the cache but still around 25% of actions have cache miss. This is random, it is not the same tasks over multiple builds.
Surprisingly, I'm not able to reproduce the issue on my machine, with local cache disable and no-pantsd.
The main problem for me at the moment is that it is very hard to debug reproducibility over the CI. Enabling debug logging is essential but creates such a big log files that we are not able to download those from github actions.
Aside from trying to fix the reproducibility problem, I'm trying to make it easier to debug those later on.
Trying to bisect the exact commit or version is hard as we have some pants plugins that we have to fix to each version.
Pants version
2.20.2
OS
Linux.
pantsbuild/pants