Given the monthly tradition of someone running into issues with pytorch not working cross platform, or rendering their GHA cache unusable by downloading 15GB of torch releases just to use one, I thought this may be of interest:
https://github.com/pytorch/builder/pull/1457
If my understanding is right, this change landing might be the missing piece (along with them backporting hashes to s3) to solving the various weird issues that come with pytorch and only pytorch. Might be worth trying the "obvious" solutions again afterwards and seeing if this allows them to work