I m seeing consistent deterministic failures on CI

Question

I m seeing consistent deterministic failures on CI Seem unrelated to my change because they also appear in other PRs Are these okay to skip currently trying to repro with docker

average-vr-56795 · Answer

That doesn t sound fun S I d be curious to know if the problem here is that we have a bad download of coursier and haven t verified a checksum of the download or something

red-balloon-89377 · Answer

Opened <https github com pantsbuild pants issues 7952> to track

red-balloon-89377 · Answer

I d also be curious but it s deterministic enough that I don t think it s a network flake and I don t see anything in recent history that modified coursier Will look again while docker is running

red-balloon-89377 · Answer

No luck with docker disappointed

average-vr-56795 · Answer

I m still betting on this being network flakiness we re not detecting

red-balloon-89377 · Answer

It s deterministic in master disappointed I can believe it s network wrongness downloading the wrong thing but we haven t seen it succeed ever after the commit mentioned in the issue

average-vr-56795 · Answer

Did you try blowing away ~ cache and re running a shard

average-vr-56795 · Answer

If we cached the bad download it would explain why it s now deterministic

red-balloon-89377 · Answer

Where is `~` I thought these were all fresh docker images on travis

average-vr-56795 · Answer

I m pretty sure we cache a bunch of things Have a look at how we run the docker image and any volumes we mount in

red-balloon-89377 · Answer

Cool slightly smiling face

red-balloon-89377 · Answer

Interesting according to travis the newest cache is from 12 days ago but this issue started happening 2 days ago

red-balloon-89377 · Answer

We definitely do cache coursier I m going to try to get the file that we have cached

hundreds-father-404 · Answer

< red balloon 89377> do you know how to wipe caches You can either use the web UI or the CLI The CLI works much much better UI doesn t surface everything

red-balloon-89377 · Answer

I was messing around with the CLI to see if I could figure out which one is the faulty one but it doesn t look trivial to say which cache belongs to this PR

red-balloon-89377 · Answer

once logged in I think it s `travis cache delete`

red-balloon-89377 · Answer

Nvm got it

red-balloon-89377 · Answer

Well it wasn t that

red-balloon-89377 · Answer

Will blacklist the test and reference the issue

red-balloon-89377 · Answer

I m not sure this is the right thing to do so would love as many comments as possible <https github com pantsbuild pants pull 7953>

witty-crayon-22786 · Answer

cc < wide energy 11069> ^

wide-energy-11069 · Answer

yep ran into that too can t repro locally either

happy-kitchen-89482 · Answer

Hitting the same issue

happy-kitchen-89482 · Answer

Looks like your change exposed a whole can of worms with similar but not identical breakages

happy-kitchen-89482 · Answer

I ll restart those shards just in case that helps but we have a real problem here

wide-energy-11069 · Answer

pushed a commit in the PR to print sha of coursier jar

wide-energy-11069 · Answer

for context is pantsd enabled for these tests

hundreds-father-404 · Answer

In general no The parent process itself only uses pantsd during the cron job Some integration tests use pantsd via ensure pantsd or directly invoking it though

witty-crayon-22786 · Answer

so master continues to sortof apocalyptically broken it looks like there has been some action on linked from i m going to see what happens with a cache purge and retry of a recent master

witty-crayon-22786 · Answer

commented on 7953

hundreds-father-404 · Answer

so far all green just one shard left

witty-crayon-22786 · Answer

one known flake the rsc basic binary but other than that looks green pheew

witty-crayon-22786 · Answer

Will bump that one

hundreds-father-404 · Answer

Yay Thanks to < wide energy 11069> and < witty crayon 22786> for working with Travis to fix this Stu should we wipe all caches

witty-crayon-22786 · Answer

I don t actually know whether the cache wiping was relevant But sure Can whack a few more moles

hundreds-father-404 · Answer

Okay purging now