#general
Title
l
loud-laptop-17949
10/20/2022, 5:34 PMIs there any way to get external tool downloads to be retried automatically? We see a regular low-level of flaky failures that I would like to avoid with some automatic retries.
w
witty-crayon-22786
10/20/2022, 5:34 PMDan added retry in 2.14.x… which version are you using?
l
loud-laptop-17949
10/20/2022, 5:48 PMcurrently on
Copy code
$ p --version
2.14.0rc1+gitf06bf22bh
hundreds-father-404
10/20/2022, 5:49 PM2.14.0rc3 is the latest, and I'm trying to get out 2.14.0rc4 today
l
loud-laptop-17949
10/20/2022, 5:49 PMyeah I think we either don't have that change or its not effective for this error:
Copy code
Exception: Error reading URL fetch response: request or response body error: error reading a body from connection: connection resetits also a crashing error on this version:
Copy code
NoneType: None
FATAL: exception not rethrown
build/pants/ci/common.sh: line 14: 543 Aborted (core dumped) ./pants "${pants_args[@]}" 2>&1we can try an upgrade first and see if it persists
w
witty-crayon-22786
10/20/2022, 5:53 PM…hm. no, @sparse-lifeguard-95737’s patch should be in
2.14.0rc1+gitf06bf22bor in a DM if sensitive.
l
loud-laptop-17949
10/20/2022, 5:54 PMsure. I wonder if this is not getting retried because it is not an http-level failure but a tcp one?
w
witty-crayon-22786
10/20/2022, 5:54 PMunclear: i don’t see enough context there to know.
… i’m actually not even sure that this is the same codepath. need more context
l
loud-laptop-17949
10/20/2022, 5:56 PMmight be plugin loading instead
w
witty-crayon-22786
10/20/2022, 6:06 PMah… no. i see what is happening. will comment on the ticket, and look into this today.
l
loud-laptop-17949
10/20/2022, 7:42 PMthanks
semi-related: what is the status of caching tool and plugin downloads?
w
witty-crayon-22786
10/20/2022, 7:45 PMtool downloads will always be cached locally, and they’ll be cached remotely, but only if
CacheContentBehavior != fetchhttps://github.com/pantsbuild/pants/issues/13968 covers plugin downloads: they are cached in the local process cache, but they won’t ever be cached remotely (due to a bootstrap problem with auth). that ticket discusses moving the local cache for plugin fetching outside of the default local cache location, so that it can be saved even if you’re not otherwise saving the local cache
adding a bit there about your use case (and what you do cache locally) would be helpful
fix for the retry issue: https://github.com/pantsbuild/pants/pull/17298
this was released on Friday in rc5: https://pypi.org/project/pantsbuild.pants/2.14.0rc5/
@sparse-lifeguard-95737: have you all had a chance to try out rc5? we’ll probably do
2.14.0s
sparse-lifeguard-95737
10/25/2022, 4:45 PMwe upgraded yesterday, so far seems good
this morning we got a report that somebody’s local
named_cachesand we upgraded a ton of dependencies yesterday too, so it’s unclear if any of it is pants’ fault
investigating now
👍 1
the error is:
Copy code
ERROR: Failed building wheel for cairocffi
ERROR: Command errored out with exit status 1:
...
FileExistsError: [Errno 17] File exists: '/Users/jayson/.cache/pants/named_caches/pex_root/pip_cache/.tmp/pip-req-build-dfkg3pky/.eggs/cffi-1.15.1-py3.8-macosx-11.6-x86_64.egg'
----------------------------------------
ERROR: Failed cleaning build dir for cairocffi
ERROR: Failed to build one or more wheelsthe person hitting it says the file doesn’t exist after the failure (something probably cleaning up the temp dir)
w
witty-crayon-22786
10/25/2022, 4:48 PM… weird. yea, that seems related to the version bumps potentially… maybe an issue in that library’s
setup.pys
sparse-lifeguard-95737
10/25/2022, 4:48 PMdowngrading pants and the library individually didn’t fix it, but downgrading both fixed it
6 Views