https://pantsbuild.org/ logo
l

loud-laptop-17949

10/20/2022, 5:34 PM
Is there any way to get external tool downloads to be retried automatically? We see a regular low-level of flaky failures that I would like to avoid with some automatic retries.
w

witty-crayon-22786

10/20/2022, 5:34 PM
Dan added retry in 2.14.x… which version are you using?
l

loud-laptop-17949

10/20/2022, 5:48 PM
currently on
Copy code
$ p --version
2.14.0rc1+gitf06bf22b
I will see if we need to upgrade and bug @sparse-lifeguard-95737 about it
h

hundreds-father-404

10/20/2022, 5:49 PM
2.14.0rc3 is the latest, and I'm trying to get out 2.14.0rc4 today
l

loud-laptop-17949

10/20/2022, 5:49 PM
yeah I think we either don't have that change or its not effective for this error:
Copy code
Exception: Error reading URL fetch response: request or response body error: error reading a body from connection: connection reset
its also a crashing error on this version:
Copy code
NoneType: None
FATAL: exception not rethrown
build/pants/ci/common.sh: line 14:   543 Aborted                 (core dumped) ./pants "${pants_args[@]}" 2>&1
we can try an upgrade first and see if it persists
w

witty-crayon-22786

10/20/2022, 5:53 PM
…hm. no, @sparse-lifeguard-95737’s patch should be in
2.14.0rc1+gitf06bf22b
. would you mind filing an issue with more of the context?
or in a DM if sensitive.
l

loud-laptop-17949

10/20/2022, 5:54 PM
sure. I wonder if this is not getting retried because it is not an http-level failure but a tcp one?
w

witty-crayon-22786

10/20/2022, 5:54 PM
unclear: i don’t see enough context there to know.
… i’m actually not even sure that this is the same codepath. need more context
l

loud-laptop-17949

10/20/2022, 5:56 PM
might be plugin loading instead
w

witty-crayon-22786

10/20/2022, 6:06 PM
ah… no. i see what is happening. will comment on the ticket, and look into this today.
l

loud-laptop-17949

10/20/2022, 7:42 PM
thanks
semi-related: what is the status of caching tool and plugin downloads?
w

witty-crayon-22786

10/20/2022, 7:45 PM
tool downloads will always be cached locally, and they’ll be cached remotely, but only if
CacheContentBehavior != fetch
(although whether that’s actually better for performance is unclear, since you’re still downloading it either way)
https://github.com/pantsbuild/pants/issues/13968 covers plugin downloads: they are cached in the local process cache, but they won’t ever be cached remotely (due to a bootstrap problem with auth). that ticket discusses moving the local cache for plugin fetching outside of the default local cache location, so that it can be saved even if you’re not otherwise saving the local cache
adding a bit there about your use case (and what you do cache locally) would be helpful
this was released on Friday in rc5: https://pypi.org/project/pantsbuild.pants/2.14.0rc5/
@sparse-lifeguard-95737: have you all had a chance to try out rc5? we’ll probably do
2.14.0
tomorrow if it looks good
s

sparse-lifeguard-95737

10/25/2022, 4:45 PM
we upgraded yesterday, so far seems good
this morning we got a report that somebody’s local
named_caches
somehow got corrupted but we can’t repro anywhere else
and we upgraded a ton of dependencies yesterday too, so it’s unclear if any of it is pants’ fault
investigating now
👍 1
the error is:
Copy code
ERROR: Failed building wheel for cairocffi
  ERROR: Command errored out with exit status 1:
...
  FileExistsError: [Errno 17] File exists: '/Users/jayson/.cache/pants/named_caches/pex_root/pip_cache/.tmp/pip-req-build-dfkg3pky/.eggs/cffi-1.15.1-py3.8-macosx-11.6-x86_64.egg'
  ----------------------------------------
  ERROR: Failed cleaning build dir for cairocffi
ERROR: Failed to build one or more wheels
the person hitting it says the file doesn’t exist after the failure (something probably cleaning up the temp dir)
w

witty-crayon-22786

10/25/2022, 4:48 PM
… weird. yea, that seems related to the version bumps potentially… maybe an issue in that library’s
setup.py
?
s

sparse-lifeguard-95737

10/25/2022, 4:48 PM
downgrading pants and the library individually didn’t fix it, but downgrading both fixed it
6 Views