https://pantsbuild.org/ logo
Join Slack
Powered by
Is there any way to get external tool downloads to...
# general
l
Is there any way to get external tool downloads to be retried automatically? We see a regular low-level of flaky failures that I would like to avoid with some automatic retries.
w
Dan added retry in 2.14.x… which version are you using?
l
currently on
Copy code
$ p --version
2.14.0rc1+gitf06bf22b
I will see if we need to upgrade and bug @sparse-lifeguard-95737 about it
h
2.14.0rc3 is the latest, and I'm trying to get out 2.14.0rc4 today
l
yeah I think we either don't have that change or its not effective for this error:
Copy code
Exception: Error reading URL fetch response: request or response body error: error reading a body from connection: connection reset
its also a crashing error on this version:
Copy code
NoneType: None
FATAL: exception not rethrown
build/pants/ci/common.sh: line 14:   543 Aborted                 (core dumped) ./pants "${pants_args[@]}" 2>&1
we can try an upgrade first and see if it persists
w
…hm. no, @sparse-lifeguard-95737’s patch should be in 
2.14.0rc1+gitf06bf22b
. would you mind filing an issue with more of the context?
or in a DM if sensitive.
l
sure. I wonder if this is not getting retried because it is not an http-level failure but a tcp one?
w
unclear: i don’t see enough context there to know.
… i’m actually not even sure that this is the same codepath. need more context
l
might be plugin loading instead
w
ah… no. i see what is happening. will comment on the ticket, and look into this today.
l
thanks
semi-related: what is the status of caching tool and plugin downloads?
w
tool downloads will always be cached locally, and they’ll be cached remotely, but only if 
CacheContentBehavior != fetch
(although whether that’s actually better for performance is unclear, since you’re still downloading it either way)
https://github.com/pantsbuild/pants/issues/13968 covers plugin downloads: they are cached in the local process cache, but they won’t ever be cached remotely (due to a bootstrap problem with auth). that ticket discusses moving the local cache for plugin fetching outside of the default local cache location, so that it can be saved even if you’re not otherwise saving the local cache
adding a bit there about your use case (and what you do cache locally) would be helpful
this was released on Friday in rc5: https://pypi.org/project/pantsbuild.pants/2.14.0rc5/
@sparse-lifeguard-95737: have you all had a chance to try out rc5? we’ll probably do 
2.14.0
tomorrow if it looks good
s
we upgraded yesterday, so far seems good
this morning we got a report that somebody’s local 
named_caches
somehow got corrupted but we can’t repro anywhere else
and we upgraded a ton of dependencies yesterday too, so it’s unclear if any of it is pants’ fault
investigating now
👍 1
the error is:
Copy code
ERROR: Failed building wheel for cairocffi
  ERROR: Command errored out with exit status 1:
...
  FileExistsError: [Errno 17] File exists: '/Users/jayson/.cache/pants/named_caches/pex_root/pip_cache/.tmp/pip-req-build-dfkg3pky/.eggs/cffi-1.15.1-py3.8-macosx-11.6-x86_64.egg'
  ----------------------------------------
  ERROR: Failed cleaning build dir for cairocffi
ERROR: Failed to build one or more wheels
the person hitting it says the file doesn’t exist after the failure (something probably cleaning up the temp dir)
w
… weird. yea, that seems related to the version bumps potentially… maybe an issue in that library’s 
setup.py
?
s
downgrading pants and the library individually didn’t fix it, but downgrading both fixed it
6 Views
Open in Slack
PreviousNext
Powered by