Hey Pants Team, seeing a new error when bootstraping pants:

Error: 6.38 [ERROR] 1 Exception encountered:

Engine traceback:
  in select
  in pants.init.plugin_resolver.resolve_plugins
  in pants.backend.python.util_rules.pex.create_venv_pex (pants_plugins.pex)
  in pants.backend.python.util_rules.pex.build_pex (pants_plugins.pex)
  in pants.backend.python.util_rules.pex.build_pex_component (pants_plugins.pex)
  in pants.backend.python.util_rules.pex_cli.setup_pex_cli_process
  in pants.backend.python.util_rules.pex_cli.download_pex_pex
  in pants.core.util_rules.external_tool.download_external_tool
  in downloaded_file
Traceback (no traceback):
  <pants native internals>
Exception: Client error (403) downloading file pex from <https://github.com/pantsbuild/pex/releases/download/v2.1.44/pex>
Traceback (most recent call last):
  File "/home/runner/.cache/pants/setup/bootstrap-Linux-x86_64/2.7.0_py38/lib/python3.8/site-packages/pants/bin/daemon_pants_runner.py", line 124, in single_daemonized_run
    scheduler, options_initializer = self._core.prepare(options_bootstrapper, complete_env)
  File "/home/runner/.cache/pants/setup/bootstrap-Linux-x86_64/2.7.0_py38/lib/python3.8/site-packages/pants/pantsd/pants_daemon_core.py", line 129, in prepare
    build_config, options = self._options_initializer.build_config_and_options(
  File "/home/runner/.cache/pants/setup/bootstrap-Linux-x86_64/2.7.0_py38/lib/python3.8/site-packages/pants/init/options_initializer.py", line 107, in build_config_and_options
    build_config = _initialize_build_configuration(
  File "/home/runner/.cache/pants/setup/bootstrap-Linux-x86_64/2.7.0_py38/lib/python3.8/site-packages/pants/init/options_initializer.py", line 47, in _initialize_build_configuration
    working_set = plugin_resolver.resolve(options_bootstrapper, env)
  File "/home/runner/.cache/pants/setup/bootstrap-Linux-x86_64/2.7.0_py38/lib/python3.8/site-packages/pants/init/plugin_resolver.py", line 123, in resolve
    for resolved_plugin_location in self._resolve_plugins(options_bootstrapper, env):
  File "/home/runner/.cache/pants/setup/bootstrap-Linux-x86_64/2.7.0_py38/lib/python3.8/site-packages/pants/init/plugin_resolver.py", line 146, in _resolve_plugins
    session.product_request(ResolvedPluginDistributions, [self._request])[0],
  File "/home/runner/.cache/pants/setup/bootstrap-Linux-x86_64/2.7.0_py38/lib/python3.8/site-packages/pants/engine/internals/scheduler.py", line 562, in product_request
    self._raise_on_error([t for _, t in throws])
  File "/home/runner/.cache/pants/setup/bootstrap-Linux-x86_64/2.7.0_py38/lib/python3.8/site-packages/pants/engine/internals/scheduler.py", line 497, in _raise_on_error
    raise ExecutionError(
pants.engine.internals.scheduler.ExecutionError: 1 Exception encountered:

Engine traceback:
  in select
  in pants.init.plugin_resolver.resolve_plugins
  in pants.backend.python.util_rules.pex.create_venv_pex (pants_plugins.pex)
  in pants.backend.python.util_rules.pex.build_pex (pants_plugins.pex)
  in pants.backend.python.util_rules.pex.build_pex_component (pants_plugins.pex)
  in pants.backend.python.util_rules.pex_cli.setup_pex_cli_process
  in pants.backend.python.util_rules.pex_cli.download_pex_pex
  in pants.core.util_rules.external_tool.download_external_tool
  in downloaded_file
Traceback (no traceback):
  <pants native internals>
Exception: Client error (403) downloading file pex from <https://github.com/pantsbuild/pex/releases/download/v2.1.44/pex>

(Use --no-process-execution-local-cleanup to inspect chroots and/or -ldebug for more logs. See <https://www.pantsbuild.org/v2.7/docs/troubleshooting> for common issues. Consider reaching out for help: <https://www.pantsbuild.org/v2.7/docs/getting-help.>)
Error: Process completed with exit code 1.

any way we can prevent this?

403 from GH downloading the Pex PEX looks like a rate limit.

we had a CI failure with the same error here at Toolchain

I'll dig in a bit.

also might not be related, but I saw a similar issue a few weeks ago with one of the kubernetes repos. github had a bug in their S3 URL signing (which I ended up reporting to them). so if you see a redirect to a S3 URL, could be related.

Hrm: https://docs.github.com/en/repositories/releasing-projects-on-github/about-releases#storage-and-bandwidth-quotas so this is either a GH bug or it's the client side being rate limited.

maybe we should move stuff to cloudflare r2 since their egress is FREE (unlike s3, which is what GH uses so the have to limit bandwidth usage in some way since AWS egress is expensive) https://blog.cloudflare.com/introducing-r2-object-storage/

yeah we're missing this going to take a look at that now

it will make things faster... if you don't cache anything pants does in CI you end up running everything all the time... so basically you are not taking advantage of pant's powerful caching mechanisms..

Following uo on Tom's tip - GH is serving these GH releases from S3.

@polite-garden-50641 we added support for caching to help with this. Thank you for the suggestion!