https://pantsbuild.org/ logo
r

rapid-bird-79300

10/18/2021, 7:13 PM
Hey Pants Team, seeing a new error when bootstraping pants:
Copy code
Error: 6.38 [ERROR] 1 Exception encountered:

Engine traceback:
  in select
  in pants.init.plugin_resolver.resolve_plugins
  in pants.backend.python.util_rules.pex.create_venv_pex (pants_plugins.pex)
  in pants.backend.python.util_rules.pex.build_pex (pants_plugins.pex)
  in pants.backend.python.util_rules.pex.build_pex_component (pants_plugins.pex)
  in pants.backend.python.util_rules.pex_cli.setup_pex_cli_process
  in pants.backend.python.util_rules.pex_cli.download_pex_pex
  in pants.core.util_rules.external_tool.download_external_tool
  in downloaded_file
Traceback (no traceback):
  <pants native internals>
Exception: Client error (403) downloading file pex from <https://github.com/pantsbuild/pex/releases/download/v2.1.44/pex>
Traceback (most recent call last):
  File "/home/runner/.cache/pants/setup/bootstrap-Linux-x86_64/2.7.0_py38/lib/python3.8/site-packages/pants/bin/daemon_pants_runner.py", line 124, in single_daemonized_run
    scheduler, options_initializer = self._core.prepare(options_bootstrapper, complete_env)
  File "/home/runner/.cache/pants/setup/bootstrap-Linux-x86_64/2.7.0_py38/lib/python3.8/site-packages/pants/pantsd/pants_daemon_core.py", line 129, in prepare
    build_config, options = self._options_initializer.build_config_and_options(
  File "/home/runner/.cache/pants/setup/bootstrap-Linux-x86_64/2.7.0_py38/lib/python3.8/site-packages/pants/init/options_initializer.py", line 107, in build_config_and_options
    build_config = _initialize_build_configuration(
  File "/home/runner/.cache/pants/setup/bootstrap-Linux-x86_64/2.7.0_py38/lib/python3.8/site-packages/pants/init/options_initializer.py", line 47, in _initialize_build_configuration
    working_set = plugin_resolver.resolve(options_bootstrapper, env)
  File "/home/runner/.cache/pants/setup/bootstrap-Linux-x86_64/2.7.0_py38/lib/python3.8/site-packages/pants/init/plugin_resolver.py", line 123, in resolve
    for resolved_plugin_location in self._resolve_plugins(options_bootstrapper, env):
  File "/home/runner/.cache/pants/setup/bootstrap-Linux-x86_64/2.7.0_py38/lib/python3.8/site-packages/pants/init/plugin_resolver.py", line 146, in _resolve_plugins
    session.product_request(ResolvedPluginDistributions, [self._request])[0],
  File "/home/runner/.cache/pants/setup/bootstrap-Linux-x86_64/2.7.0_py38/lib/python3.8/site-packages/pants/engine/internals/scheduler.py", line 562, in product_request
    self._raise_on_error([t for _, t in throws])
  File "/home/runner/.cache/pants/setup/bootstrap-Linux-x86_64/2.7.0_py38/lib/python3.8/site-packages/pants/engine/internals/scheduler.py", line 497, in _raise_on_error
    raise ExecutionError(
pants.engine.internals.scheduler.ExecutionError: 1 Exception encountered:

Engine traceback:
  in select
  in pants.init.plugin_resolver.resolve_plugins
  in pants.backend.python.util_rules.pex.create_venv_pex (pants_plugins.pex)
  in pants.backend.python.util_rules.pex.build_pex (pants_plugins.pex)
  in pants.backend.python.util_rules.pex.build_pex_component (pants_plugins.pex)
  in pants.backend.python.util_rules.pex_cli.setup_pex_cli_process
  in pants.backend.python.util_rules.pex_cli.download_pex_pex
  in pants.core.util_rules.external_tool.download_external_tool
  in downloaded_file
Traceback (no traceback):
  <pants native internals>
Exception: Client error (403) downloading file pex from <https://github.com/pantsbuild/pex/releases/download/v2.1.44/pex>

(Use --no-process-execution-local-cleanup to inspect chroots and/or -ldebug for more logs. See <https://www.pantsbuild.org/v2.7/docs/troubleshooting> for common issues. Consider reaching out for help: <https://www.pantsbuild.org/v2.7/docs/getting-help.>)
Error: Process completed with exit code 1.
any way we can prevent this?
I see our pants script is outdated. Maybe it's related.
e

enough-analyst-54434

10/18/2021, 7:18 PM
403 from GH downloading the Pex PEX looks like a rate limit.
f

fast-nail-55400

10/18/2021, 7:18 PM
we had a CI failure with the same error here at Toolchain
e

enough-analyst-54434

10/18/2021, 7:19 PM
I'll dig in a bit.
👍 1
f

fast-nail-55400

10/18/2021, 7:21 PM
also might not be related, but I saw a similar issue a few weeks ago with one of the kubernetes repos. github had a bug in their S3 URL signing (which I ended up reporting to them). so if you see a redirect to a S3 URL, could be related.
e

enough-analyst-54434

10/18/2021, 7:21 PM
Hrm: https://docs.github.com/en/repositories/releasing-projects-on-github/about-releases#storage-and-bandwidth-quotas so this is either a GH bug or it's the client side being rate limited.
Ah, ok. Thanks @fast-nail-55400 that's useful to know.
p

polite-garden-50641

10/18/2021, 7:23 PM
maybe we should move stuff to cloudflare r2 since their egress is FREE (unlike s3, which is what GH uses so the have to limit bandwidth usage in some way since AWS egress is expensive) https://blog.cloudflare.com/introducing-r2-object-storage/
just a though.
also as a more concrete suggestion, @rapid-bird-79300 the ~/.cache/pants/setup directory should be cached with whatever CI system you are using.
there is a bunch of guidance here: https://www.pantsbuild.org/docs/using-pants-in-ci
1
r

rapid-bird-79300

10/18/2021, 7:28 PM
yeah we're missing this going to take a look at that now
p

polite-garden-50641

10/18/2021, 7:29 PM
it will make things faster... if you don't cache anything pants does in CI you end up running everything all the time... so basically you are not taking advantage of pant's powerful caching mechanisms..
👍 1
e

enough-analyst-54434

10/18/2021, 7:45 PM
Following uo on Tom's tip - GH is serving these GH releases from S3.
r

rapid-bird-79300

10/18/2021, 10:14 PM
@polite-garden-50641 we added support for caching to help with this. Thank you for the suggestion!
8 Views