Hi everyone! I am running into a transient error in my CICD jobs when running

pants package

that I'm not sure how to approach debugging. I seem to occasionally be getting an error when a specific dependency,

JSON-log-formatter==0.5.2

, is retrieved from pypi.

stderr:
pid 1536 -> /root/.cache/pants/named_caches/pex_root/venvs/17852bb64f5af73a30c4759e2b1c34fcbb900e7c/32b867138fd0be8ea082900614eeab8e11f87e6b/bin/python -sE /root/.cache/pants/named_caches/pex_root/venvs/17852bb64f5af73a30c4759e2b1c34fcbb900e7c/32b867138fd0be8ea082900614eeab8e11f87e6b/pex --disable-pip-version-check --no-python-version-warning --exists-action a --no-input --use-deprecated legacy-resolver --isolated -q --cache-dir /root/.cache/pants/named_caches/pex_root/pip_cache wheel --no-deps --wheel-dir /root/.cache/pants/named_caches/pex_root/built_wheels/sdists/JSON-log-formatter-0.5.2.tar.gz/7b191ae4056468baf2b7445c5ce651bd9ee6b76b4162a479a7d85db6ae029f2d/cp310-cp310-manylinux_2_35_x86_64.eeb85f38b1bb41398b8443db6675b8a2.work /root/.cache/pants/named_caches/pex_root/downloads/7b191ae4056468baf2b7445c5ce651bd9ee6b76b4162a479a7d85db6ae029f2d/JSON-log-formatter-0.5.2.tar.gz --index-url <https://pypi.org/simple/> --retries 5 --timeout 15 exited with 2 and STDERR:
ERROR: Exception:
Traceback (most recent call last):
  File "/root/.cache/pants/named_caches/pex_root/venvs/17852bb64f5af73a30c4759e2b1c34fcbb900e7c/32b867138fd0be8ea082900614eeab8e11f87e6b/lib/python3.10/site-packages/pip/_internal/cli/base_command.py", line 169, in exc_logging_wrapper
    status = run_func(*args)
  File "/root/.cache/pants/named_caches/pex_root/venvs/17852bb64f5af73a30c4759e2b1c34fcbb900e7c/32b867138fd0be8ea082900614eeab8e11f87e6b/lib/python3.10/site-packages/pip/_internal/cli/req_command.py", line 248, in wrapper
    return func(self, options, args)
  File "/root/.cache/pants/named_caches/pex_root/venvs/17852bb64f5af73a30c4759e2b1c34fcbb900e7c/32b867138fd0be8ea082900614eeab8e11f87e6b/lib/python3.10/site-packages/pip/_internal/commands/wheel.py", line 157, in run
    build_successes, build_failures = build(
  File "/root/.cache/pants/named_caches/pex_root/venvs/17852bb64f5af73a30c4759e2b1c34fcbb900e7c/32b867138fd0be8ea082900614eeab8e11f87e6b/lib/python3.10/site-packages/pip/_internal/wheel_builder.py", line 334, in build
    wheel_cache.record_download_origin(cache_dir, req.download_info)
  File "/root/.cache/pants/named_caches/pex_root/venvs/17852bb64f5af73a30c4759e2b1c34fcbb900e7c/32b867138fd0be8ea082900614eeab8e11f87e6b/lib/python3.10/site-packages/pip/_internal/cache.py", line 261, in record_download_origin
    origin = DirectUrl.from_json(origin_path.read_text())
  File "/root/.cache/pants/named_caches/pex_root/venvs/17852bb64f5af73a30c4759e2b1c34fcbb900e7c/32b867138fd0be8ea082900614eeab8e11f87e6b/lib/python3.10/site-packages/pip/_internal/models/direct_url.py", line 231, in from_json
    return cls.from_dict(json.loads(s))
  File "/root/.pyenv/versions/3.10.12/lib/python3.10/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
  File "/root/.pyenv/versions/3.10.12/lib/python3.10/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/root/.pyenv/versions/3.10.12/lib/python3.10/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

I'm not able to consistently reproduce the issue, one in every few builds seems to be failing on this same error, on the same package. I attempted to regenerate my project's lockfiles, in case there was something wrong with the artifact that this was fetching, but no changes were made to it. I checked and the artifact's URL (https://files.pythonhosted.org/packages/80/0f/daee9657d9860f505afab484b3d80f3f1f757ec3a65ab93212558cfac468/JSON-log-formatter-0.5.2.tar.gz) matches what's on pypi.org. Any pointers in debugging this or ideas on the root cause would be appreciated! This is on pants 2.18.0 with interpreter constraint ==3.10.*

this feels like an occasional 5xx/4xx from pypi - try running aith

-ldebug

and see if the output gives you any hints?

Thank you! I will give it a try.