https://pantsbuild.org/ logo
Join Slack
Powered by
We've been getting intermittent build failures in ...
# general
a
We've been getting intermittent build failures in CI that we think are attributable to a known issue in Pip (see here). That issue was resolved in Pip 23.2. However, Pants doesn't seem to support 23.2 for some reason. We'd like to bring in the version of Pip that supposedly resolves the issue, but Pants isn't allowing us to bring in that version of Pip. Can someone advise regarding why Pip 23.2 is not supported at this time?
e
Pants uses Pex and Pex supports Pip 23.2 since version 2.1.139: https://github.com/pantsbuild/pex/releases/tag/v2.1.139 If you're using Pants 2.17.x you can ensure you have Pex >= 2.1.139 (here I use 2.1.143, which is latest) by adding the following to `pants.toml`:
Copy code
[pex-cli]
version = "v2.1.143"
known_versions = [
  "v2.1.143|macos_arm64|7dba8776000b4f75bc9af850cb65b2dc7720ea211733e8cb5243c0b210ef3c19|4194291",
  "v2.1.143|macos_x86_64|7dba8776000b4f75bc9af850cb65b2dc7720ea211733e8cb5243c0b210ef3c19|4194291",
  "v2.1.143|linux_x86_64|7dba8776000b4f75bc9af850cb65b2dc7720ea211733e8cb5243c0b210ef3c19|4194291",
  "v2.1.143|linux_arm64|7dba8776000b4f75bc9af850cb65b2dc7720ea211733e8cb5243c0b210ef3c19|4194291"
]
And then select the latest Pip version Pex supports - which happens to be 23.2: https://www.pantsbuild.org/v2.17/docs/reference-python#pip_version by adding this to `pants.toml`:
Copy code
[python]
pip_version = "latest"
All that said!: How sure are you that your problem is exactly the problem you linked? Can you provide a full output log?
👍 1
As an aside, it's pretty unfortunate Pants tries to verify the 
pip_version
value by constraining it to an enum. I really can't discern a Pants rule for when it tries to middle-man vs when it just allows pass-through to the underlying tool. If there is a philosophy, it seems to be applied haphazardly. Here Pants middle-mans and prevents you from saying "23.2". Maybe it's to fail faster with a nice message? But clearly it doesn't do this for other tools where it'll happily slurp a config file, not middle-man inspect it, and pass that on to the tool which may error very late.
a
Hmm, perhaps I'm not understanding the 
pip_version
option. The documentation does not actually list 23.2 as a supported Pip version: https://www.pantsbuild.org/v2.17/docs/reference-python#pip_version. Can you confirm that 23.2 is in fact supported by Pants 2.17.x? Is that just an error in the documentation?
As for the problem itself, here's the relevant excerpt from CI showing the backtrace that matches the backtrace in the problem I linked:
Copy code
Engine traceback:
  in `package` goal
  
ProcessExecutionFailure: Process 'Building 27 requirements for astranis.simulation.components.thermal/board_test.pex from the constraints.txt resolve: aiohttp<4,>=3.8.3, click<9.0,>=8.0, coloredlogs<16,>=15, crcmod<1.8,>=1.7, cryptography<38.0,>=37.0, hvac<1,>=0.11.2, numpy<1.22.0,>=1.20.1, pexpect<5,>=4.4.0, pika<2,>=1.2.1, protobuf==3.19.5, psutil<6,>=5.8.0, pyjwt<3,>=2, pyserial<4,>=3.4, pytest<8,>=7, pyudev<1,>=0.24.0, pyvcd<1,>=0.3.0, pyyaml<6,>=5.4.1, pyzmq<26,>=25.1, requests<3,>=2.28, slackclient<3,>=2.7.2, tabulate<1,>=0.8.1, tqdm<5,>=4.64.1, typeguard<3,>=2.12.1, types-PyYAML<=6.0.12.6,>=6.0.0, types-click<8,>=7.0.0, typing-extensions<5,>=4.1.1, yubihsm[http]<3,>=2.1' failed with exit code 1.
stdout:

stderr:
pid 594578 -> /home/buildbot/.cache/pants/named_caches/pex_root/venvs/793b81cd6d9ac3cd40f816c461d3cddb4b46fce3/ddab8011daaee380698ac2fb9701af18c90c03f6/bin/python -sE /home/buildbot/.cache/pants/named_caches/pex_root/venvs/793b81cd6d9ac3cd40f816c461d3cddb4b46fce3/ddab8011daaee380698ac2fb9701af18c90c03f6/pex --disable-pip-version-check --no-python-version-warning --exists-action a --no-input --use-deprecated legacy-resolver --isolated -q --cache-dir /home/buildbot/.cache/pants/named_caches/pex_root/pip_cache wheel --no-deps --wheel-dir /home/buildbot/.cache/pants/named_caches/pex_root/built_wheels/sdists/crcmod-1.7.tar.gz/dc7051a0db5f2bd48665a990d3ec1cc305a466a77358ca4492826f41f283601e/cp38-cp38-manylinux_2_31_x86_64.1795207a1f2541b0a39cf51c493d56aa.work /home/buildbot/.cache/pants/named_caches/pex_root/downloads/dc7051a0db5f2bd48665a990d3ec1cc305a466a77358ca4492826f41f283601e/crcmod-1.7.tar.gz --index-url <https://pypi.org/simple/> --retries 5 --timeout 15 exited with 2 and STDERR:
ERROR: Exception:
Traceback (most recent call last):
  File "/home/buildbot/.cache/pants/named_caches/pex_root/venvs/793b81cd6d9ac3cd40f816c461d3cddb4b46fce3/ddab8011daaee380698ac2fb9701af18c90c03f6/lib/python3.8/site-packages/pip/_internal/cli/base_command.py", line 160, in exc_logging_wrapper
    status = run_func(*args)
  File "/home/buildbot/.cache/pants/named_caches/pex_root/venvs/793b81cd6d9ac3cd40f816c461d3cddb4b46fce3/ddab8011daaee380698ac2fb9701af18c90c03f6/lib/python3.8/site-packages/pip/_internal/cli/req_command.py", line 247, in wrapper
    return func(self, options, args)
  File "/home/buildbot/.cache/pants/named_caches/pex_root/venvs/793b81cd6d9ac3cd40f816c461d3cddb4b46fce3/ddab8011daaee380698ac2fb9701af18c90c03f6/lib/python3.8/site-packages/pip/_internal/commands/wheel.py", line 180, in run
    build_successes, build_failures = build(
  File "/home/buildbot/.cache/pants/named_caches/pex_root/venvs/793b81cd6d9ac3cd40f816c461d3cddb4b46fce3/ddab8011daaee380698ac2fb9701af18c90c03f6/lib/python3.8/site-packages/pip/_internal/wheel_builder.py", line 361, in build
    wheel_cache.record_download_origin(cache_dir, req.download_info)
  File "/home/buildbot/.cache/pants/named_caches/pex_root/venvs/793b81cd6d9ac3cd40f816c461d3cddb4b46fce3/ddab8011daaee380698ac2fb9701af18c90c03f6/lib/python3.8/site-packages/pip/_internal/cache.py", line 282, in record_download_origin
    origin = DirectUrl.from_json(origin_path.read_text())
  File "/home/buildbot/.cache/pants/named_caches/pex_root/venvs/793b81cd6d9ac3cd40f816c461d3cddb4b46fce3/ddab8011daaee380698ac2fb9701af18c90c03f6/lib/python3.8/site-packages/pip/_internal/models/direct_url.py", line 222, in from_json
    return cls.from_dict(json.loads(s))
  File "/usr/lib/python3.8/json/__init__.py", line 357, in loads
    return _default_decoder.decode(s)
  File "/usr/lib/python3.8/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib/python3.8/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)




Use `--keep-sandboxes=on_failure` to preserve the process chroot for inspection.
For reference, we're currently on Pants 2.16. I have not tried upgrading to Pants 2.17.x. Perhaps 2.17.x does truly support Pip 23.2, though Pants documentation doesn't reflect support for Pip 23.2.
e
Can you confirm that 23.2 is in fact supported by Pants 2.17.x?
Pants does not - my aside was about this - but Pex does. Pex has an escape hatch of 
--pip-version latest
though, and Pants 2.17.x adds support for saying 
pip_version = "latest"
. As such, iff you follow both of my 
pants.toml
edit reccomendations, then you get Pants 2.17+ Using Pex 2.1.139+ using Pip 23.2.
Ok, @abundant-keyboard-19745 thanks for the backtrace - that does indeed seem like the exact problem you linked to.
a
Ah, I see how 
latest
serves as an "escape hatch" for the Pip version. That's kind of not obvious, but yes, I see what you mean.
e
For reference, the Pex help:
Copy code
$ pex --help | grep "pip-version {" -A4
  --pip-version {latest,vendored,20.3.4-patched,22.2.2,22.3,22.3.1,23.0,23.0.1,23.1,23.1.1,23.1.2,23.2}
                        The version of Pip to use for resolving dependencies.
                        The `latest` version refers to the latest version in
                        this list (23.2) which is not necessarily the latest
                        Pip version released on PyPI. (default: 23.2)
Agreed though that the Pants help is confusing:
Copy code
pip_version

--python-pip-version=<PipVersion>

PANTS_PYTHON_PIP_VERSION
one of: 20.3.4-patched, 22.2.2, 22.3, 22.3.1, 23.0, 23.0.1, 23.1, 23.1.1, 23.1.2, latest
default: 20.3.4-patched

Use this version of Pip for resolving requirements and generating lockfiles.

N.B.: The latest value selects the latest of the listed choices which is not necessarily the latest Pip version released on PyPI.
Since it implies "latest" only applies to the list above, which is the list Pants knows about. It actually applies to the list of 
--pip-version
supported by the Pex version you're using.
Pants really should stop being a middle-man here since it doesn't actually care about the Pip version at all.
a
Hmm, so it sounds like Pex is really what should matter for choosing Pip versions, not Pants. However, Pants itself does dictate the Pip versions that can be chosen. Would it be the right thing to do to change 
pip_version
from an enum to a string and then change the plumbing such that the Pip version just goes to Pex directly?
👍 2
c
I think that, or if there’s desire to preserve an early fail, to pick up the list of supported pip versions directly from pex somehow.. but that feels overkill.
a
Should a PR be created for this? If so, who wants to create that PR?
I think I would be willing to take a stab at it if it's not too hard.
👍 1
h
Yes, I think changing the value of that option to a string seems sensible to me, for the reasons stated above. @abundant-keyboard-19745 it would be great if you could take a crack at it. I don't think it's too hard!
It's probably just changing the option type (there are lots of examples of string-valued options to copy from) and making sure the tests still pass
a
I created https://github.com/pantsbuild/pants/pull/19643. I can't seem to add reviewers to the PR for whatever reason. If anyone has comments or would like to review, feel free to add yourself to the PR.
2 Views
Open in Slack
PreviousNext
Powered by