trying to upgrade our version of

black

and i’m getting an error i don’t understand when trying to run

./pants fmt

, does anyone understand what the issue is? (traceback in thread)

Is

black.lock

using Pex or Poetry?

martim@Argon allpdl % ./pants generate-lockfiles --resolve=black
14:57:22.94 [INFO] Completed: Generate lockfile for black
14:57:22.95 [INFO] Wrote lockfile for the resolve `black` to black.lock
martim@Argon allpdl % ./pants fmt ::                            
14:57:25.85 [INFO] Completed: Format with shfmt - shfmt made no changes.
14:57:28.77 [INFO] Completed: Building black.pex from black.lock
14:57:28.77 [ERROR] 1 Exception encountered:

  ProcessExecutionFailure: Process 'Building black.pex from black.lock' failed with exit code 1.
stdout:

stderr:
Traceback (most recent call last):
  File "/Users/martim/.cache/pants/named_caches/pex_root/unzipped_pexes/3c037481d92e64c70b4c4d598b63e24e6c2bb819/.bootstrap/pex/pex.py", line 503, in execute
    exit_value = self._wrap_coverage(self._wrap_profiling, self._execute)
  File "/Users/martim/.cache/pants/named_caches/pex_root/unzipped_pexes/3c037481d92e64c70b4c4d598b63e24e6c2bb819/.bootstrap/pex/pex.py", line 408, in _wrap_coverage
    return runner(*args)
  File "/Users/martim/.cache/pants/named_caches/pex_root/unzipped_pexes/3c037481d92e64c70b4c4d598b63e24e6c2bb819/.bootstrap/pex/pex.py", line 439, in _wrap_profiling
    return runner(*args)
  File "/Users/martim/.cache/pants/named_caches/pex_root/unzipped_pexes/3c037481d92e64c70b4c4d598b63e24e6c2bb819/.bootstrap/pex/pex.py", line 559, in _execute
    return self.execute_entry(self._pex_info.entry_point)
  File "/Users/martim/.cache/pants/named_caches/pex_root/unzipped_pexes/3c037481d92e64c70b4c4d598b63e24e6c2bb819/.bootstrap/pex/pex.py", line 695, in execute_entry
    return self.execute_pkg_resources(entry_point)
  File "/Users/martim/.cache/pants/named_caches/pex_root/unzipped_pexes/3c037481d92e64c70b4c4d598b63e24e6c2bb819/.bootstrap/pex/pex.py", line 727, in execute_pkg_resources
    return runner()
  File "/Users/martim/.cache/pants/named_caches/pex_root/installed_wheels/9fb4bc59f7085c6133bed415fd8806a166d6f472798f2350158f7f0a710ec0f6/pex-2.1.71-py2.py3-none-any.whl/pex/bin/pex.py", line 724, in main
    do_main(
  File "/Users/martim/.cache/pants/named_caches/pex_root/installed_wheels/9fb4bc59f7085c6133bed415fd8806a166d6f472798f2350158f7f0a710ec0f6/pex-2.1.71-py2.py3-none-any.whl/pex/bin/pex.py", line 770, in do_main
    seed_info = seed_cache(options, pex, verbose=options.seed == Seed.VERBOSE)
  File "/Users/martim/.cache/pants/named_caches/pex_root/installed_wheels/9fb4bc59f7085c6133bed415fd8806a166d6f472798f2350158f7f0a710ec0f6/pex-2.1.71-py2.py3-none-any.whl/pex/bin/pex.py", line 808, in seed_cache
    venv_pex = ensure_venv(pex)
  File "/Users/martim/.cache/pants/named_caches/pex_root/installed_wheels/9fb4bc59f7085c6133bed415fd8806a166d6f472798f2350158f7f0a710ec0f6/pex-2.1.71-py2.py3-none-any.whl/pex/pex_bootstrapper.py", line 461, in ensure_venv
    shebang = populate_venv(
  File "/Users/martim/.cache/pants/named_caches/pex_root/installed_wheels/9fb4bc59f7085c6133bed415fd8806a166d6f472798f2350158f7f0a710ec0f6/pex-2.1.71-py2.py3-none-any.whl/pex/venv/pex.py", line 121, in populate_venv
    record_provenance(_populate_deps(venv, pex, venv_python, symlink))
  File "/Users/martim/.cache/pants/named_caches/pex_root/installed_wheels/9fb4bc59f7085c6133bed415fd8806a166d6f472798f2350158f7f0a710ec0f6/pex-2.1.71-py2.py3-none-any.whl/pex/venv/pex.py", line 117, in record_provenance
    for src, dst in src_to_dst:
  File "/Users/martim/.cache/pants/named_caches/pex_root/installed_wheels/9fb4bc59f7085c6133bed415fd8806a166d6f472798f2350158f7f0a710ec0f6/pex-2.1.71-py2.py3-none-any.whl/pex/venv/pex.py", line 205, in _populate_deps
    for dist in pex.resolve():
  File "/Users/martim/.cache/pants/named_caches/pex_root/installed_wheels/9fb4bc59f7085c6133bed415fd8806a166d6f472798f2350158f7f0a710ec0f6/pex-2.1.71-py2.py3-none-any.whl/pex/pex.py", line 120, in resolve
    for dist in env.resolve():
  File "/Users/martim/.cache/pants/named_caches/pex_root/installed_wheels/9fb4bc59f7085c6133bed415fd8806a166d6f472798f2350158f7f0a710ec0f6/pex-2.1.71-py2.py3-none-any.whl/pex/environment.py", line 495, in resolve
    for fingerprinted_distribution in self.resolve_dists(all_reqs)
  File "/Users/martim/.cache/pants/named_caches/pex_root/installed_wheels/9fb4bc59f7085c6133bed415fd8806a166d6f472798f2350158f7f0a710ec0f6/pex-2.1.71-py2.py3-none-any.whl/pex/environment.py", line 582, in resolve_dists
    raise ResolveError(
pex.environment.ResolveError: Failed to resolve requirements from PEX environment @ /private/var/folders/4s/0qxzthl16y3_cjxt_ygbw__m0000gn/T/process-executionzeKz6X/.tmp/tmpunx2po8k.
Needed cp39-cp39-macosx_12_0_arm64 compatible dependencies for:
 1: typing-extensions>=3.10.0.0; python_version < "3.10"
    Required by:
      FingerprintedDistribution(distribution=black 22.6.0 (/Users/martim/.cache/pants/named_caches/pex_root/installed_wheels/560558527e52ce8afba936fcce93a7411ab40c7d5fe8c2463e279e843c0328ee/black-22.6.0-cp39-cp39-macosx_11_0_arm64.whl), fingerprint='69432c67fe53fd28c8759344e21f9026ae4573e374b7e907d67940c73ed73c1d')
    But this pex had no 'typing-extensions' distributions.



Use `--no-process-cleanup` to preserve process chroots for inspection.

this is my

black

config

[black]
config = "pyproject.toml"
version = "black>=22.0.0,<23.0.0"
lockfile = "black.lock"

and the

black.lock

file has this requirement for

typing-extensions

typing-extensions==4.1.1; python_version < "3.8" and python_full_version >= "3.6.2" and python_version >= "3.6" \
    --hash=sha256:21c85e0fe4b9a155d0799430b0ad741cdce7e359660ccbd8b530613e8df88ce2 \
    --hash=sha256:1a9462dcc3347a79b1f1c0271fbe79e844580bb598bafa1ed208b94da3cdcd42

probably https://www.pantsbuild.org/docs/python-third-party-dependencies#poetry-issue-with-environment-markers if I had to guess

so adding something like this should solve it?

extra_requirements.add = ["typing-extensions"]

exactly, then regen the lock

it works! 🙂

great! shouldn't be an issue w/ Pex lockfiles

though it sounds like using pex will resolve the issue?

tl;dr:

poetry export

is trying to solve a tough problem of how to make a lockfile "universal" where it works on multiple platforms.

poetry export

is a lossy process, so it doesn't always work - it loses important info about platforms Pex avoids that by using its own lockfile format, which is richer than

requirements.txt

style In Poetry's defense, this is only a problem w/

poetry export

. Their native lockfile format works. And we have to use

poetry export

b/c Pex/pip doesn't know how to understand

poetry.lock