I just upgraded from Pants 2.7.1 to 2.8.0 and have...
# general
b
I just upgraded from Pants 2.7.1 to 2.8.0 and have a confusing failure when building a
pex_binary
(see thread)
Copy code
11:15:27.39 [ERROR] 1 Exception encountered:

  ProcessExecutionFailure: Process 'Extracting 16 requirements to build src.python.graplctl.graplctl/graplctl.pex from repository.pex: boto3, botocore, click, mypy_boto3_cloudwatch, mypy_boto3_dynamodb, mypy_boto3_ec2, mypy_boto3_lambda, mypy_boto3_route53, mypy_boto3_s3, mypy_boto3_secretsmanager, mypy_boto3_sns, mypy_boto3_sqs, mypy_boto3_ssm, protobuf>=3.12.1, typing_extensions, zstd==1.4.9.1' failed with exit code 1.
stdout:

stderr:
Traceback (most recent call last):
  File "/home/maier/.cache/pants/named_caches/pex_root/unzipped_pexes/01a8d22400b26b1d132a9849fb6e28962f1ecf9e/.bootstrap/pex/pex.py", line 476, in execute
    exit_value = self._wrap_coverage(self._wrap_profiling, self._execute)
  File "/home/maier/.cache/pants/named_caches/pex_root/unzipped_pexes/01a8d22400b26b1d132a9849fb6e28962f1ecf9e/.bootstrap/pex/pex.py", line 401, in _wrap_coverage
    return runner(*args)
  File "/home/maier/.cache/pants/named_caches/pex_root/unzipped_pexes/01a8d22400b26b1d132a9849fb6e28962f1ecf9e/.bootstrap/pex/pex.py", line 432, in _wrap_profiling
    return runner(*args)
  File "/home/maier/.cache/pants/named_caches/pex_root/unzipped_pexes/01a8d22400b26b1d132a9849fb6e28962f1ecf9e/.bootstrap/pex/pex.py", line 532, in _execute
    return self.execute_entry(self._pex_info.entry_point)
  File "/home/maier/.cache/pants/named_caches/pex_root/unzipped_pexes/01a8d22400b26b1d132a9849fb6e28962f1ecf9e/.bootstrap/pex/pex.py", line 668, in execute_entry
    return self.execute_pkg_resources(entry_point)
  File "/home/maier/.cache/pants/named_caches/pex_root/unzipped_pexes/01a8d22400b26b1d132a9849fb6e28962f1ecf9e/.bootstrap/pex/pex.py", line 700, in execute_pkg_resources
    return runner()
  File "/home/maier/.cache/pants/named_caches/pex_root/installed_wheels/7fea57708be4117c2d962e01abd1585a8846dc06/pex-2.1.54-py2.py3-none-any.whl/pex/bin/pex.py", line 672, in main
    env=env,
  File "/home/maier/.cache/pants/named_caches/pex_root/installed_wheels/7fea57708be4117c2d962e01abd1585a8846dc06/pex-2.1.54-py2.py3-none-any.whl/pex/bin/pex.py", line 692, in do_main
    cache=ENV.PEX_ROOT,
  File "/home/maier/.cache/pants/named_caches/pex_root/installed_wheels/7fea57708be4117c2d962e01abd1585a8846dc06/pex-2.1.54-py2.py3-none-any.whl/pex/bin/pex.py", line 600, in build_pex
    pex_builder.add_distribution(installed_dist.distribution)
  File "/home/maier/.cache/pants/named_caches/pex_root/installed_wheels/7fea57708be4117c2d962e01abd1585a8846dc06/pex-2.1.54-py2.py3-none-any.whl/pex/pex_builder.py", line 458, in add_distribution
    dist_hash = self._add_dist_dir(dist.location, dist_name)
  File "/home/maier/.cache/pants/named_caches/pex_root/installed_wheels/7fea57708be4117c2d962e01abd1585a8846dc06/pex-2.1.54-py2.py3-none-any.whl/pex/pex_builder.py", line 421, in _add_dist_dir
    self._copy_or_link(path, target_dir, label=dist_name)
  File "/home/maier/.cache/pants/named_caches/pex_root/installed_wheels/7fea57708be4117c2d962e01abd1585a8846dc06/pex-2.1.54-py2.py3-none-any.whl/pex/pex_builder.py", line 541, in _copy_or_link
    self._chroot.symlink(src, dst, label)
  File "/home/maier/.cache/pants/named_caches/pex_root/installed_wheels/7fea57708be4117c2d962e01abd1585a8846dc06/pex-2.1.54-py2.py3-none-any.whl/pex/common.py", line 686, in symlink
    os.symlink(abs_src, abs_dst)
FileExistsError: [Errno 17] File exists: '/home/maier/.cache/pants/named_caches/pex_root/installed_wheels/ac5b86417eca78aae3e75284a7a6ce9d6127efa5/boto3-1.20.10-py3-none-any.whl' -> '/tmp/process-executionbSILsu/.tmp/tmpshxk3mfn/.deps/boto3-1.20.10-py3-none-any.whl'
The relevant
BUILD
file has the following contents:
Copy code
python_sources(
    name="graplctl_lib",
    dependencies=[
        "src/python/grapl-common:grapl-common",
    ],
)

pex_binary(
    entry_point="cli.py:main",
)
Not quite sure where things are going sideways... any help is appreciated 🙇
c
👀 1
I had a very similar
FileExists
error, when including a local python dist in a pex.
b
@curved-television-6568 cool, thanks for the pointer
👍 1
h
For that to be the fix, I think it would mean that you depend on a
python_distribution
in the transitive closure. Is that the case? I think you could run something like
./pants dependencies --transitive path/to/dir:dir
(you left off
name=
, so it has the shorthand), then
xargs ./pants filter --target-type=python_distribution
c
If not, I suggest revisiting https://github.com/pantsbuild/pex/pull/1517
b
Yes, we've got a
python_distribution
(it's the one thing declared in
dependencies
above)
👍 1
Is there anything I can do as a workaround for this, aside from going back to an earlier Pants release?
c
I think you could exclude the python dist from the pex, perhaps, with
!!src/python/grapl-common:grapl-common
as dependency on the
pex_binary
target.. as earlier versions of pants didn’t treat a dependency on the dist in any way, and now it packages it, which is new. Not 100% sure how accurate what I just wrote is, but worth a shot? 😉
👍 1
➕ 1
b
🤔 That may have actually worked...
😆 1
😅 1
Running it through CI to verify 😄
🙏 2
👍 1