I just upgraded from Pants 2 7 1 to 2 8 0 and have a confusi Pants #general

I just upgraded from Pants 2.7.1 to 2.8.0 and have...

brash-baker-91190

11/22/2021, 4:16 PM

I just upgraded from Pants 2.7.1 to 2.8.0 and have a confusing failure when building a

pex_binary

(see thread)

brash-baker-91190

11/22/2021, 4:17 PM

Copy code

11:15:27.39 [ERROR] 1 Exception encountered:

  ProcessExecutionFailure: Process 'Extracting 16 requirements to build src.python.graplctl.graplctl/graplctl.pex from repository.pex: boto3, botocore, click, mypy_boto3_cloudwatch, mypy_boto3_dynamodb, mypy_boto3_ec2, mypy_boto3_lambda, mypy_boto3_route53, mypy_boto3_s3, mypy_boto3_secretsmanager, mypy_boto3_sns, mypy_boto3_sqs, mypy_boto3_ssm, protobuf>=3.12.1, typing_extensions, zstd==1.4.9.1' failed with exit code 1.
stdout:

stderr:
Traceback (most recent call last):
  File "/home/maier/.cache/pants/named_caches/pex_root/unzipped_pexes/01a8d22400b26b1d132a9849fb6e28962f1ecf9e/.bootstrap/pex/pex.py", line 476, in execute
    exit_value = self._wrap_coverage(self._wrap_profiling, self._execute)
  File "/home/maier/.cache/pants/named_caches/pex_root/unzipped_pexes/01a8d22400b26b1d132a9849fb6e28962f1ecf9e/.bootstrap/pex/pex.py", line 401, in _wrap_coverage
    return runner(*args)
  File "/home/maier/.cache/pants/named_caches/pex_root/unzipped_pexes/01a8d22400b26b1d132a9849fb6e28962f1ecf9e/.bootstrap/pex/pex.py", line 432, in _wrap_profiling
    return runner(*args)
  File "/home/maier/.cache/pants/named_caches/pex_root/unzipped_pexes/01a8d22400b26b1d132a9849fb6e28962f1ecf9e/.bootstrap/pex/pex.py", line 532, in _execute
    return self.execute_entry(self._pex_info.entry_point)
  File "/home/maier/.cache/pants/named_caches/pex_root/unzipped_pexes/01a8d22400b26b1d132a9849fb6e28962f1ecf9e/.bootstrap/pex/pex.py", line 668, in execute_entry
    return self.execute_pkg_resources(entry_point)
  File "/home/maier/.cache/pants/named_caches/pex_root/unzipped_pexes/01a8d22400b26b1d132a9849fb6e28962f1ecf9e/.bootstrap/pex/pex.py", line 700, in execute_pkg_resources
    return runner()
  File "/home/maier/.cache/pants/named_caches/pex_root/installed_wheels/7fea57708be4117c2d962e01abd1585a8846dc06/pex-2.1.54-py2.py3-none-any.whl/pex/bin/pex.py", line 672, in main
    env=env,
  File "/home/maier/.cache/pants/named_caches/pex_root/installed_wheels/7fea57708be4117c2d962e01abd1585a8846dc06/pex-2.1.54-py2.py3-none-any.whl/pex/bin/pex.py", line 692, in do_main
    cache=ENV.PEX_ROOT,
  File "/home/maier/.cache/pants/named_caches/pex_root/installed_wheels/7fea57708be4117c2d962e01abd1585a8846dc06/pex-2.1.54-py2.py3-none-any.whl/pex/bin/pex.py", line 600, in build_pex
    pex_builder.add_distribution(installed_dist.distribution)
  File "/home/maier/.cache/pants/named_caches/pex_root/installed_wheels/7fea57708be4117c2d962e01abd1585a8846dc06/pex-2.1.54-py2.py3-none-any.whl/pex/pex_builder.py", line 458, in add_distribution
    dist_hash = self._add_dist_dir(dist.location, dist_name)
  File "/home/maier/.cache/pants/named_caches/pex_root/installed_wheels/7fea57708be4117c2d962e01abd1585a8846dc06/pex-2.1.54-py2.py3-none-any.whl/pex/pex_builder.py", line 421, in _add_dist_dir
    self._copy_or_link(path, target_dir, label=dist_name)
  File "/home/maier/.cache/pants/named_caches/pex_root/installed_wheels/7fea57708be4117c2d962e01abd1585a8846dc06/pex-2.1.54-py2.py3-none-any.whl/pex/pex_builder.py", line 541, in _copy_or_link
    self._chroot.symlink(src, dst, label)
  File "/home/maier/.cache/pants/named_caches/pex_root/installed_wheels/7fea57708be4117c2d962e01abd1585a8846dc06/pex-2.1.54-py2.py3-none-any.whl/pex/common.py", line 686, in symlink
    os.symlink(abs_src, abs_dst)
FileExistsError: [Errno 17] File exists: '/home/maier/.cache/pants/named_caches/pex_root/installed_wheels/ac5b86417eca78aae3e75284a7a6ce9d6127efa5/boto3-1.20.10-py3-none-any.whl' -> '/tmp/process-executionbSILsu/.tmp/tmpshxk3mfn/.deps/boto3-1.20.10-py3-none-any.whl'

brash-baker-91190

11/22/2021, 4:18 PM

The relevant

BUILD

file has the following contents:

Copy code

python_sources(
    name="graplctl_lib",
    dependencies=[
        "src/python/grapl-common:grapl-common",
    ],
)

pex_binary(
    entry_point="cli.py:main",
)

Not quite sure where things are going sideways... any help is appreciated 🙇

curved-television-6568

11/22/2021, 4:24 PM

Fix for that could be here: https://github.com/pantsbuild/pants/pull/13690

👀 1

curved-television-6568

11/22/2021, 4:26 PM

I had a very similar

FileExists

error, when including a local python dist in a pex.

brash-baker-91190

11/22/2021, 4:27 PM

@curved-television-6568 cool, thanks for the pointer

👍 1

hundreds-father-404

11/22/2021, 4:27 PM

For that to be the fix, I think it would mean that you depend on a

python_distribution

in the transitive closure. Is that the case? I think you could run something like

./pants dependencies --transitive path/to/dir:dir

(you left off

name=

, so it has the shorthand), then

xargs ./pants filter --target-type=python_distribution

curved-television-6568

11/22/2021, 4:28 PM

If not, I suggest revisiting https://github.com/pantsbuild/pex/pull/1517

brash-baker-91190

11/22/2021, 4:36 PM

Yes, we've got a

python_distribution

(it's the one thing declared in

dependencies

above)

👍 1

brash-baker-91190

11/22/2021, 4:37 PM

Is there anything I can do as a workaround for this, aside from going back to an earlier Pants release?

curved-television-6568

11/22/2021, 4:39 PM

I think you could exclude the python dist from the pex, perhaps, with

!!src/python/grapl-common:grapl-common

as dependency on the

pex_binary

target.. as earlier versions of pants didn’t treat a dependency on the dist in any way, and now it packages it, which is new. Not 100% sure how accurate what I just wrote is, but worth a shot? 😉

👍 1

➕ 1

brash-baker-91190

11/22/2021, 4:41 PM

🤔 That may have actually worked...

😆 1

😅 1

brash-baker-91190

11/22/2021, 4:44 PM

Running it through CI to verify 😄

🙏 2

👍 1

Open in Slack

Previous Next