https://pantsbuild.org/ logo
Join Slack
Powered by
I'm a lost :dog: . :thread: for more details. Whe...
# general
g
I'm a lost 🐶 . 🧵 for more details. When I try to build a pex I get this error. When I try installing these in the container natively via pip it just works. I don't know how to diagnose the root cause of this.
Copy code
raise ProcessExecutionFailure(
pants.engine.process.ProcessExecutionFailure: Process 'Building 2 requirements for apps.backups-s3-lambda/pex_binary.pex from the apps/pants.lock resolve: awslambdaric<3.0.0,>=2.0.4, datadog-lambda<6.0.0,>=5' failed with exit code 1.
stdout:

stderr:
There was 1 error downloading required artifacts:
1. awslambdaric 2.0.12 from <https://company-123456789012.d.codeartifact.us-east-1.amazonaws.com/pypi/altana/simple/awslambdaric/2.0.12/awslambdaric-2.0.12.tar.gz>
    hint: See above for details.
This works without any issues:
Copy code
pip install 'awslambdaric>=2.0.4,<3.0.0' 'datadog-lambda>=5.0.0,<6.0.0'
full log:
Copy code
15:23:01.66 [INFO] Completed: Building 2 requirements for apps.backups-s3-lambda/pex_binary.pex from the apps/pants.lock resolve: awslambdaric<3.0.0,>=2.0.4, datadog-lambda<6.0.0,>=5
15:23:01.66 [ERROR] 1 Exception encountered:

Engine traceback:
  in root
    ..
  in pants.core.goals.package.package_asset
    `package` goal

Traceback (most recent call last):
  File "/Users/jasonbarnett/Library/Caches/nce/f46d2c12132bad9c27e0dd509186c190371abfe1add7e5ca42245466f35bea81/bindings/venvs/2.21.0/lib/python3.9/site-packages/pants/core/goals/package.py", line 165, in package_asset
    packages = await MultiGet(
  File "/Users/jasonbarnett/Library/Caches/nce/f46d2c12132bad9c27e0dd509186c190371abfe1add7e5ca42245466f35bea81/bindings/venvs/2.21.0/lib/python3.9/site-packages/pants/engine/internals/selectors.py", line 376, in MultiGet
    return await _MultiGet(tuple(__arg0))
  File "/Users/jasonbarnett/Library/Caches/nce/f46d2c12132bad9c27e0dd509186c190371abfe1add7e5ca42245466f35bea81/bindings/venvs/2.21.0/lib/python3.9/site-packages/pants/engine/internals/selectors.py", line 174, in __await__
    result = yield self.gets
  File "/Users/jasonbarnett/Library/Caches/nce/f46d2c12132bad9c27e0dd509186c190371abfe1add7e5ca42245466f35bea81/bindings/venvs/2.21.0/lib/python3.9/site-packages/pants/core/goals/package.py", line 116, in environment_aware_package
    package = await Get(
  File "/Users/jasonbarnett/Library/Caches/nce/f46d2c12132bad9c27e0dd509186c190371abfe1add7e5ca42245466f35bea81/bindings/venvs/2.21.0/lib/python3.9/site-packages/pants/engine/internals/selectors.py", line 124, in __await__
    result = yield self
  File "/Users/jasonbarnett/Library/Caches/nce/f46d2c12132bad9c27e0dd509186c190371abfe1add7e5ca42245466f35bea81/bindings/venvs/2.21.0/lib/python3.9/site-packages/pants/backend/python/goals/package_pex_binary.py", line 178, in built_pacakge_for_pex_from_targets_request
    pex = await Get(Pex, PexFromTargetsRequest, pft_request)
  File "/Users/jasonbarnett/Library/Caches/nce/f46d2c12132bad9c27e0dd509186c190371abfe1add7e5ca42245466f35bea81/bindings/venvs/2.21.0/lib/python3.9/site-packages/pants/engine/internals/selectors.py", line 124, in __await__
    result = yield self
  File "/Users/jasonbarnett/Library/Caches/nce/f46d2c12132bad9c27e0dd509186c190371abfe1add7e5ca42245466f35bea81/bindings/venvs/2.21.0/lib/python3.9/site-packages/pants/backend/python/util_rules/pex.py", line 826, in create_pex
    result = await Get(BuildPexResult, PexRequest, request)
  File "/Users/jasonbarnett/Library/Caches/nce/f46d2c12132bad9c27e0dd509186c190371abfe1add7e5ca42245466f35bea81/bindings/venvs/2.21.0/lib/python3.9/site-packages/pants/engine/internals/selectors.py", line 124, in __await__
    result = yield self
  File "/Users/jasonbarnett/Library/Caches/nce/f46d2c12132bad9c27e0dd509186c190371abfe1add7e5ca42245466f35bea81/bindings/venvs/2.21.0/lib/python3.9/site-packages/pants/backend/python/util_rules/pex.py", line 749, in build_pex
    result = await Get(
  File "/Users/jasonbarnett/Library/Caches/nce/f46d2c12132bad9c27e0dd509186c190371abfe1add7e5ca42245466f35bea81/bindings/venvs/2.21.0/lib/python3.9/site-packages/pants/engine/internals/selectors.py", line 124, in __await__
    result = yield self
  File "/Users/jasonbarnett/Library/Caches/nce/f46d2c12132bad9c27e0dd509186c190371abfe1add7e5ca42245466f35bea81/bindings/venvs/2.21.0/lib/python3.9/site-packages/pants/engine/process.py", line 314, in fallible_to_exec_result_or_raise
    raise ProcessExecutionFailure(
pants.engine.process.ProcessExecutionFailure: Process 'Building 2 requirements for apps.backups-s3-lambda/pex_binary.pex from the apps/pants.lock resolve: awslambdaric<3.0.0,>=2.0.4, datadog-lambda<6.0.0,>=5' failed with exit code 1.
stdout:

stderr:
There was 1 error downloading required artifacts:
1. awslambdaric 2.0.12 from <https://company-123456789012.d.codeartifact.us-east-1.amazonaws.com/pypi/altana/simple/awslambdaric/2.0.12/awslambdaric-2.0.12.tar.gz>
    hint: See above for details.
If I don't use environment in the pex_binary target, it works. It keeps failing when using linux for some reason.
Copy code
pex_binary(
    name="pex_binary",
    entry_point="<http://backups_s3_lambda.app:lambda_handler|backups_s3_lambda.app:lambda_handler>",
    venv_hermetic_scripts=False,
    complete_platforms=[
        "3rdparty/platforms:docker_python_3_9_bookworm",
        "3rdparty/platforms:python_3_9_mac_os_arm64",
    ],
    dependencies=["apps/backups-s3-lambda:poetry#awslambdaric"],
    environment="linux", # Works if I comment this out
    layout="packed",
    execution_mode="venv",
Copy code
# pants.toml
[environments-preview.names]
linux = "//:local_linux"
linux_python_docker = "//:local_python_39_slim_bullseye"
Copy code
# BUILD
__defaults__(all=dict(environment="linux"))

local_environment(
    name="local_linux",
    compatible_platforms=["linux_x86_64"],
    fallback_environment="linux_python_docker",
)

docker_environment(
    name="local_python_39_slim_bullseye",
    platform="linux_x86_64",
    image="python:3.9-slim-bullseye@sha256:9ac27d4ecadc3ef02f980a8e2b37c7e8cdbf24039c5eddd9f98aff00d23b9e03",
)
So I hacked into the docker container and modified the __run.sh in the sandbox and dropped the other two --complete-platform flags that weren't linux, and it worked.
So even though I got it working in the hacked way, I'm not sure how to fix this.
well removing the mac os in complete_platforms arg in pex_binary target fixed it 🤷
w
On reading this, I was just gonna say that this looked like something similar from yesterday or so, so I was going to just chime in with 
complete_platforms
and see where that went: https://pantsbuild.slack.com/archives/C046T6T9U/p1719325406448559 Also, one idea to try, if you still want a mac variant - maybe look into 
parametrize
? That way, you can make two binaries in isolation, but still similarly named?
👀 1
g
@wide-midnight-78598 Thanks for the comments. That's a good idea if I still need both. I mostly had both for compatibility between both and so having a path forward for a different platform will be nice.
h
So, you're running on linux, I assume?
I would expect building a macos wheel from an sdist on linux to fail in all but trivial cases. If there are no published wheels (for all transitive deps...) for a foreign platform, then building a multiplatform pex in this way isn't going to work.
Or are there published wheels, and it's downloading the sdist unnecessarily?
That "There was 1 error downloading required artifacts:" error seems to indicate that the issue was with the download itself, so you could investigate that on the server logs if you have access to them. But I suspect (without evidence, just a hunch) that the error is misleading and the real issue is building the sdist, not the download.
👍 1
g
I was building on Mac OS using environments-preview and building it in a linux container.
h
So, building a multiplatform pex on MacOS in a Linux container
👍 1
Sounds masochistic...
What happens if you build natively on MacOS?
2 Views
Open in Slack
PreviousNext
Powered by