I’ve run into an issue with AWS Lambda + the 3.11 runtime + Lambdex - I’m getting a dependency resolution error in my Lambda logs:

[ERROR] ResolveError: Failed to resolve requirements from PEX environment @ /var/task.
Needed cp311-cp311-manylinux_2_26_x86_64 compatible dependencies for:
 1: cryptography>=1.8.1
    Required by:
      dynamodb-encryption-sdk 3.2.0
    But this pex had no ProjectName(raw='cryptography', normalized='cryptography') distributions.
Traceback (most recent call last):
  File "/var/lang/lib/python3.11/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1204, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1176, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1147, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 690, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 940, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/var/task/lambdex_handler.py", line 34, in <module>
    bootstrap_pex_env(__entry_point__)
  File "/var/task/.bootstrap/pex/pex_bootstrapper.py", line 702, in bootstrap_pex_env
    PEXEnvironment.mount(entry_point, pex_info).activate()
  File "/var/task/.bootstrap/pex/environment.py", line 321, in activate
    self._activated_dists = self._activate()
  File "/var/task/.bootstrap/pex/environment.py", line 671, in _activate
    resolved = self.resolve()
  File "/var/task/.bootstrap/pex/environment.py", line 502, in resolve
    for fingerprinted_distribution in self.resolve_dists(all_reqs)
  File "/var/task/.bootstrap/pex/environment.py", line 589, in resolve_dists
    raise ResolveError(

cryptography

is indeed bundled in the

.deps

directory of the ZIP file at:

cryptography-41.0.3-cp37-abi3-manylinux_2_28_x86_64.whl

EDIT: not sure if the clue lies in

Needed cp311-cp311-manylinux_2_26_x86_64 compatibile dependencies

and the

cryptography

distribution included is

-cp37-abi3-

? if this is the case, do I need to adjust my

interpreter_constraints

in

pants.toml

?

yeah, that clue is exactly right. The most reliable way to build these is to set the

complete_platforms

parameter to

python_awslambda

. https://github.com/pantsbuild/pants/discussions/18756 walks through what we do

this is great, thank you so much @broad-processor-92400!

Ah, yeah, we've been "lucky" with having wheel-only dependencies, no sdist-only ones, i.e. everything has been precompiled. The JSON file is a description of reality of the target environment, i.e. what's supported in Lambda. Adding something to the file manually is essentially making Pants/PEX believe that the target supports something it doesn't actually support, and so may lead to errors like the original one you have. There's a three broad approaches: 1. remove/replace the dependency 😅 2. ensure a wheel exists, so pants doesn't need to run the compilation as part of the build process: ◦ either work with upstream to publish one to PyPI (not always possible), and there's tools like https://cibuildwheel.readthedocs.io/en/stable/ that make this 'easy' ◦ publish one to a custom index and point pants to that index in addition to normal PyPI (I've never done this, so I don't know the details, but https://www.pantsbuild.org/docs/python-third-party-dependencies#custom-repositories is a starting point) 3. instead of using

complete_platforms

, run the build within a docker image matching the target environment (likely one of the

sam

ones, e.g.

public.ecr.aws/sam/build-python3.11

), with the 'environments' feature in pants https://www.pantsbuild.org/docs/environments, meaning the dependency is built on every run, and built in a compatible way. As I said, I've not encountered this so can't speak to the details. Options 1 or 2 will result in faster builds compared to option 3, by avoiding needing to do the C compilation on every single

pants package

call.

Excellent, I appreciate very much the detailed writeup here - I took a look at that linked issue and I might just git pull it and run the tests against py3.11.4 to see if any fail - if any do I think option #1 may be best here - I’ve done exactly 0 work on publishing python packages so I’m already a bit over my skis here. If all of the tests pass with Py3.11.4 then I may just hack the tags list until we wind up doing #1 anyway (or the author decides to update the wheels on their own)