We've started using

complete_platforms

instead of

runtime

for

python_awslambda

zips. I don't know the original reason my team switched but I did find this discussion: https://github.com/pantsbuild/pants/discussions/18756 Would it be helpful to have the various json files with the output for different lambda runtimes stored somewhere? I have been using the aws docker images to generate them and I think this process could be automated. Or are there too many variables in those files? Do they differ across aws regions, etc.

Yeah, it would be helpful! It’s been added to pants 2.18, and so will come out with that soon-ish https://github.com/pantsbuild/pants/discussions/19544 walks through that and some other improvements to lambdas in 2.17 (releasing soon, I hope) and 2.18

If it weren't for the insanity of including

platform_version

in PEP-508 (https://peps.python.org/pep-0508/#environment-markers) these files could be pre-generated for ~all Pythons on Linux and Mac and hosted centrally. I'm not sure if

platform_version

is used in the wild, but experience says yes - all these features are used or abused at least once.

Ok. I am able to generate the json locally fairly easily. I used docker images but I could try to run these on actual lambdas if necessary. I just don't have enough knowledge on this to know if running the aws lambda docker images are sufficient.

I think they are, but@broad-processor-92400 also outlines a simple lambda function to generate the json somewhere. That sidesteps being unsure by actually generating the json out on a lambda runtime instance inside AWS. No magic - you could guess it yourself I imagine if you can generate it locally!

Locally I ran like this:

import json
import subprocess


def lambda_handler(event, context):
    output_file_path = "/tmp/subdir/output.txt"

    with open(output_file_path, "w") as output:
        subprocess.run(
            """
            pip install --target=/tmp/subdir pex
            PYTHONPATH=/tmp/subdir /tmp/subdir/bin/pex3 interpreter inspect --markers --tags
            """,
            shell=True,
            stdout=output,
        )

    with open(output_file_path, "r") as output_file:
        output_lines = output_file.readlines()
        last_line = output_lines[-1].strip()

    try:
        body = json.loads(last_line)

    except Exception:
        pass

    return {'statusCode': 200, 'body': body if body else {""}}

Yeah, so you could clearly run that handler out in AWS if you want to be super-sure. But I do think the images match the AMI in all the important ways (CPython version + glibc version).

So if I hypothetically generate these for several runtimes (3.8, 3.9, …) and architectures (arm, x86_64), is there a convenient place to store them all?

As @broad-processor-92400 said - this exists / has been done for Pants 2.18: https://github.com/pantsbuild/pants/tree/main/src/python/pants/backend/awslambda/python @cold-jackal-89755 you can of course store your complete platforms in your repo and then just throw them away once you can upgrade to Pants 2.18+.

Ah. I see. Didn’t read the reply correctly