Hi, I'm trying to build a lambda zip using the

python_awslambda

in my Mac but I'm getting this error when I run

./pants package appcode/api/transaction_details/src/main.py

:

pip._vendor.packaging.markers.UndefinedEnvironmentName: '*python_full_version' does not exist in evaluation environment.*

pid 81510 -> /Users/rifonseca/.cache/pants/named_caches/pex_root/venvs/44f0d229c64d262df3c9196eaa181a922a2a17cd/81d027995b16273d21e19dc7b1441abee1a3e3d1/pex --disable-pip-version-check --no-python-version-warning --exists-action a --isolated -q --cache-dir /Users/rifonseca/.cache/pants/named_caches/pex_root --log /private/var/folders/63/28d_yhnj2tn6f3xg1g_mp0lr0000gq/T/process-executionnyaMyQ/.tmp/tmp47qnomzj/pip.log download --dest /private/var/folders/63/28d_yhnj2tn6f3xg1g_mp0lr0000gq/T/process-executionnyaMyQ/.tmp/tmphaqx3pp0/cp39-cp39-linux_x86_64 --only-binary all --no-deps --requirement 3rdparty/python/default.lock --platform manylinux2014_x86_64 --platform linux_x86_64 --implementation cp --python-version 39 --abi cp39 --index-url https://pypi.org/simple/ --retries 5 --timeout 15 exited with 2 and STDERR:

Failed to resolve for platform linux_x86_64-cp-39-cp39. Resolve requires evaluation of unknown environment marker: 'python_full_version' does not exist in evaluation environment.

Reading through the docs, I found this:

Running from macOS and failing to build?

AWS Lambdas must run on Linux, so Pants tells PEX and Pip to build for Linux when resolving your third party dependencies. This means that you can only use pre-built wheels (bdists). If your project requires any source distributions (sdists) that must be built locally, PEX and pip will fail to run.

If this happens, you must either change your dependencies to only use dependencies with pre-built wheels or find a Linux environment to run

./pants package

.

Wondering how to do I tell

./pant package....

to only use pre-built wheels? I know if I was using

pip

directly, I could pass the

--only-binary=:all

flag.

Hello! Welcome! You will want to use this field to say the specific Linux version. Pants already says to only use wheels for Linux, but it looks like one of your requirements needs something more precise https://www.pantsbuild.org/docs/reference-python_awslambda#codecomplete_platformscode

Thanks for quick response! I ran

pex3 interpreter inspect --markers --tags

to generate a complete platform json and then used it to create a

complete_platform.json

in the local directory containing the generated

marker_environment

and

compatible_tags

. I then updated my

python_awslambda

with

complete_platforms=["./complete_platforms.json"]

. I'm now getting this error:

ValueError: The address

appcode/api/transaction_details/src/complete_platforms.json

is not generated by the

python_sources

target

appcode/api/transaction_details/src:src

, which only generates these addresses:

*appcode/api/transaction_details/src/__init__.py

*appcode/api/transaction_details/src/main.py

Did you mean to use one of those addresses?

Thoughts?

Ah I'm not sure why it was modeled this way:

Complete platforms should be addresses of file targets that point to files that contain complete platform JSON as described by Pex

Meaning, add this:

file(name="platforms", source="./complete_platforms.json")

Then, set

complete_platforms=[":platforms"])

After making the suggested changes, I'm now back to the original error 🙂 ....

pip._vendor.packaging.markers.UndefinedEnvironmentName: 'python_full_version' does not exist in evaluation environment.
pid 32167 -> /Users/rifonseca/.cache/pants/named_caches/pex_root/venvs/44f0d229c64d262df3c9196eaa181a922a2a17cd/81d027995b16273d21e19dc7b1441abee1a3e3d1/pex --disable-pip-version-check --no-python-version-warning --exists-action a --isolated -q --cache-dir /Users/rifonseca/.cache/pants/named_caches/pex_root --log /private/var/folders/63/28d_yhnj2tn6f3xg1g_mp0lr0000gq/T/process-execution44HIOZ/.tmp/tmpse5ogntr/pip.log download --dest /private/var/folders/63/28d_yhnj2tn6f3xg1g_mp0lr0000gq/T/process-execution44HIOZ/.tmp/tmp0ffclkld/cp39-cp39-linux_x86_64 --only-binary :all: --no-deps --requirement 3rdparty/python/default.lock --platform manylinux2014_x86_64 --platform linux_x86_64 --implementation cp --python-version 39 --abi cp39 --index-url <https://pypi.org/simple/> --retries 5 --timeout 15 exited with 2 and STDERR:
Failed to resolve for platform linux_x86_64-cp-39-cp39. Resolve requires evaluation of unknown environment marker: 'python_full_version' does not exist in evaluation environment.

Here is what the updated BUILD file looks like:

pex_binary(
    name="main",
    entry_point="main.py",
)

python_sources()

file(name="platforms", source="./complete_platforms.json")

python_awslambda(
    name="lambda",
    runtime="python3.9",
    # Pants will convert this to `project.lambda_example:example_handler`.
    handler="main.py:handler",
    complete_platforms=[":platforms"]
)

Hm, sorry you're having a negative experience here! I think this might be a bug, thank you for reporting it. Would you be able to share your third-party dependencies? I am curious which one is causing this so that we can reproduce. No worries if you cannot share

No worries at all and thank you for all of the help. Here is what my

requirements.txt

looks like:

Faker==13.4.0
pytest~=7.

Partial bug / partial doc bug. You need to delete your runtime setting. It's either runtime or complete_pmatform.

Removing the runtime got rid of the error but I'm not sure it is working as expected? When I tried to unzip the generated zip file, I get an

Unable to expand "lambda.zip". It is in an unsupported format

error. When I upload the generated zip to lambda, I notice that Faker is missing from the list of dependencies which is confirm by the error I get when I run the lambda:

"errorMessage": "Unable to import module 'main': No module named 'faker'"

You need to delete your runtime setting. It's either runtime or complete_pmatform.

@enough-analyst-54434 our error message is:

The `{self.alias}` target {self.address} must specify either a
                    `{self[PythonAwsLambdaRuntime].alias}` or
                    `{self[PexCompletePlatformsField].alias}` or both.

So, it was intentional, but a mistake it sounds like?

No, I think not. Since complete_platforms itself is plural, that says it all. You can try and build a multiplatform lambda just like a multiplatform PEX. You probably don't want that though.

@hundreds-father-404 / @enough-analyst-54434 - any thoughts why I can't unzip the generated

lambda.zip

and it doesn't seem to have the third-party dependencies (e.g. Faker)?

Nope. That part is strange. If at all possible to share ~full details, opening an issue with the generated lambda.zip, etc might be best.

I open issue: https://github.com/pantsbuild/pants/issues/15239. Thank you for all of the help @hundreds-father-404 and @enough-analyst-54434!

Thank you for posting all that information! That is so helpful to debug. It would be helpful if you are able to post the entire contents of platforms.json rather than using

...

Issued updated with the entire content of the platforms.json. Thank you gain!

Thanks, that data revealed a basic misunderstanding. Noted on issue. Perhaps correcting that will solve the zip file issue magically, although it shouldn't. Nonetheless, one thing that needs to be fixed anyhow.

That makes sense although I agree with you in that this is probably a separate issue. I'll make the change as it is needed not matter what. Thanks for pointing it out!

Pants could probably include those, since they are not likely to change frequently.

Not yet. I tried using ec2 instances and lambda docker based images but didn't get too far. Any help will be greatly appreciated!!

To clarify, you're deploying your lambda as a .zip, not as an image, so you need to know what image the lambda framework chooses to run your .zip in?

I was trying to get an instance that mirror the host used by lambda so that I could run the pex cmd. Didn't realize I could actually run the command programmatically within the lambda

Yeah, depend on

pex==2.1.82

(say) as a requirement, and then do something like this in the function:

Thanks @happy-kitchen-89482 - that worked!. I'm now getting the exception below even though the json within the file is valid. If I just list one of the platforms files (doesn't matter which), I get the original error message. This makes me think it may have to do with how I'm listing multiple files?

rgparse.ArgumentTypeError: Failed to load complete platform data from appcode/api/transaction_details/src/lambda_complete_platform.json: Expecting property name enclosed in double quotes: line 574 column 1 (char 21573)

BUILD.bazel:

pex_binary(
    name="main",
    entry_point="main.py",
)

python_sources()

file(name="mac_complete_platform", source="./mac_complete_platform.json")
file(name="lambda_complete_platform", source="./lambda_complete_platform.json")

python_awslambda(
    name="lambda",
    handler="main.py:handler",
    runtime="python3.9",
    complete_platforms=[
        ":lambda_complete_platform",
        ":mac_complete_platform"
        ]
)

@adorable-thailand-45835 can you instead attach the file? The error message indicates Pex is seeing this as a single file and trying to parse it using

json.load

and failing: https://github.com/pantsbuild/pex/blob/9b889ff72f8dd90828dfc6ef0006e1a8388b885b/pex/resolve/target_options.py#L208-L217

@enough-analyst-54434 - attached is the file!

@adorable-thailand-45835 I'm not sure what to say. That file appears to be valid JSON and line 574 column 1 is the closing

}

at the end. I'll see if I can come up with a basic repro.

I'm not sure either @enough-analyst-54434. When I list one of the files by itself, I get a different error which makes me wonder if it is not the file itself but how we are reading in the list of files? For example:

#BAZEL.bazel
python_awslambda(
    name="lambda",
    handler="main.py:handler",
    runtime="python3.9",
    complete_platforms=[
        ":lambda_complete_platform",
        ]
)

This results in the following error:

pip._vendor.packaging.markers.UndefinedEnvironmentName: 'python_full_version' does not exist in evaluation environment.
pid 8102 -> /Users/rifonseca/.cache/pants/named_caches/pex_root/venvs/44f0d229c64d262df3c9196eaa181a922a2a17cd/81d027995b16273d21e19dc7b1441abee1a3e3d1/pex --disable-pip-version-check --no-python-version-warning --exists-action a --isolated -q --cache-dir /Users/rifonseca/.cache/pants/named_caches/pex_root --log /private/var/folders/63/28d_yhnj2tn6f3xg1g_mp0lr0000gq/T/process-executionF1LKUc/.tmp/tmp57oxch7i/pip.log download --dest /private/var/folders/63/28d_yhnj2tn6f3xg1g_mp0lr0000gq/T/process-executionF1LKUc/.tmp/tmpzpshq15r/cp39-cp39-linux_x86_64 --only-binary :all: --no-deps --requirement 3rdparty/python/default.lock --platform manylinux2014_x86_64 --platform linux_x86_64 --implementation cp --python-version 39 --abi cp39 --index-url <https://pypi.org/simple/> --retries 5 --timeout 15 exited with 2 and STDERR:
Failed to resolve for platform linux_x86_64-cp-39-cp39. Resolve requires evaluation of unknown environment marker: 'python_full_version' does not exist in evaluation environment.

I get the same error if I list the

mac_complete_platform

file by itself. Attache is the mac file.

@adorable-thailand-45835 you may have missed it in the jungle above, but you must delete the

runtime="python3.9"

target attribute.

That was it! I'm now able to generate the zip, upload it to Lambda and successfully run the function. Woot, Woot!!!! Thank you sooo much for all of help and support @enough-analyst-54434, @happy-kitchen-89482 and @hundreds-father-404!!!!!

Excellent. Much too painful though and there are still bugs here? (using multiple complete_platforms addresses). Some cleanup yet to do to make this better.

I think it would be helpful to include the

complete_platform

files for the different AWS Lambda hosts as part of the

python_awslambda

integration. It may also not hurt to update the docs with the exact commands on how to generate the platform files on dev machines (e.g. macs).

This is a good idea. Can you open a ticket for it? Actually it might make a good first issue for a new contributor (hint... hint... 😉 )

It looks like the complete platforms idea may get traction / a PEP: https://github.com/pypa/pip/issues/10050#issuecomment-1113595053