https://pantsbuild.org/ logo
#general
Title
# general
a

adorable-thailand-45835

04/22/2022, 7:24 PM
Hi, I'm trying to build a lambda zip using the
python_awslambda
in my Mac but I'm getting this error when I run
./pants package appcode/api/transaction_details/src/main.py
:
pip._vendor.packaging.markers.UndefinedEnvironmentName: '*python_full_version' does not exist in evaluation environment.*
pid 81510 -> /Users/rifonseca/.cache/pants/named_caches/pex_root/venvs/44f0d229c64d262df3c9196eaa181a922a2a17cd/81d027995b16273d21e19dc7b1441abee1a3e3d1/pex --disable-pip-version-check --no-python-version-warning --exists-action a --isolated -q --cache-dir /Users/rifonseca/.cache/pants/named_caches/pex_root --log /private/var/folders/63/28d_yhnj2tn6f3xg1g_mp0lr0000gq/T/process-executionnyaMyQ/.tmp/tmp47qnomzj/pip.log download --dest /private/var/folders/63/28d_yhnj2tn6f3xg1g_mp0lr0000gq/T/process-executionnyaMyQ/.tmp/tmphaqx3pp0/cp39-cp39-linux_x86_64 --only-binary all --no-deps --requirement 3rdparty/python/default.lock --platform manylinux2014_x86_64 --platform linux_x86_64 --implementation cp --python-version 39 --abi cp39 --index-url https://pypi.org/simple/ --retries 5 --timeout 15 exited with 2 and STDERR:
Failed to resolve for platform linux_x86_64-cp-39-cp39. Resolve requires evaluation of unknown environment marker: 'python_full_version' does not exist in evaluation environment.
Reading through the docs, I found this:
Running from macOS and failing to build?
AWS Lambdas must run on Linux, so Pants tells PEX and Pip to build for Linux when resolving your third party dependencies. This means that you can only use pre-built wheels (bdists). If your project requires any source distributions (sdists) that must be built locally, PEX and pip will fail to run.
If this happens, you must either change your dependencies to only use dependencies with pre-built wheels or find a Linux environment to run
./pants package
.
Wondering how to do I tell
./pant package....
to only use pre-built wheels? I know if I was using
pip
directly, I could pass the
--only-binary=:all
flag.
h

hundreds-father-404

04/22/2022, 7:34 PM
Hello! Welcome! You will want to use this field to say the specific Linux version. Pants already says to only use wheels for Linux, but it looks like one of your requirements needs something more precise https://www.pantsbuild.org/docs/reference-python_awslambda#codecomplete_platformscode
Good to know that we should update the guide docs to mention this. Thank you for reporting!
a

adorable-thailand-45835

04/22/2022, 8:33 PM
Thanks for quick response! I ran
pex3 interpreter inspect --markers --tags
to generate a complete platform json and then used it to create a
complete_platform.json
in the local directory containing the generated
marker_environment
and
compatible_tags
. I then updated my
python_awslambda
with
complete_platforms=["./complete_platforms.json"]
. I'm now getting this error:
ValueError: The address
appcode/api/transaction_details/src/complete_platforms.json
is not generated by the
python_sources
target
appcode/api/transaction_details/src:src
, which only generates these addresses:
*appcode/api/transaction_details/src/__init__.py
*appcode/api/transaction_details/src/main.py
Did you mean to use one of those addresses?
Thoughts?
Here is what my BUILD file looks like:
Copy code
pex_binary(
    name="main",
    entry_point="main.py",
)

python_sources()

python_awslambda(
    name="lambda",
    runtime="python3.9", 
    handler="main.py:handler",
    complete_platforms=["./complete_platforms.json"]
)
h

hundreds-father-404

04/22/2022, 9:19 PM
Ah I'm not sure why it was modeled this way:
Complete platforms should be addresses of file targets that point to files that contain complete platform JSON as described by Pex
Meaning, add this:
Copy code
file(name="platforms", source="./complete_platforms.json")
Then, set
complete_platforms=[":platforms"])
a

adorable-thailand-45835

04/22/2022, 9:28 PM
After making the suggested changes, I'm now back to the original error 🙂 ....
Copy code
pip._vendor.packaging.markers.UndefinedEnvironmentName: 'python_full_version' does not exist in evaluation environment.
pid 32167 -> /Users/rifonseca/.cache/pants/named_caches/pex_root/venvs/44f0d229c64d262df3c9196eaa181a922a2a17cd/81d027995b16273d21e19dc7b1441abee1a3e3d1/pex --disable-pip-version-check --no-python-version-warning --exists-action a --isolated -q --cache-dir /Users/rifonseca/.cache/pants/named_caches/pex_root --log /private/var/folders/63/28d_yhnj2tn6f3xg1g_mp0lr0000gq/T/process-execution44HIOZ/.tmp/tmpse5ogntr/pip.log download --dest /private/var/folders/63/28d_yhnj2tn6f3xg1g_mp0lr0000gq/T/process-execution44HIOZ/.tmp/tmp0ffclkld/cp39-cp39-linux_x86_64 --only-binary :all: --no-deps --requirement 3rdparty/python/default.lock --platform manylinux2014_x86_64 --platform linux_x86_64 --implementation cp --python-version 39 --abi cp39 --index-url <https://pypi.org/simple/> --retries 5 --timeout 15 exited with 2 and STDERR:
Failed to resolve for platform linux_x86_64-cp-39-cp39. Resolve requires evaluation of unknown environment marker: 'python_full_version' does not exist in evaluation environment.
Here is what the updated BUILD file looks like:
Copy code
pex_binary(
    name="main",
    entry_point="main.py",
)

python_sources()

file(name="platforms", source="./complete_platforms.json")

python_awslambda(
    name="lambda",
    runtime="python3.9",
    # Pants will convert this to `project.lambda_example:example_handler`.
    handler="main.py:handler",
    complete_platforms=[":platforms"]
)
1
🤔 1
h

hundreds-father-404

04/22/2022, 9:35 PM
Hm, sorry you're having a negative experience here! I think this might be a bug, thank you for reporting it. Would you be able to share your third-party dependencies? I am curious which one is causing this so that we can reproduce. No worries if you cannot share
fyi I opened https://github.com/pantsbuild/pants/issues/15234 to track improving the UX confusion
👍 1
a

adorable-thailand-45835

04/22/2022, 9:43 PM
No worries at all and thank you for all of the help. Here is what my
requirements.txt
looks like:
Copy code
Faker==13.4.0
pytest~=7.
e

enough-analyst-54434

04/22/2022, 9:59 PM
Partial bug / partial doc bug. You need to delete your runtime setting. It's either runtime or complete_pmatform.
👍 2
👀 1
a

adorable-thailand-45835

04/22/2022, 10:19 PM
Removing the runtime got rid of the error but I'm not sure it is working as expected? When I tried to unzip the generated zip file, I get an
Unable to expand "lambda.zip". It is in an unsupported format
error. When I upload the generated zip to lambda, I notice that Faker is missing from the list of dependencies which is confirm by the error I get when I run the lambda:
"errorMessage": "Unable to import module 'main': No module named 'faker'"
h

hundreds-father-404

04/22/2022, 10:31 PM
You need to delete your runtime setting. It's either runtime or complete_pmatform.
@enough-analyst-54434 our error message is:
Copy code
The `{self.alias}` target {self.address} must specify either a
                    `{self[PythonAwsLambdaRuntime].alias}` or
                    `{self[PexCompletePlatformsField].alias}` or both.
So, it was intentional, but a mistake it sounds like?
e

enough-analyst-54434

04/22/2022, 10:54 PM
No, I think not. Since complete_platforms itself is plural, that says it all. You can try and build a multiplatform lambda just like a multiplatform PEX. You probably don't want that though.
So, error message correct, my comment above incorrect. You don't need to either or, but it's probably what you want.
a

adorable-thailand-45835

04/22/2022, 11:11 PM
@hundreds-father-404 / @enough-analyst-54434 - any thoughts why I can't unzip the generated
lambda.zip
and it doesn't seem to have the third-party dependencies (e.g. Faker)?
e

enough-analyst-54434

04/22/2022, 11:15 PM
Nope. That part is strange. If at all possible to share ~full details, opening an issue with the generated lambda.zip, etc might be best.
a

adorable-thailand-45835

04/22/2022, 11:50 PM
I open issue: https://github.com/pantsbuild/pants/issues/15239. Thank you for all of the help @hundreds-father-404 and @enough-analyst-54434!
❤️ 1
h

hundreds-father-404

04/22/2022, 11:51 PM
Thank you for posting all that information! That is so helpful to debug. It would be helpful if you are able to post the entire contents of platforms.json rather than using
...
a

adorable-thailand-45835

04/22/2022, 11:54 PM
Issued updated with the entire content of the platforms.json. Thank you gain!
👍 1
e

enough-analyst-54434

04/22/2022, 11:57 PM
Thanks, that data revealed a basic misunderstanding. Noted on issue. Perhaps correcting that will solve the zip file issue magically, although it shouldn't. Nonetheless, one thing that needs to be fixed anyhow.
a

adorable-thailand-45835

04/23/2022, 12:01 AM
That makes sense although I agree with you in that this is probably a separate issue. I'll make the change as it is needed not matter what. Thanks for pointing it out!
Any thoughts on the easiest way to run
pex3 interpreter inspect --markers --tags
command on a host/image equivalent to the x86 with Python 3.9 used by Lambda? Also wondering if pants should include platform files for host/images used by lambda out of the box?
h

happy-kitchen-89482

04/26/2022, 4:40 PM
Pants could probably include those, since they are not likely to change frequently.
And if they did you'd be no worse off than today...
@adorable-thailand-45835 Were you able to make progress, or is there still something for us to address in that issue?
In particular, did you figure out how to get the complete_platforms for the lambda image?
a

adorable-thailand-45835

04/26/2022, 4:43 PM
Not yet. I tried using ec2 instances and lambda docker based images but didn't get too far. Any help will be greatly appreciated!!
h

happy-kitchen-89482

04/26/2022, 4:48 PM
To clarify, you're deploying your lambda as a .zip, not as an image, so you need to know what image the lambda framework chooses to run your .zip in?
Seems like running the relevant pex cmd programmatically from within a lambda is the right way to do this...
a

adorable-thailand-45835

04/26/2022, 4:53 PM
I was trying to get an instance that mirror the host used by lambda so that I could run the pex cmd. Didn't realize I could actually run the command programmatically within the lambda
h

happy-kitchen-89482

04/26/2022, 4:59 PM
Yeah, depend on
pex==2.1.82
(say) as a requirement, and then do something like this in the function:
Copy code
import sys
from pex.cli import pex

sys.argv = ["pex", "interpreter", "inspect", "--markers", "--tags"]
pex.main()
That will output to stdout, so you might have to do something to capture that
Oh, you can add
"--output", "path/to/tmpfile"
to the sys.argv and then read from that file
a

adorable-thailand-45835

04/28/2022, 9:50 PM
Thanks @happy-kitchen-89482 - that worked!. I'm now getting the exception below even though the json within the file is valid. If I just list one of the platforms files (doesn't matter which), I get the original error message. This makes me think it may have to do with how I'm listing multiple files?
Copy code
rgparse.ArgumentTypeError: Failed to load complete platform data from appcode/api/transaction_details/src/lambda_complete_platform.json: Expecting property name enclosed in double quotes: line 574 column 1 (char 21573)
BUILD.bazel:
Copy code
pex_binary(
    name="main",
    entry_point="main.py",
)

python_sources()

file(name="mac_complete_platform", source="./mac_complete_platform.json")
file(name="lambda_complete_platform", source="./lambda_complete_platform.json")

python_awslambda(
    name="lambda",
    handler="main.py:handler",
    runtime="python3.9",
    complete_platforms=[
        ":lambda_complete_platform",
        ":mac_complete_platform"
        ]
)
I added the lambda_complete_platform to the issue: https://github.com/pantsbuild/pants/issues/15239#issuecomment-1112687855
e

enough-analyst-54434

04/28/2022, 10:14 PM
@adorable-thailand-45835 can you instead attach the file? The error message indicates Pex is seeing this as a single file and trying to parse it using
json.load
and failing: https://github.com/pantsbuild/pex/blob/9b889ff72f8dd90828dfc6ef0006e1a8388b885b/pex/resolve/target_options.py#L208-L217
a

adorable-thailand-45835

04/28/2022, 10:56 PM
@enough-analyst-54434 - attached is the file!
e

enough-analyst-54434

04/28/2022, 11:53 PM
@adorable-thailand-45835 I'm not sure what to say. That file appears to be valid JSON and line 574 column 1 is the closing
}
at the end. I'll see if I can come up with a basic repro.
a

adorable-thailand-45835

04/29/2022, 2:32 PM
I'm not sure either @enough-analyst-54434. When I list one of the files by itself, I get a different error which makes me wonder if it is not the file itself but how we are reading in the list of files? For example:
Copy code
#BAZEL.bazel
python_awslambda(
    name="lambda",
    handler="main.py:handler",
    runtime="python3.9",
    complete_platforms=[
        ":lambda_complete_platform",
        ]
)
This results in the following error:
Copy code
pip._vendor.packaging.markers.UndefinedEnvironmentName: 'python_full_version' does not exist in evaluation environment.
pid 8102 -> /Users/rifonseca/.cache/pants/named_caches/pex_root/venvs/44f0d229c64d262df3c9196eaa181a922a2a17cd/81d027995b16273d21e19dc7b1441abee1a3e3d1/pex --disable-pip-version-check --no-python-version-warning --exists-action a --isolated -q --cache-dir /Users/rifonseca/.cache/pants/named_caches/pex_root --log /private/var/folders/63/28d_yhnj2tn6f3xg1g_mp0lr0000gq/T/process-executionF1LKUc/.tmp/tmp57oxch7i/pip.log download --dest /private/var/folders/63/28d_yhnj2tn6f3xg1g_mp0lr0000gq/T/process-executionF1LKUc/.tmp/tmpzpshq15r/cp39-cp39-linux_x86_64 --only-binary :all: --no-deps --requirement 3rdparty/python/default.lock --platform manylinux2014_x86_64 --platform linux_x86_64 --implementation cp --python-version 39 --abi cp39 --index-url <https://pypi.org/simple/> --retries 5 --timeout 15 exited with 2 and STDERR:
Failed to resolve for platform linux_x86_64-cp-39-cp39. Resolve requires evaluation of unknown environment marker: 'python_full_version' does not exist in evaluation environment.
I get the same error if I list the
mac_complete_platform
file by itself. Attache is the mac file.
e

enough-analyst-54434

04/29/2022, 2:34 PM
@adorable-thailand-45835 you may have missed it in the jungle above, but you must delete the
runtime="python3.9"
target attribute.
a

adorable-thailand-45835

04/29/2022, 3:04 PM
That was it! I'm now able to generate the zip, upload it to Lambda and successfully run the function. Woot, Woot!!!! Thank you sooo much for all of help and support @enough-analyst-54434, @happy-kitchen-89482 and @hundreds-father-404!!!!!
e

enough-analyst-54434

04/29/2022, 3:10 PM
Excellent. Much too painful though and there are still bugs here? (using multiple complete_platforms addresses). Some cleanup yet to do to make this better.
a

adorable-thailand-45835

04/29/2022, 3:19 PM
I think it would be helpful to include the
complete_platform
files for the different AWS Lambda hosts as part of the
python_awslambda
integration. It may also not hurt to update the docs with the exact commands on how to generate the platform files on dev machines (e.g. macs).
h

happy-kitchen-89482

04/29/2022, 9:40 PM
This is a good idea. Can you open a ticket for it? Actually it might make a good first issue for a new contributor (hint... hint... 😉 )
❤️ 1
e

enough-analyst-54434

04/29/2022, 10:58 PM
It looks like the complete platforms idea may get traction / a PEP: https://github.com/pypa/pip/issues/10050#issuecomment-1113595053
👍 1
6 Views