https://pantsbuild.org/ logo
Join Slack
Powered by
Hi folks. Just bumped my lambda project to pants `...
# general
n
Hi folks. Just bumped my lambda project to pants 
2.17.0
and switched lambda zip format to "zip" instead of lambdex. I have some new errors and I am not sure how to solve them:
Copy code
Needed cp39-cp39-linux_x86_64 compatible dependencies for:
 1: Pillow
    Required by:
      amazon-textract-textractor 1.3.2
    But this pex had no ProjectName(raw='Pillow', normalized='pillow') distributions.
 2: Pillow>=9.4.*
    Required by:
      amazon-textract-pipeline-pagedimensions 0.0.8
    But this pex had no ProjectName(raw='Pillow', normalized='pillow') distributions.
and
Copy code
Found 1 distribution for pillow that do not apply:
1.) The wheel tags for Pillow 10.0.0 are cp39-cp39-manylinux_2_28_x86_64 which do not match the supported tags of cp39-cp39-linux_x86_64:
cp39-cp39-manylinux2014_x86_64
Would you have any clue on how I could resolve that ? Thanks !
e
@nice-eye-10512 you got burned by using 
platforms
instead of 
complete_platforms
is the short answer. The short solution is do like it shows here: https://www.pantsbuild.org/docs/awslambda-python#step-2-define-a-python_awslambda-target - @broad-processor-92400 made it easier to use 
complete_platforms
in 2.17.+ (they now get used on your behalf without you having to configure them explicitly) and now saying less in your BUILD target gets you more. Before trying that though, please paste your existing 
python_awslambda
target that is failing to confirm this direction is correct.
The 
pants.toml
edit it shows:
Copy code
[lambdex]
layout = "zip"
Is needed in addition to ditching 
platforms
in your target.
n
Hi @enough-analyst-54434! Thanks for the answer. I don't have any 
platforms
in my target. I merely specify the 
runtime
:
Copy code
python_awslambda(
    name="lambda",
    runtime="python3.9",
    handler="my.lambda_runner:lambda_handler",
    dependencies=[":bot"],
)
I upgrade from 
2.16.0
as well. I did not see the 
platforms
attribute on the target.
e
Ok, that uses 
platforms
behind the scenes IIRC. If you make the 
pants.toml
edit above, that should change automagically to use 
complete_platforms
.
But please also read https://www.pantsbuild.org/docs/awslambda-python#migrating-from-pants-216-and-earlier - there is a change in handler name that you'll have to deal with in AWS configs.
n
Yes, I saw that documentation before, thank ! This is not an issue. The only thing left is really to package the Lambda code into a zip which is blocked by the error I showed. I'll play around those platforms attributes and I'll follow up. Thanks for the clues.
e
My point is the error should be gone with the 
pants.toml
edit. Is that not so?
n
I does not seems so. I upgraded to 
2.17
but had to fallback to 
layout=lambdex
(with the old handler names) to make this works.
But as I said, my 
python_awslambda
targets are using 
runtime
attributes.
e
Ugrading to 2.17 stays defaulted to 
layout=lambdex
- you need do nothing.
I am confused. What exactly did you do? Just upgrade to 2.17, or did you do more when you originally reported?
n
Yeah but if now give a warning that it will default to "zip" in 2.18 šŸ˜„
I did: ā€¢ Bump pants 2.16.0 -> 2.17.0 ā€¢ Added to the `pants.toml`:
Copy code
[lambdex]
layout = "zip"
ā€¢ Switched 
entrypoint
from lambdex.handler -> lambda.handler
e
So you did the second bullet point without my prompt, or after my prompt?
n
Without your prompt. All this activity started when I red this document: https://www.pantsbuild.org/docs/awslambda-python#migrating-from-pants-216-and-earlier
e
Ok. That was a good nugget to reveal.
n
Sorry for the confusion. šŸ˜ž
e
So, please provide a full command line and command output for this if you can:
Copy code
pkill pantsd
pants <whatever it is you run>
If you don't have pkill, just make sure the pants daemon 
pantsd
process is dead before running your pants command.
šŸ‘ 1
The crux is this. Iff you're using the new stuff, then this complete platform should be auto-picked for you based on your runtime setting: https://github.com/pantsbuild/pants/blob/main/src/python/pants/backend/awslambda/python/complete_platform_3.9-x86_64.json And that has this primary tag 
"cp39-cp39-manylinux_2_26_x86_64"
, and so from Pillow 10.0's files (https://pypi.org/project/Pillow/#files), 
Pillow-10.0.0-cp39-cp39-manylinux_2_28_x86_64.whl
should be rejected but 
Pillow-10.0.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
should be selected.
šŸ‘€ 1
Basically the 2_28 thing is the glibc version and they are backwards compatible; so the AWS Python 3.9 runtime supports 2.26 and older (2.28 is too new).
n
Crux? Are you a climber ?
šŸ˜„
e
Yeah. I am primarily a climber.
šŸ‘ 1
n
I'm usually the one saying crux and nobody understand ... šŸ˜„
Do you want me to run the commands without the daemon 
PANTS_PANTSD=false
?
e
Sure - that'll do too.
n
Got this message that I did not took care yet:
Copy code
19:13:48.64 [WARN] DEPRECATED: using a `pants` launcher binary older than 0.10.0 is scheduled to be removed in version 2.18.0.dev6.
e
Sure - yeah - not relevant to this. Pants deprecates things fast and furious.
n
Copy code
>> PANTS_PANTSD=false pants package my/target/lambda::
19:13:48.64 [WARN] DEPRECATED: using a `pants` launcher binary older than 0.10.0 is scheduled to be removed in version 2.18.0.dev6.

Run `PANTS_BOOTSTRAP_VERSION=report pants` to see the current version of the `pants` launcher binary, and see <https://www.pantsbuild.org/v2.17/docs/installation> for how to upgrade.
19:14:54.34 [INFO] Completed: Building 22 requirements for faas_repository.pex from the pants.lock resolve: aws-lambda-powertools[aws-sdk]==2.23.0, aws-lambda-typing==2.17.1, boto3==1.28.30, botocore-stubs==1.31.30, botocore==1.31... (417 characters truncated)
19:14:56.84 [INFO] Completed: Build python_awslambda artifact for my/target/lambda:lambda
19:14:56.84 [ERROR] 1 Exception encountered:

Engine traceback:
  in `package` goal

ProcessExecutionFailure: Process 'Build python_awslambda artifact for my/target/lambda:lambda' failed with exit code 1.
stdout:

stderr:
A distribution for pillow could not be resolved for cp39-cp39-linux_x86_64.
Found 1 distribution for pillow that do not apply:
1.) The wheel tags for Pillow 10.0.0 are cp39-cp39-manylinux_2_28_x86_64 which do not match the supported tags of cp39-cp39-linux_x86_64:
cp39-cp39-manylinux2014_x86_64
... 95 more ...



Use `--keep-sandboxes=on_failure` to preserve the process chroot for inspection.
e
Can you please follow the last line of advice, then switch to that sandbox dir and paste the contents of 
__run.sh
here?
šŸ‘ 1
n
Copy code
cat __run.sh 
#!/usr/bin/env bash
# This command line should execute the same process as pants did internally.
cd /tmp/pants-sandbox-18jY3e
env -i CPPFLAGS= LANG=C.UTF-8 LDFLAGS= PATH=$'/ide/bin/remote-cli:/usr/local/bin:/usr/local/sbin:/usr/sbin:/usr/bin:/sbin:/bin' PEX_IGNORE_RCFILES=true PEX_PYTHON=/home/gitpod/.cache/nce/466c13d805cfedffe1debdc484ce9cda9c007101044055b736ba37e4fea056b3/bindings/venvs/2.17.0/bin/python PEX_ROOT=.cache/pex_root PEX_SCRIPT=pex3 /home/gitpod/.cache/nce/466c13d805cfedffe1debdc484ce9cda9c007101044055b736ba37e4fea056b3/bindings/venvs/2.17.0/bin/python ./pex venv create --tmpdir .tmp --python-path $'/ide/bin/remote-cli:/usr/local/bin:/usr/local/sbin:/usr/sbin:/usr/bin:/sbin:/bin' $'--dest-dir=my.target/lambda' $'--pex-repository=faas_repository.pex' $'--layout=flat-zipped' --platform linux_x86_64-cp-39-cp39
There is this version of Pillow in the sandbox:
Copy code
./faas_repository.pex/.deps/Pillow-10.0.0-cp39-cp39-manylinux_2_28_x86_64.whl
e
Yeah, so 
--platform linux_x86_64-cp-39-cp39
and not 
--complete-platform
.
I thought the complete platform was handled for you, but maybe not yet. I'll read some 2.17.0 code here for a bit and report back.
n
I am not sure what you mean by this:
Copy code
Yeah, so --platform linux_x86_64-cp-39-cp39 and not --complete-platform.
Ho, on the command. Gotcha
What would be the complete syntax for --complete-platform ?
e
So, yeah, the auto-complete-platforms thing is only in 2.18.x.
n
e
So you need ... all these things: + layout=zip + manual complete platforms - copy this down in the directory: https://github.com/pantsbuild/pants/blob/main/src/python/pants/backend/awslambda/python/complete_platform_3.9-x86_64.json and then hook it up with:
Copy code
file(
    name="aws-lambda-py39",
    source="complete_platform_3.9-x86_64.json",
)

python_awslambda(
    name="lambda",
    runtime="python3.9",
    complete_platforms=[":aws-lambda-py39"],
    handler="my.lambda_runner:lambda_handler",
    dependencies=[":bot"],
)
n
ā˜ļø I think we need to remove 
runtime
, no ?
e
With both of those things, you should be able to re-run 
pants --keep-sandboxes=always package my/target/lambda::
and observe the 
__run.sh
in the sandbox use 
--complete-platform
but not 
--platform
.
n
What's in this
Copy code
complete_platform_3.9-x86_64.json
?
e
ā˜ļø I think we need to remove runtime, no ?
I think that's only true for layout=lambdex after reading source code.
What's in this
The contents of the file I linked to.
n
gotcha
e
That's the complete platform json for AWS's Python 3.9 AMI.
n
Missed the link
All right, thanks you, fellow climber ! šŸ’Ŗ
e
You're welcome. Please report back either way.
Where do you climb / home base?
Poitras is very New Hampshirey and I grew up there / climbed there in my youth.
n
Got a new error ! šŸ™‚
Copy code
A single target is required for creating a flat sys.path directory entry.
There were 2 targets selected:
1. cp39-cp39-linux_x86_64
2. cp39-cp39-manylinux_2_26_x86_64
e
The 
__run.sh
contents will be helpful.
Perhaps nuking the runtime field as you said is needed here.
n
Copy code
#!/usr/bin/env bash
# This command line should execute the same process as pants did internally.
cd /tmp/pants-sandbox-Jpb74S
env -i CPPFLAGS= LANG=C.UTF-8 LDFLAGS= PATH=$'/ide/bin/remote-cli:/usr/local/bin:/usr/local/sbin:/usr/sbin:/usr/bin:/sbin:/bin' PEX_IGNORE_RCFILES=true PEX_PYTHON=/home/gitpod/.cache/nce/466c13d805cfedffe1debdc484ce9cda9c007101044055b736ba37e4fea056b3/bindings/venvs/2.17.0/bin/python PEX_ROOT=.cache/pex_root PEX_SCRIPT=pex3 /home/gitpod/.cache/nce/466c13d805cfedffe1debdc484ce9cda9c007101044055b736ba37e4fea056b3/bindings/venvs/2.17.0/bin/python ./pex venv create --tmpdir .tmp --python-path $'/ide/bin/remote-cli:/usr/local/bin:/usr/local/sbin:/usr/sbin:/usr/bin:/sbin:/bin' $'--dest-dir=my.target/lambda' $'--pex-repository=faas_repository.pex' $'--layout=flat-zipped' --platform linux_x86_64-cp-39-cp39 --complete-platform platforms/complete_platform_3.9-x86_64.json
e
~runtime -> platform
Yeah - there is --platform and --complete-platform - we just want the latter.
n
šŸŽ‰
e
My 2 bullet points are 3, 3 bullet points (

https://www.youtube.com/watch?v=Cj8n4MfhjUcā–¾

)
@nice-eye-10512 so killing 
runtime=
was the final bit?
n
Ho yeah
b
Sorry for the trouble. The simplifications to automatically provide a complete platform are coming in 2.18.
Open in Slack
PreviousNext
Powered by