https://pantsbuild.org/ logo
Join Slack
Powered by
I’m so close to handing over my service to pants b...
# general
c
I’m so close to handing over my service to pants but have hit the last hurdle pretty hard - I hit the issue described here: https://pantsbuild.slack.com/archives/C046T6T9U/p1670366698746769 running 
./pants package ::
in an AWS codebuild container. Following that thread, I’ve managed to use complete_platforms to build and test inside a local dev container, and added 
pex3 interpreter inspect --markers --tags
for both the codebuild container (where it builds in CI) and for my dev container (for local dev). I still get the same error in Lambda though - am I doing something wrong, or do I need to build for the lambda’s runtime container too in 
complete_platforms?
in lambda:
Copy code
[ERROR] ResolveError: Failed to resolve requirements from PEX environment @ /var/task.
Needed cp39-cp39-manylinux_2_26_x86_64 compatible dependencies for:
 1: orjson>=3.2.1; extra == "all"
    Required by:
      fastapi 0.89.1
    But this pex had no ProjectName(raw='orjson', normalized='orjson') distributions.
...
e
Dave, please provide:
Copy code
unzip -qc your.pex PEX-INFO | jq .
But the issue is almost certainly, yes, you're missing the most important --complete-platform, the actual production one.
c
PEX-INFO.json
PEX-INFO.json
e
Yeah:
Copy code
orjson-3.8.5-cp39-cp39-manylinux_2_28_x86_64.whl
See that 2_28? That means glibc 2.28. Thats newer than 2.26
So AWS uses older glibc
c
BUILD:
Copy code
python_awslambda(
    name="externalapilambda",
    runtime="python3.9",
    handler="main.py:external_api_handler",
    complete_platforms=(
        "3rdparty/platforms:codebuild",
        "3rdparty/platforms:devcontainer",
    ),
)
e
And you need a 3rd - production
c
gotcha - thank you for your patience and support John! I’m being forced to learn a lot of new internal stuff 🙂
e
The fact codebuild doesn't match production is a bit odd. Its AWS for both.
Are you sure you can't align codebuild to use the same AMI / image as production?
c
I once tried to align python versions down to a patch ver on different services and failed while losing a week - there’s very little consistency there
e
Its not Python versions that are important here
Its glibc
Basically, generically, its not too useful to build for production on anything but a production image - for anything, Pex aside.
c
hm that’s a good point - I guess mine was mainly to illustrate the differences between services though. Codebuild’s heavily cached build image is >gb and has all browsers, chrome, even x, etc whereas lambda’s built for being as slim as possible. My misconception here was it was more container/arch related than C version
I’m hoping
Copy code
public.ecr.aws/lambda/python:3.9
is actually the execution environment lambda runs in natively 🙂
e
I think that's right.
But, if you're creative, it doesn't matter right? You can write a 1-off lambda that prints the complete-platform.
c
indeed - subprocess call to
Copy code
ldd --version
?
e
What?
I got lost on that last from you.
c
ah, as you suggested creativity I was googling “how to get glibc version”
e
You don't need the glibc version, that's just the explanation.
What you need is to run the command to get the complete platform on the actual target platform.
You should never hope and pray the complete platform you generate in 1 image happens to work for another.
Generate 1 for each actual image you run in.
c
I appreciate that - my concern is that (potentially) the native python execution environment isn’t actually a container by default, it’s a microvm - but you’re suggesting running the equivalent of 
pip install pex; pex3 interpreter inspect --markers --tags
in the lambda env right?
e
Yes. If you can't get access to the image, then writing a 1-off lambda to do just that is guaranteed correct.
This works, because Pex itself is not platform-specific. It will just work in whatever environment.
c
gotcha - I understand and appreciate the education 🙂
e
Does the 1-off lambda make sense? Build a lambda with just pex as a dep and use subprocess to run pex3 ...
c
yep, I’ll do that for the public image and inside a lambda and report back for future adventurers
e
But I am pretty sure the container you mentioned is correct; so I don't think you need to do that.
c
agree, I’m just trying to avoid new corners to get stuck in 🙂
e
Yup, gotcha.
c
thanks again John - have a good weekend. I’ll report back Monday!
e
Great, thank you.
c
so I had to try it before signing off for the weekend… the first compatible tag generated seems to show that the docker image is good:
Copy code
{
  "path": "/var/lang/bin/python3.9",
  "compatible_tags": [
    "cp39-cp39-manylinux_2_26_x86_64",
...
but redeploying still failed 😕 I can’t see an obvious misconfig in my build files - I assume this should be working? (I didn’t manage to check within lambda yet - apparently not as simple as I’d hoped).
e
If you start by providing the PEX-INFO as you did before, that's ground truth.
c
hmm and it’s okay to be doing 
./pants package ::
in a different container (the dev container) right?
Copy code
21:26:05.93 [INFO] Wrote dist/projects.api/externalapilambda.zip
              Runtime: python3.9
    Complete platform: 3rdparty/platforms/docker_aws_codebuild_standard_5_0_x86_64.json
    Complete platform: 3rdparty/platforms/docker_python_3_9_16_devcontainer_x86.json
    Complete platform: 3rdparty/platforms/docker_python_3_9_lambda_x86.json
unzip -qc dist/projects.api/externalapilambda.zip PEX-INFO | jq . > out.json
in the PEX-INFO, which seems the problem? 
"orjson-3.8.5-cp39-cp39-manylinux_2_28_x86_64.whl": "552321182e28a4150ae47f606a4693c7307f0806c6bdd7488372f0e8d47d491c",
e
I'd simplify 1st to sanity check. Remove all but the production complete platform, re-package and check the PEX contents or PEX-INFO, it should then have 0 2.28 wheels.
I'm assuming here you also sanity checked your production compete platform is 2.26? The 1st tag should tell.
c
I think I tried that sanity check: just to overshare rather than hide a misconception here:
Copy code
python_awslambda(
    name="externalapilambda",
    runtime="python3.9",
    handler="main.py:external_api_handler",
    complete_platforms=(
        # "3rdparty/platforms:codebuild",
        # "3rdparty/platforms:devcontainer",
        "3rdparty/platforms:awslambda",
    ),
)
PEX-INFO still contains 
"orjson-3.8.5-cp39-cp39-manylinux_2_28_x86_64.whl": "552321182e28a4150ae47f606a4693c7307f0806c6bdd7488372f0e8d47d491c",
e
What is the 1st tag in the prod complete platform json.
c
Copy code
{
  "path": "/var/lang/bin/python3.9",
  "compatible_tags": [
    "cp39-cp39-manylinux_2_26_x86_64",
e
Then I don't believe you.
c
😅
e
It should be impossible for that PEX to have 2.28 wheels.
Ah ... wait a sec. Are you running pants on Linux?
c
I am - I have an EC2 instance to avoid my m1 mac arch
e
That's the problem
c
ironically, I think I’m only using it to build this lambda
e
So, you should try building on your Mac.
I can explain more when I get to keyboard.
Fixed bug in Pex the way Pants uses it that only affects Linux machines.
Need bleeding edge Pants or custom Pex version in older Pants. ... or, simplest, just try a Mac build right now for a preview of results
If it fails to build on Mac, the fixes won't help
c
so on my mac I still get this failure, 
Failed to resolve for platform linux_x86_64-cp-39-cp39. Resolve requires evaluation of unknown environment marker: 'python_full_version' does not exist in evaluation environment.
which is (iirc) what put me on the remote build path
e
Nack, I don't believe you
With a complete platform you can't get that error
Only with a plain platform
c
can I assert that I’m actually using the complete platform setting at all?
e
Ah, right. You should not be using 
runtime
If you have complete platforms
Kill it.
c
aha
that builds on mac!
e
Ok, great. Should work on AWS now.
I'll update those docs today. That's a mess.
c
the mac built PEX-INFO contains:
Copy code
"orjson-3.8.5-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl": "9dbb9375310a5b24cae206e7a84b892c0af38299628474aa27dbd386985ecacc",
    "orjson-3.8.5-cp39-cp39-manylinux_2_28_x86_64.whl": "552321182e28a4150ae47f606a4693c7307f0806c6bdd7488372f0e8d47d491c",
e
So you're back to multiple complete platforms right?
c
ah, yes
e
Right. The 1st will be picked on AWS
Should be good
c
just trying to assert that I meet all the rules I breached earlier :D
and if I cut the other platforms again, no >26
e
Well, the docs there are not helpful to say the least
Yeah, that should be true.
c
happy to write up or contribute anything I’ve learnt once I grok the world enough, I think Pants really solves a lot of the python developer experience for serverless like nothing else can
e
Well, thanks. I'd confirm the thing works 1st!
c
successful lambdex execution! I had to manually upload the lambdex .zip from my mac rather than deploy it through a linux-based pipeline though which still seemed to fail with a ResolveError, which I’m assuming (?) is related to the same problem we had building on my linux ec2 box?
Interestingly as an aside to this - the 
fastapi[all]
install (which brought in a bunch of these C deps) was getting dragged back two years to a 0.58.1 version because of (I think)
Copy code
Dependency on python-multipart not satisfied, 1 incompatible candidate found:
1.) python-multipart 0.0.5 (via: fastapi[all] -> python-multipart>=0.0.5; extra == "all") does not have any compatible artifacts:
    <https://files.pythonhosted.org/packages/46/40/a933ac570bf7aad12a298fc53458115cc74053474a72fbb8201d7dc06d3d/python-multipart-0.0.5.tar.gz>
but I don’t need that thankfully so will just rework my lock file.
e
Ok, 1st - as to the confusion here re runtime vs complete_platforms generally being XOR, hopefully this new doc helps: https://github.com/pantsbuild/pants/pull/18001
I had to manually upload the lambdex .zip from my mac rather than deploy it through a linux-based pipeline though which still seemed to fail with a ResolveError, which I’m assuming (?) is related to the same problem we had building on my linux ec2 box?
That's probably right. You can try to upgrade to latest Pex to solve. That would look like this in your `pants.toml`:
Copy code
[pex-cli]
version = "v2.1.120"
known_versions = [
  "v2.1.120|macos_arm64|c8f2db310ea3e6dd400689b5993667a877d8540a4d206355fa102fff1d146ec0|4071055",
  "v2.1.120|macos_x86_64|c8f2db310ea3e6dd400689b5993667a877d8540a4d206355fa102fff1d146ec0|4071055",
  "v2.1.120|linux_x86_64|c8f2db310ea3e6dd400689b5993667a877d8540a4d206355fa102fff1d146ec0|4071055",
  "v2.1.120|linux_arm64|c8f2db310ea3e6dd400689b5993667a877d8540a4d206355fa102fff1d146ec0|4071055"
]

[lambdex]
lockfile = "you/choose/a/path/in/your/repo/lambdex.lock
I'm pretty sure only the 
[lambdex]
section is needed, so you might try just that 1st. Either way, you then need to run:
Copy code
./pants generate-lockfiles --resolve=lambdex
That will lock lambdex at the path you specified and you should inspect that file (~JSON, but Pants is frustrating here and adds an invalid comment header) and ensure the locked version of Pex is 2.1.120. That file should be checked in. You can get rid of it and the new pants.toml config once upgraded to Pants 2.16.x or newer.
10 Views
Open in Slack
PreviousNext
Powered by