https://pantsbuild.org/ logo
#general
Title
# general
c

chilly-holiday-77415

01/13/2023, 6:21 PM
I’m so close to handing over my service to pants but have hit the last hurdle pretty hard - I hit the issue described here: https://pantsbuild.slack.com/archives/C046T6T9U/p1670366698746769 running
./pants package ::
in an AWS codebuild container. Following that thread, I’ve managed to use complete_platforms to build and test inside a local dev container, and added
pex3 interpreter inspect --markers --tags
for both the codebuild container (where it builds in CI) and for my dev container (for local dev). I still get the same error in Lambda though - am I doing something wrong, or do I need to build for the lambda’s runtime container too in
complete_platforms?
in lambda:
Copy code
[ERROR] ResolveError: Failed to resolve requirements from PEX environment @ /var/task.
Needed cp39-cp39-manylinux_2_26_x86_64 compatible dependencies for:
 1: orjson>=3.2.1; extra == "all"
    Required by:
      fastapi 0.89.1
    But this pex had no ProjectName(raw='orjson', normalized='orjson') distributions.
...
e

enough-analyst-54434

01/13/2023, 6:24 PM
Dave, please provide:
Copy code
unzip -qc your.pex PEX-INFO | jq .
But the issue is almost certainly, yes, you're missing the most important --complete-platform, the actual production one.
c

chilly-holiday-77415

01/13/2023, 6:28 PM
PEX-INFO.json
e

enough-analyst-54434

01/13/2023, 6:29 PM
Yeah:
Copy code
orjson-3.8.5-cp39-cp39-manylinux_2_28_x86_64.whl
See that 2_28? That means glibc 2.28. Thats newer than 2.26
So AWS uses older glibc
c

chilly-holiday-77415

01/13/2023, 6:30 PM
BUILD:
Copy code
python_awslambda(
    name="externalapilambda",
    runtime="python3.9",
    handler="main.py:external_api_handler",
    complete_platforms=(
        "3rdparty/platforms:codebuild",
        "3rdparty/platforms:devcontainer",
    ),
)
e

enough-analyst-54434

01/13/2023, 6:30 PM
And you need a 3rd - production
c

chilly-holiday-77415

01/13/2023, 6:31 PM
gotcha - thank you for your patience and support John! I’m being forced to learn a lot of new internal stuff 🙂
e

enough-analyst-54434

01/13/2023, 6:31 PM
The fact codebuild doesn't match production is a bit odd. Its AWS for both.
Are you sure you can't align codebuild to use the same AMI / image as production?
c

chilly-holiday-77415

01/13/2023, 6:32 PM
I once tried to align python versions down to a patch ver on different services and failed while losing a week - there’s very little consistency there
e

enough-analyst-54434

01/13/2023, 6:33 PM
Its not Python versions that are important here
Its glibc
Basically, generically, its not too useful to build for production on anything but a production image - for anything, Pex aside.
c

chilly-holiday-77415

01/13/2023, 6:34 PM
hm that’s a good point - I guess mine was mainly to illustrate the differences between services though. Codebuild’s heavily cached build image is >gb and has all browsers, chrome, even x, etc whereas lambda’s built for being as slim as possible. My misconception here was it was more container/arch related than C version
I’m hoping
Copy code
public.ecr.aws/lambda/python:3.9
is actually the execution environment lambda runs in natively 🙂
e

enough-analyst-54434

01/13/2023, 6:35 PM
I think that's right.
But, if you're creative, it doesn't matter right? You can write a 1-off lambda that prints the complete-platform.
c

chilly-holiday-77415

01/13/2023, 6:36 PM
indeed - subprocess call to
Copy code
ldd --version
?
e

enough-analyst-54434

01/13/2023, 6:36 PM
What?
I got lost on that last from you.
c

chilly-holiday-77415

01/13/2023, 6:36 PM
ah, as you suggested creativity I was googling “how to get glibc version”
e

enough-analyst-54434

01/13/2023, 6:36 PM
You don't need the glibc version, that's just the explanation.
What you need is to run the command to get the complete platform on the actual target platform.
You should never hope and pray the complete platform you generate in 1 image happens to work for another.
Generate 1 for each actual image you run in.
c

chilly-holiday-77415

01/13/2023, 6:39 PM
I appreciate that - my concern is that (potentially) the native python execution environment isn’t actually a container by default, it’s a microvm - but you’re suggesting running the equivalent of
pip install pex; pex3 interpreter inspect --markers --tags
in the lambda env right?
e

enough-analyst-54434

01/13/2023, 6:40 PM
Yes. If you can't get access to the image, then writing a 1-off lambda to do just that is guaranteed correct.
This works, because Pex itself is not platform-specific. It will just work in whatever environment.
c

chilly-holiday-77415

01/13/2023, 6:41 PM
gotcha - I understand and appreciate the education 🙂
e

enough-analyst-54434

01/13/2023, 6:41 PM
Does the 1-off lambda make sense? Build a lambda with just pex as a dep and use subprocess to run pex3 ...
c

chilly-holiday-77415

01/13/2023, 6:42 PM
yep, I’ll do that for the public image and inside a lambda and report back for future adventurers
e

enough-analyst-54434

01/13/2023, 6:42 PM
But I am pretty sure the container you mentioned is correct; so I don't think you need to do that.
c

chilly-holiday-77415

01/13/2023, 6:42 PM
agree, I’m just trying to avoid new corners to get stuck in 🙂
e

enough-analyst-54434

01/13/2023, 6:42 PM
Yup, gotcha.
c

chilly-holiday-77415

01/13/2023, 6:42 PM
thanks again John - have a good weekend. I’ll report back Monday!
e

enough-analyst-54434

01/13/2023, 6:43 PM
Great, thank you.
c

chilly-holiday-77415

01/13/2023, 8:15 PM
so I had to try it before signing off for the weekend… the first compatible tag generated seems to show that the docker image is good:
Copy code
{
  "path": "/var/lang/bin/python3.9",
  "compatible_tags": [
    "cp39-cp39-manylinux_2_26_x86_64",
...
but redeploying still failed 😕 I can’t see an obvious misconfig in my build files - I assume this should be working? (I didn’t manage to check within lambda yet - apparently not as simple as I’d hoped).
e

enough-analyst-54434

01/13/2023, 8:22 PM
If you start by providing the PEX-INFO as you did before, that's ground truth.
c

chilly-holiday-77415

01/13/2023, 9:29 PM
hmm and it’s okay to be doing
./pants package ::
in a different container (the dev container) right?
Copy code
21:26:05.93 [INFO] Wrote dist/projects.api/externalapilambda.zip
              Runtime: python3.9
    Complete platform: 3rdparty/platforms/docker_aws_codebuild_standard_5_0_x86_64.json
    Complete platform: 3rdparty/platforms/docker_python_3_9_16_devcontainer_x86.json
    Complete platform: 3rdparty/platforms/docker_python_3_9_lambda_x86.json
unzip -qc dist/projects.api/externalapilambda.zip PEX-INFO | jq . > out.json
in the PEX-INFO, which seems the problem?
"orjson-3.8.5-cp39-cp39-manylinux_2_28_x86_64.whl": "552321182e28a4150ae47f606a4693c7307f0806c6bdd7488372f0e8d47d491c",
e

enough-analyst-54434

01/13/2023, 9:36 PM
I'd simplify 1st to sanity check. Remove all but the production complete platform, re-package and check the PEX contents or PEX-INFO, it should then have 0 2.28 wheels.
I'm assuming here you also sanity checked your production compete platform is 2.26? The 1st tag should tell.
c

chilly-holiday-77415

01/13/2023, 9:39 PM
I think I tried that sanity check: just to overshare rather than hide a misconception here:
Copy code
python_awslambda(
    name="externalapilambda",
    runtime="python3.9",
    handler="main.py:external_api_handler",
    complete_platforms=(
        # "3rdparty/platforms:codebuild",
        # "3rdparty/platforms:devcontainer",
        "3rdparty/platforms:awslambda",
    ),
)
PEX-INFO still contains
"orjson-3.8.5-cp39-cp39-manylinux_2_28_x86_64.whl": "552321182e28a4150ae47f606a4693c7307f0806c6bdd7488372f0e8d47d491c",
e

enough-analyst-54434

01/13/2023, 9:39 PM
What is the 1st tag in the prod complete platform json.
c

chilly-holiday-77415

01/13/2023, 9:39 PM
Copy code
{
  "path": "/var/lang/bin/python3.9",
  "compatible_tags": [
    "cp39-cp39-manylinux_2_26_x86_64",
e

enough-analyst-54434

01/13/2023, 9:40 PM
Then I don't believe you.
c

chilly-holiday-77415

01/13/2023, 9:40 PM
😅
e

enough-analyst-54434

01/13/2023, 9:40 PM
It should be impossible for that PEX to have 2.28 wheels.
Ah ... wait a sec. Are you running pants on Linux?
c

chilly-holiday-77415

01/13/2023, 9:43 PM
I am - I have an EC2 instance to avoid my m1 mac arch
e

enough-analyst-54434

01/13/2023, 9:43 PM
That's the problem
c

chilly-holiday-77415

01/13/2023, 9:44 PM
ironically, I think I’m only using it to build this lambda
e

enough-analyst-54434

01/13/2023, 9:44 PM
So, you should try building on your Mac.
I can explain more when I get to keyboard.
Fixed bug in Pex the way Pants uses it that only affects Linux machines.
Need bleeding edge Pants or custom Pex version in older Pants. ... or, simplest, just try a Mac build right now for a preview of results
If it fails to build on Mac, the fixes won't help
c

chilly-holiday-77415

01/13/2023, 9:48 PM
so on my mac I still get this failure,
Failed to resolve for platform linux_x86_64-cp-39-cp39. Resolve requires evaluation of unknown environment marker: 'python_full_version' does not exist in evaluation environment.
which is (iirc) what put me on the remote build path
e

enough-analyst-54434

01/13/2023, 9:48 PM
Nack, I don't believe you
With a complete platform you can't get that error
Only with a plain platform
c

chilly-holiday-77415

01/13/2023, 9:51 PM
can I assert that I’m actually using the complete platform setting at all?
e

enough-analyst-54434

01/13/2023, 9:51 PM
Ah, right. You should not be using
runtime
If you have complete platforms
Kill it.
c

chilly-holiday-77415

01/13/2023, 9:51 PM
aha
that builds on mac!
e

enough-analyst-54434

01/13/2023, 9:53 PM
Ok, great. Should work on AWS now.
I'll update those docs today. That's a mess.
c

chilly-holiday-77415

01/13/2023, 9:55 PM
the mac built PEX-INFO contains:
Copy code
"orjson-3.8.5-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl": "9dbb9375310a5b24cae206e7a84b892c0af38299628474aa27dbd386985ecacc",
    "orjson-3.8.5-cp39-cp39-manylinux_2_28_x86_64.whl": "552321182e28a4150ae47f606a4693c7307f0806c6bdd7488372f0e8d47d491c",
e

enough-analyst-54434

01/13/2023, 9:56 PM
So you're back to multiple complete platforms right?
c

chilly-holiday-77415

01/13/2023, 9:56 PM
ah, yes
e

enough-analyst-54434

01/13/2023, 9:56 PM
Right. The 1st will be picked on AWS
Should be good
c

chilly-holiday-77415

01/13/2023, 9:56 PM
just trying to assert that I meet all the rules I breached earlier :D
and if I cut the other platforms again, no >26
e

enough-analyst-54434

01/13/2023, 9:57 PM
Well, the docs there are not helpful to say the least
Yeah, that should be true.
c

chilly-holiday-77415

01/13/2023, 9:58 PM
happy to write up or contribute anything I’ve learnt once I grok the world enough, I think Pants really solves a lot of the python developer experience for serverless like nothing else can
e

enough-analyst-54434

01/13/2023, 9:59 PM
Well, thanks. I'd confirm the thing works 1st!
c

chilly-holiday-77415

01/13/2023, 11:07 PM
successful lambdex execution! I had to manually upload the lambdex .zip from my mac rather than deploy it through a linux-based pipeline though which still seemed to fail with a ResolveError, which I’m assuming (?) is related to the same problem we had building on my linux ec2 box?
Interestingly as an aside to this - the
fastapi[all]
install (which brought in a bunch of these C deps) was getting dragged back two years to a 0.58.1 version because of (I think)
Copy code
Dependency on python-multipart not satisfied, 1 incompatible candidate found:
1.) python-multipart 0.0.5 (via: fastapi[all] -> python-multipart>=0.0.5; extra == "all") does not have any compatible artifacts:
    <https://files.pythonhosted.org/packages/46/40/a933ac570bf7aad12a298fc53458115cc74053474a72fbb8201d7dc06d3d/python-multipart-0.0.5.tar.gz>
but I don’t need that thankfully so will just rework my lock file.
e

enough-analyst-54434

01/14/2023, 6:40 PM
Ok, 1st - as to the confusion here re runtime vs complete_platforms generally being XOR, hopefully this new doc helps: https://github.com/pantsbuild/pants/pull/18001
I had to manually upload the lambdex .zip from my mac rather than deploy it through a linux-based pipeline though which still seemed to fail with a ResolveError, which I’m assuming (?) is related to the same problem we had building on my linux ec2 box?
That's probably right. You can try to upgrade to latest Pex to solve. That would look like this in your `pants.toml`:
Copy code
[pex-cli]
version = "v2.1.120"
known_versions = [
  "v2.1.120|macos_arm64|c8f2db310ea3e6dd400689b5993667a877d8540a4d206355fa102fff1d146ec0|4071055",
  "v2.1.120|macos_x86_64|c8f2db310ea3e6dd400689b5993667a877d8540a4d206355fa102fff1d146ec0|4071055",
  "v2.1.120|linux_x86_64|c8f2db310ea3e6dd400689b5993667a877d8540a4d206355fa102fff1d146ec0|4071055",
  "v2.1.120|linux_arm64|c8f2db310ea3e6dd400689b5993667a877d8540a4d206355fa102fff1d146ec0|4071055"
]

[lambdex]
lockfile = "you/choose/a/path/in/your/repo/lambdex.lock
I'm pretty sure only the
[lambdex]
section is needed, so you might try just that 1st. Either way, you then need to run:
Copy code
./pants generate-lockfiles --resolve=lambdex
That will lock lambdex at the path you specified and you should inspect that file (~JSON, but Pants is frustrating here and adds an invalid comment header) and ensure the locked version of Pex is 2.1.120. That file should be checked in. You can get rid of it and the new pants.toml config once upgraded to Pants 2.16.x or newer.
10 Views