Hello! My team recently updated from pants 2.16 -...
# general
g
Hello! My team recently updated from pants 2.16 -> 2.18, and we are now having issues with
pants publish ::
and
pants package ::
We are getting this error now when attempting to package certain AWS lambdas .
Copy code
stderr:
56
No pre-built wheel was available for fuzzysearch 0.7.3.
57
Successfully built the wheel fuzzysearch-0.7.3-cp39-cp39-linux_x86_64.whl from the sdist fuzzysearch-0.7.3.tar.gz but it is not compatible with the requested foreign target complete platform cp311-cp311-manylinux_2_26_x86_64.
58
You'll need to build a wheel from fuzzysearch-0.7.3.tar.gz on the foreign target platform and make it available to Pex via a --find-links repo or a custom --index.
59
these lambdas are packaged as docker images, and we followed the documentation for the necessary build file changes from 2.16 -> 2.18. Nothing else has changed with these specific lambdas. My assumption is that this must be related to the updates around 'complete platforms' , Im curious if there is anything I can add/pass in the build file to have the package/publish behavior operate like it did before the update? here is what the BUILD file looks like for example.
Copy code
python_sources(name="lib")

python_aws_lambda_function(
    name="referral_lambda",
    runtime="python3.11",
    handler="referral_lambda.py:lambda_handler",
    dependencies=["//:reqs#psycopg2-binary", "//:reqs#jmespath"],
)

docker_image(
    name="referral",
    dependencies=[":referral_lambda"],
    image_tags=["{build_args.GIT_COMMIT}"],
    repository="referral",
)
And here is the relevant portion of the python-default lockfile for that specific library/dependency.
Copy code
{
  "artifacts": [
    {
      "algorithm": "sha256",
      "hash": "d5a1b114ceee50a5e181b2fe1ac1b4371ac8db92142770a48fed49ecbc37ca4c",
      "url": "<https://files.pythonhosted.org/packages/f7/28/3e9e4e55fd35356f331a22976694e151eb0214b68d3cd471936f9c09deba/fuzzysearch-0.7.3.tar.gz>"
    }
  ],
  "project_name": "fuzzysearch",
  "requires_dists": [
    "attrs>=19.3"
  ],
  "requires_python": null,
  "version": "0.7.3"
},
Thanks in advance!
b
Sorry for the trouble. Could you share your
pants.toml
?
g
Copy code
[GLOBAL]
pants_version = "2.18.0"
backend_packages.add = [
  "pants.backend.python",
  "pants.backend.python.lint.docformatter",
  "pants.backend.python.lint.black",
  "pants.backend.python.lint.flake8",
  "pants.backend.python.lint.isort",
  "pants.backend.python.typecheck.mypy",
  "pants.backend.awslambda.python",
  "pants.backend.experimental.python.lint.autoflake",
  "pants.backend.shell",
  "pants.backend.docker",
]


[anonymous-telemetry]
enabled = false

[source]
# The Python source root is the repo root. See <https://www.pantsbuild.org/docs/source-roots>.
root_patterns = ["/", "sailebot", "sailestack"]

[python]
# The default interpreter constraints for code in this repo. Individual targets can override
#  this with the `interpreter_constraints` field. See
#  <https://www.pantsbuild.org/docs/python-interpreter-compatibility>.

# Modify this if you don't have Python 3.9 on your machine.
# This can be a range, such as [">=3.8,<3.11"], but it's usually recommended to restrict
# to a single minor version.
interpreter_constraints = [">=3.11.4,<3.12"]

# Enable the "resolves" mechanism, which turns on lockfiles for user code. See
# <https://www.pantsbuild.org/docs/python-third-party-dependencies>. This also adds the
# `generate-lockfiles` goal for Pants to generate the lockfile for you.
enable_resolves = true

resolves = { python-default = "python-default.lock", flake8 = "3rdparty/python/flake8.lock" }

[docformatter]
args = ["--wrap-summaries=100", "--wrap-descriptions=100"]

[test]
use_coverage=true
# open_coverage=true
report=true

[coverage-py]
fail_under=67
global_report=true
# report=["html"]

[flake8]
install_from_resolve="flake8"


[docker]
build_args = ["GIT_COMMIT"]

[docker.registries.saile-dev-registry]
address = "<our aws ecr address>"
default = true
extra_image_tags = ["latest"]
@broad-processor-92400 thanks for the quick reply!
b
Does it work if you revert to the old style by adding this to
pants.toml
?
Copy code
[lambdex]
layout = "lambdex"
g
@broad-processor-92400 well, that worked. Thanks! We did not use to have that field defined in the toml, I was not aware of it. Will this cause our packaging of normal zip lambdas (on docker) to package with the old lambdex way as well?
b
Ah, sorry, that's for debugging, it's not the long term fix. There is hopefully a bunch of warnings suggesting that support for lambdex disappears in 2.19.
g
well darn 😂
b
That flag causes all
python_aws_lambda_function
targets to be packaged using the old style approach, rather than the new better (mostly) one.
g
I did get this warning
Copy code
Remove the whole [lambdex] section, as Lambdex is deprecated and its functionality will be removed. If you have `layout = "zip"`, no further action is required, as you are already using the recommended layout.

If you have `layout = "lambdex"`, removing the section will switch any `python_aws_lambda_function` (formerly `python_awslambda`) and `python_google_cloud_function` targets to using the `zip` layout, as recommended by cloud vendors. (If you are using `python_aws_lambda_function`, you will need to also update the handlers configured in the cloud from `lambdex_handler.handler` to `lambda_function.handler`.)

See the docs for more details:
b
I've got to go now, but I'll come back to it in a bit. Hopefully that at least unblocks you for right now
g
Ill keep plugging away at it , thank you!!
We never specified this field in the toml before, And while reading through the docks I noticed this
Copy code
interpreter_constraints
--lambdex-interpreter-constraints="['<str>', '<str>', ...]"

PANTS_LAMBDEX_INTERPRETER_CONSTRAINTS
default:
[
  "CPython>=3.7,<3.12"
]

Python interpreter constraints for this tool.
Im assuming then , if we were not specifying this before then it would have been defaulting to this interpreter constraint, correct? If so, would that explain why it was able to package before? The library in question only officially supports up to 3.10, so it would have accepted the wheel built for 3.9 as that falls within the interpreter constraints? (Im a little out of my element here so forgive me if I am way off base)
b
Yeah, the
[lambdex] layout
field is new, as part of the switch from the Lambdex wrapper, to the new AWS-recommended layout, to allow staging a roll-out across 2.17 & 2.18 better without breaking people (or at least, when there is a breaking problem, give more options/workarounds). The interpreter constraints there are a bit different, they say that the Lambdex tool itself needs that an interpreter in that range to run the tool, not so much what the output of the tool is.
Another question to try to resolve the problem for real here: can you share the
Dockerfile
? In particular, what's the base image
FROM ...
?
g
Copy code
FROM public.ecr.aws/lambda/python:3.11
RUN yum install unzip-6.0-57.amzn2.0.1.x86_64 -y
COPY sailebot.referral/referral_lambda.zip .
RUN unzip referral_lambda.zip -d ${LAMBDA_TASK_ROOT}
CMD ["lambdex_handler.handler"]
Noticing now that this is still referencing "lambdex_handler.handler" which I am assuming will need to be updated.
b
yep, that'll need an update
It's not the cause of the build issue, though
👍 1
g
figured, Ill make that update then come back to this in the morning. I appreciate your help so far! thank you!
b
Hm, thinking about it a bit more, I think that the changes to the lambda layouts were conceptually moving work done at runtime to be work done at build time, and this counts for errors too... so I would've expected the lambda in question to fail on start-up, when the PEX dynamic initialisation decided that there wasn't an appropriate
fuzzysearch
library available for it. Can you confirm the lambda runs successfully in the old style? (not just builds successfully)