Hi there, can I please get some help with

python_awslambda

target? I’m using

pants.backend.awslambda.python

backend package to create lambda function and the lambda function has a dependency to a 3rd party

pyyaml

I’m getting the following error when I run

./pants awslambda

on my target

stderr:
                ERROR: Could not find a version that satisfies the requirement pyyaml==5.3.1

I’m only getting error for

pyyaml

and all the other 3rd-party dependencies are fine

Hello! What Pants version are you using? (

./pants --version

). Are you using the new docs at https://pants.readme.io/docs/awslambda-python? Also what did you set for

runtime

for the target?

I’m on

1.28.0

and runtime

3.8

yes that’s the doc that i’m using too

Are you using macOS?

yep

That’s the issue 😕 See the warning box in https://pants.readme.io/docs/awslambda-python#step-3-run-awslambda

ahh okk 👍 the dependencies used to work when we were using the plugins

pantsbuild.pants.contrib.awslambda-python

(before we moved to using backend package)

Ah, did it actually work when you deployed the lambda built from a mac? Lambdas always run in Linux, so we made things more correct by setting the

platform

to linux, no matter what. Whereas earlier we would silently do the wrong thing

no no the lambda is not deployed from mac. yea setting the

platform

to linux is the correct thing to do

Hm, that’s failing on Linux too? What’s the error? Also no need to apologize :)

hey yes, it’s failing on linux too it’s the same error

stderr:

[2020-07-07T13:03:55.557Z]                 ERROR: Could not find a version that satisfies the requirement pyyaml==5.3.1 (from versions: none)

[2020-07-07T13:03:55.557Z] ERROR: No matching distribution found for pyyaml==5.3.1

Okay, is the full Pex command included in the error message? Usually Pants will say the command it’s telling Pex to run. The most helpful thing to debug now is to see if we can get Pex itself running what you want. That way, we can isolate whether this is an issue with Pex (and Pip) or with Pants I think it will look something like:

pip install pex (or pipx install pex)
pex --platform=manylinux2014_x86_64-cp38 pyyaml==5.3.1

We executed

pex --platform=manylinux2014_x86_64-cp-38-cp38 pyyaml==5.3.1

but it’s failing with the error

ERROR: Could not find a version that satisfies the requirement pyyaml==5.3.1 (from versions: none)

we think the issue might be with PEX @many-agent-62725

Okay, that is indeed an issue with Pex then, and more specifically Pip, which does all the resolution for Pex. What happens if you change to a lower many Linux year, like 2010? Also if you remove the platform string, it works, right?

Executing

pex --platform=manylinux2010_x86_64-cp-38-cp38 pyyaml==5.3.1

will also give the following error:

ERROR: Could not find a version that satisfies the requirement pyyaml==5.3.1 (from versions: none)

On the other hand, if we remove the platform string (

pex pyyaml==5.3.1

), we enter the PEX REPL

Python 3.8.2 (default, Mar 11 2020, 00:29:50) 
[Clang 11.0.0 (clang-1100.0.33.17)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
(InteractiveConsole)
>>>

Hm. @enough-analyst-54434 any ideas why an sdist wouldn’t work with the platform string?

Yeah, that's by design. If you specify a platform Pex always treats those as foreign and requires wheels be available. In other words, there is no logic to check that a (non

"current"

) platform string is satisfiable by a local interpreter.

Hm, John, this means that our AWS Lambda logic might need to be more complex. It hardcodes the

--platform

value. cc @happy-kitchen-89482

Aha. Is that even valid to do? This shows 4 supported platforms: https://docs.aws.amazon.com/lambda/latest/dg/lambda-python.html

See https://github.com/pantsbuild/pants/blob/39ae9b9a8be2f14343dde4198043c339148d58f4/src/python/pants/backend/awslambda/python/awslambda_python_rules.py#L65-L71. We have the user explcitly specify the

runtime

as one of those 4 values in their awslambda target. Then we set the platform accordingly

K. Yeah, logic needs to be more complex and only issue platform when not on linux, using constraints or --python if on linux.

Benjy did the

u

logic. We originally always had

m

, and a user contributed a patch to not use it with >=3.8 because it’s now

cp38

, not

cp38m

OK. So maybe u is right and probably d is not there etc. Seems likely.

Sounds good, thank you

You're welcome: https://github.com/pantsbuild/pants/issues/10286 I couldn't dream up a sane way to switch between

--platform

and

--python

(or

--interpreter-constraint

) as things stand given remoting possibly being in play so noted the alternative requiring Pex features. We still may be able to do that purely on the Pants side but depends on details of the current state of

MultiPlatformProcess

and the interaction of

PlatformConstraints

with remote / local.

I’m now curious: why is there the need for hardcoding the platform string? Can’t we just use

platforms=PexPlatforms([]),

in line 77 (https://github.com/pantsbuild/pants/blob/39ae9b9a8be2f14343dde4198043c339148d58f4/src/python/pants/backend/awslambda/python/awslambda_python_rules.py#L77) and let PEX decide?

Correctness. If you’re on macOS, you must only use pre-built wheels that are compatible with linux. It would be misleading for us to leave off the platform, because you would end up building a

macos

wheel for something like PyYAML and Pants would say that everything worked. Then when you run it in AWS, it would fail.

@clever-table-90691 you should be good to go as soon as https://github.com/pantsbuild/pants/pull/10323 lands. That will either go out in 2.0.0.dev2 in the next few days or 2.0.0.dev3 at the end of next week. Thanks for bringing this issue up, it improved both Pants and Pex.

that’s good news!! thanks 🙂