https://pantsbuild.org/ logo
#general
Title
# general
c

clever-table-90691

07/06/2020, 2:14 PM
Hi there, can I please get some help with
python_awslambda
target? I’m using
pants.backend.awslambda.python
backend package to create lambda function and the lambda function has a dependency to a 3rd party
pyyaml
I’m getting the following error when I run
./pants awslambda
on my target
Copy code
stderr:
                ERROR: Could not find a version that satisfies the requirement pyyaml==5.3.1
I’m only getting error for
pyyaml
and all the other 3rd-party dependencies are fine
h

hundreds-father-404

07/06/2020, 2:19 PM
Hello! What Pants version are you using? (
./pants --version
). Are you using the new docs at https://pants.readme.io/docs/awslambda-python? Also what did you set for
runtime
for the target?
c

clever-table-90691

07/06/2020, 2:21 PM
I’m on
1.28.0
and runtime
3.8
yes that’s the doc that i’m using too
👍 1
h

hundreds-father-404

07/06/2020, 2:22 PM
Are you using macOS?
c

clever-table-90691

07/06/2020, 2:23 PM
yep
h

hundreds-father-404

07/06/2020, 2:23 PM
That’s the issue 😕 See the warning box in https://pants.readme.io/docs/awslambda-python#step-3-run-awslambda
You can see that PyYAML is problematic because of https://pypi.org/project/PyYAML/#files. While they have prebuilt bdist wheels for Windows, for macOS and Linux you have to use the sdist
PyYAML-5.3.1.tar.gz
(https://pants.readme.io/docs/python-platforms for some more context, although page isn’t fully complete)
👍 1
c

clever-table-90691

07/06/2020, 3:15 PM
ahh okk 👍 the dependencies used to work when we were using the plugins
pantsbuild.pants.contrib.awslambda-python
(before we moved to using backend package)
thanks the reply anyway 🙂
h

hundreds-father-404

07/06/2020, 3:17 PM
Ah, did it actually work when you deployed the lambda built from a mac? Lambdas always run in Linux, so we made things more correct by setting the
platform
to linux, no matter what. Whereas earlier we would silently do the wrong thing
c

clever-table-90691

07/06/2020, 3:22 PM
no no the lambda is not deployed from mac. yea setting the
platform
to linux is the correct thing to do
👍 1
hi @hundreds-father-404 sorry it’s me again, like you said pyyaml has prebuilt bdist for window only, what should we do in this case? it seems that
./pants awslambda
would fail if the dependencies doesn’t have the right prebuild bdist available and it wouldn’t install the package using sdist (when running on linux)
h

hundreds-father-404

07/07/2020, 3:04 PM
Hm, that’s failing on Linux too? What’s the error? Also no need to apologize :)
c

clever-table-90691

07/07/2020, 3:43 PM
hey yes, it’s failing on linux too it’s the same error
Copy code
stderr:

[2020-07-07T13:03:55.557Z]                 ERROR: Could not find a version that satisfies the requirement pyyaml==5.3.1 (from versions: none)

[2020-07-07T13:03:55.557Z] ERROR: No matching distribution found for pyyaml==5.3.1
h

hundreds-father-404

07/07/2020, 3:46 PM
Okay, is the full Pex command included in the error message? Usually Pants will say the command it’s telling Pex to run. The most helpful thing to debug now is to see if we can get Pex itself running what you want. That way, we can isolate whether this is an issue with Pex (and Pip) or with Pants I think it will look something like:
Copy code
pip install pex (or pipx install pex)
pex --platform=manylinux2014_x86_64-cp38 pyyaml==5.3.1
c

clever-table-90691

07/07/2020, 5:24 PM
We executed
pex --platform=manylinux2014_x86_64-cp-38-cp38 pyyaml==5.3.1
but it’s failing with the error
Copy code
ERROR: Could not find a version that satisfies the requirement pyyaml==5.3.1 (from versions: none)
we think the issue might be with PEX @many-agent-62725
h

hundreds-father-404

07/07/2020, 5:26 PM
Okay, that is indeed an issue with Pex then, and more specifically Pip, which does all the resolution for Pex. What happens if you change to a lower many Linux year, like 2010? Also if you remove the platform string, it works, right?
m

many-agent-62725

07/07/2020, 5:29 PM
Executing
pex --platform=manylinux2010_x86_64-cp-38-cp38 pyyaml==5.3.1
will also give the following error:
ERROR: Could not find a version that satisfies the requirement pyyaml==5.3.1 (from versions: none)
On the other hand, if we remove the platform string (
pex pyyaml==5.3.1
), we enter the PEX REPL
Copy code
Python 3.8.2 (default, Mar 11 2020, 00:29:50) 
[Clang 11.0.0 (clang-1100.0.33.17)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
(InteractiveConsole)
>>>
By the way, there’s an option I see in the error message (I suspect it’s a PIP option):
--only-binary :all:
h

hundreds-father-404

07/07/2020, 5:35 PM
Hm. @enough-analyst-54434 any ideas why an sdist wouldn’t work with the platform string?
e

enough-analyst-54434

07/07/2020, 7:22 PM
Yeah, that's by design. If you specify a platform Pex always treats those as foreign and requires wheels be available. In other words, there is no logic to check that a (non
"current"
) platform string is satisfiable by a local interpreter.
So, @clever-table-90691 when you execute
pex --platform=manylinux2014_x86_64-cp-38-cp38 pyyaml==5.3.1
do you do so on a linux machine or on OSX?
If on linux, you should just say something like
--python=python3.8
instead of specifying the platform. If on OSX, you'll need to make sure linux wheels for that platform are pre-built and available in the list of --index and --find-links you let Pex know about.
h

hundreds-father-404

07/07/2020, 7:27 PM
Hm, John, this means that our AWS Lambda logic might need to be more complex. It hardcodes the
--platform
value. cc @happy-kitchen-89482
e

enough-analyst-54434

07/07/2020, 7:50 PM
Aha. Is that even valid to do? This shows 4 supported platforms: https://docs.aws.amazon.com/lambda/latest/dg/lambda-python.html
I mean, its valid, but it would seem to contradict the rest of Pants Python support which supports all 4 of those platforms.
h

hundreds-father-404

07/07/2020, 7:51 PM
See https://github.com/pantsbuild/pants/blob/39ae9b9a8be2f14343dde4198043c339148d58f4/src/python/pants/backend/awslambda/python/awslambda_python_rules.py#L65-L71. We have the user explcitly specify the
runtime
as one of those 4 values in their awslambda target. Then we set the platform accordingly
e

enough-analyst-54434

07/07/2020, 7:52 PM
K. Yeah, logic needs to be more complex and only issue platform when not on linux, using constraints or --python if on linux.
👍 1
The m / u logic also looks arbitrary without reading https://docs.aws.amazon.com/lambda/latest/dg/lambda-python.html and then further going and testing the AMIs to see what's really present.
I'm guessing @happy-kitchen-89482 did that.
h

hundreds-father-404

07/07/2020, 7:56 PM
Benjy did the
u
logic. We originally always had
m
, and a user contributed a patch to not use it with >=3.8 because it’s now
cp38
, not
cp38m
e

enough-analyst-54434

07/07/2020, 7:58 PM
OK. So maybe u is right and probably d is not there etc. Seems likely.
But can also maybe arbitrarily break when AMIs upgrade since no AMI versions are mentioned.
manylinux is broken here too I think. Forcing that kills support for non-manylinux wheels. I think that should instead be linux_x86_64... + --manylinux=manylinux2014 flag in pex which is default anyhow.
👍 1
I'll type up an issue with all this.
h

hundreds-father-404

07/07/2020, 8:02 PM
Sounds good, thank you
e

enough-analyst-54434

07/07/2020, 8:17 PM
You're welcome: https://github.com/pantsbuild/pants/issues/10286 I couldn't dream up a sane way to switch between
--platform
and
--python
(or
--interpreter-constraint
) as things stand given remoting possibly being in play so noted the alternative requiring Pex features. We still may be able to do that purely on the Pants side but depends on details of the current state of
MultiPlatformProcess
and the interaction of
PlatformConstraints
with remote / local.
m

many-agent-62725

07/07/2020, 11:36 PM
I’m now curious: why is there the need for hardcoding the platform string? Can’t we just use
platforms=PexPlatforms([]),
in line 77 (https://github.com/pantsbuild/pants/blob/39ae9b9a8be2f14343dde4198043c339148d58f4/src/python/pants/backend/awslambda/python/awslambda_python_rules.py#L77) and let PEX decide?
h

hundreds-father-404

07/07/2020, 11:39 PM
Correctness. If you’re on macOS, you must only use pre-built wheels that are compatible with linux. It would be misleading for us to leave off the platform, because you would end up building a
macos
wheel for something like PyYAML and Pants would say that everything worked. Then when you run it in AWS, it would fail.
👍 1
e

enough-analyst-54434

07/11/2020, 12:48 AM
@clever-table-90691 you should be good to go as soon as https://github.com/pantsbuild/pants/pull/10323 lands. That will either go out in 2.0.0.dev2 in the next few days or 2.0.0.dev3 at the end of next week. Thanks for bringing this issue up, it improved both Pants and Pex.
👍 1
c

clever-table-90691

07/14/2020, 3:03 PM
that’s good news!! thanks 🙂
❤️ 1
2 Views