#general
Title
r
red-ghost-2220
11/03/2021, 5:00 PMhey!
i'm quite new to pants and am using it for a couple of weeks now in my project. i'm using python and primarily lambda targets to build my apps.. however, it turned out that pex based lambdas seem to take quite long to bootstrap when having large deps like tensorflow, so i'm looking for alternative ways to package my lambda functions.
is there any convenient way in pants to create some kind of "fat wheels" consisting of all required dependencies (resolved automatically), without having to rely on pex?
thanks in advance!
👋 2
e
enough-analyst-54434
11/03/2021, 5:15 PMDo you have a pre-Pex comparision to help zero in on the perf gap between the two?
It would also be good to know what version of Pants you're using.
h
happy-kitchen-89482
11/03/2021, 5:52 PMAs John is implying, all Pex is doing is building the lambda zip file, and we suspect any other method of building it would have to do pretty much the same work
e
enough-analyst-54434
11/03/2021, 6:13 PMIt may be the case for all huge-deps cases that a zip is simply the wrong way to go and the aws lambda support for specifying an image is your best bet.
We might be closer to providing out of the box support for that style now that Docker support has been added to Pants, but I suspect there are steps left yet before we can offer that as a clean solution.
h
happy-kitchen-89482
11/03/2021, 6:21 PMI.e., you deploy to an image that has those huge deps preinstalled
r
red-ghost-2220
11/04/2021, 8:36 AMhey guys! thanks for your replies!
regarding the version, i'm using 2.7.0.
in some other context in the past, i had a quite similar app (lambda, tensorflow deps etc.) and from what i remember, the execution time was well below a second, although i'm not quite sure how long the very first invocation actually took where deps were actually loaded. but i'm sure i'll find this somewhere in my logs.
in general, what i haven't understood completely yet, is how pex's bootstrapping actually works - can it also exploit lambdas "context-preservation" mechanism as usual (frankly don't know the correct term, but i hope it's clear what i meant - so that consecutive invocations of the function do not require to reload libs etc. as long as it remains "hot"). cause in that way, loading wouldn't be that much of a problem.
@happy-kitchen-89482 yeah, good point, i actually considered this already, since my artifacts simply became huge. is there any way to exclude certain dependencies from being packed with pex? what i can think of is to simply remove them from the zip within the pipeline or is there any option in either pex or the pants lambda target i haven't found yet?
cheers!
h
happy-kitchen-89482
11/04/2021, 5:11 PMTo clarify, the issue is with the runtime of the lambda, not the build time in Pants?
I think an issue may be this: We use a tool called [lambdex](https://github.com/pantsbuild/lambdex) to turn .pex files into lambdas. All that tool does is add an entry point that bootstraps the pex, and then executes it. BUT - the "shape" of the created lambda is different than the shape of a regular lambda. The latter expects its dependencies to be pip-installed into a directory named
packages.depspackagesSo a question would be - do you have a timing comparison between the behavior you're seeing with the lambda as built by Pants, vs the behavior with a "conventional lambda" (maybe you create it by hand just for this test)?
It wouldn't be hard for Pants to create a conventional lambda layout, if that turned out to make the difference
But I have to imagine that running .zip-style lambdas with large requirements such as tensorflow is going to be painful no matter what.
You also have the option of deploying container image lambdas which Pants can build in 2.8 thanks to the new Docker support.
So, to sum up, it would be really great to find out if the pants-created zipfile lambdas perform worse than conventional lambdas with the same code and dependencies, if you're able to test that out.
2 Views