hey! i'm quite new to pants and am using it for a ...
# general
hey! i'm quite new to pants and am using it for a couple of weeks now in my project. i'm using python and primarily lambda targets to build my apps.. however, it turned out that pex based lambdas seem to take quite long to bootstrap when having large deps like tensorflow, so i'm looking for alternative ways to package my lambda functions. is there any convenient way in pants to create some kind of "fat wheels" consisting of all required dependencies (resolved automatically), without having to rely on pex? thanks in advance!
👋 2
Do you have a pre-Pex comparision to help zero in on the perf gap between the two?
It would also be good to know what version of Pants you're using.
As John is implying, all Pex is doing is building the lambda zip file, and we suspect any other method of building it would have to do pretty much the same work
It may be the case for all huge-deps cases that a zip is simply the wrong way to go and the aws lambda support for specifying an image is your best bet.
We might be closer to providing out of the box support for that style now that Docker support has been added to Pants, but I suspect there are steps left yet before we can offer that as a clean solution.
I.e., you deploy to an image that has those huge deps preinstalled
hey guys! thanks for your replies! regarding the version, i'm using 2.7.0. in some other context in the past, i had a quite similar app (lambda, tensorflow deps etc.) and from what i remember, the execution time was well below a second, although i'm not quite sure how long the very first invocation actually took where deps were actually loaded. but i'm sure i'll find this somewhere in my logs. in general, what i haven't understood completely yet, is how pex's bootstrapping actually works - can it also exploit lambdas "context-preservation" mechanism as usual (frankly don't know the correct term, but i hope it's clear what i meant - so that consecutive invocations of the function do not require to reload libs etc. as long as it remains "hot"). cause in that way, loading wouldn't be that much of a problem. @happy-kitchen-89482 yeah, good point, i actually considered this already, since my artifacts simply became huge. is there any way to exclude certain dependencies from being packed with pex? what i can think of is to simply remove them from the zip within the pipeline or is there any option in either pex or the pants lambda target i haven't found yet? cheers!
To clarify, the issue is with the runtime of the lambda, not the build time in Pants?
I think an issue may be this: We use a tool called [lambdex](https://github.com/pantsbuild/lambdex) to turn .pex files into lambdas. All that tool does is add an entry point that bootstraps the pex, and then executes it. BUT - the "shape" of the created lambda is different than the shape of a regular lambda. The latter expects its dependencies to be pip-installed into a directory named
at the root of the zipfile. But the former has the deps in the usual
directory, which is then bootstrapped. I don't know how AWS's "context-preservation" (as you call it) mechanism works, but it's possible that it relies on that
layout in some way.
So a question would be - do you have a timing comparison between the behavior you're seeing with the lambda as built by Pants, vs the behavior with a "conventional lambda" (maybe you create it by hand just for this test)?
It wouldn't be hard for Pants to create a conventional lambda layout, if that turned out to make the difference
But I have to imagine that running .zip-style lambdas with large requirements such as tensorflow is going to be painful no matter what.
You also have the option of deploying container image lambdas which Pants can build in 2.8 thanks to the new Docker support.
So, to sum up, it would be really great to find out if the pants-created zipfile lambdas perform worse than conventional lambdas with the same code and dependencies, if you're able to test that out.