Hi guys ! I am using pants to build a bunch of lambda functions in python and I am looking for a way to leverage the tools that are already installed in the AWS lambda python images:

docker run -ti --rm --entrypoint /bin/bash amazon/aws-lambda-python:latest -c "pip list"
Package         Version
--------------- -------
awslambdaric    2.0.4
boto3           1.27.1
botocore        1.30.1
jmespath        1.0.1
pip             23.1.2
python-dateutil 2.8.2
s3transfer      0.6.1
setuptools      65.5.0
simplejson      3.17.2
six             1.16.0
urllib3         1.26.15

☝️ Those are already part of the image and I would like to avoid re-packaging them again in the lambdex zip file. In some other build tools, these are called "provided" dependencies that come from the environment. Maybe I am missing something but I did not figured out how to do that with pants. Thanks !

cc @broad-processor-92400 . I don't think there is currently a way, other than explicitly removing these deps with the

!!

syntax, but this would be really good to support.

Hi sir. Yes that looks similar to what I want to do.

When I get done free time this weekend I’m going to write those up into an issue and experiment with it a bit

I think

!!

removal is the best option at the moment but it will only work well for direct deps: if those packages appear as transitive deps, they’ll still be included. Eg if a lambda depend on

aws_lambda_powertools

that depends on boto, pex will pull in both even with a

!!boto3

, aiui. The exclusion feature in https://github.com/pantsbuild/pants/issues/19256 is related

Okay, I've read the ticket and the related stuff, and I think this does relate. The common thread here is adding a notion of Python dependencies that exist either outside of Pants' control or are provided by another ecosystem besides pip. I think there's two sides to this: • Creating a module mapping union for third party dependencies or for python dependencies in general that can allow other rules to hook into this and inject targets as deps based on inference results (fourth party dependencies? 😁 ). I've added https://github.com/pantsbuild/pants/issues/19552 to describe that. • Getting Pex to build incomplete resolves, with all the caveat emptor that implies https://github.com/pantsbuild/pex/issues/2097 Given that the nature of how an environment provides dependencies is probably extremely variable, I don't want to try to abstract any further than that for now.

Hi @flat-zoo-31952, @broad-processor-92400, thanks for the replies. For your question, Josh, I would call them "provided" dependencies as this terminology is used in other build tools and I think it's pretty clean. I can also confirm you that we have a lot of cases of "provided" dependencies being pulled as transitive dependencies in our project and I suspect this to be common in contexts like AWS Lambda.

I think what I'm looking at here to start with is a way to make it even possible to reason about these in Pants. As it stands, both the Pants Python backend and Pex assume that Python dependencies can only come from first-party sources and targets, or from the pip ecosystem. The two tickets I linked are what I think it would take to break those assumptions. Each dependency "provider" would have to implement its own way of dealing with this, which is likely to be really specific to the providers themselves. In the case of AWS Lambda, for instance, you'd probably want to be able to query the target runtime about what deps are actually provided there, and then you could mirror those in pip for

pants test

and not include them for

pants package

. That's possible because the dependencies provided are still part of the pip ecosystem, just provided at a different step down the road. For a DEB/RPM provider, we can't assume that pip versions are equal to versions on those system packages, since distro maintainers can and do cherry-pick and backport fixes, so we have to use those distro packages directly (and have a way of making sure they are installed at the versions we expect in the selected environment, and that we're using the right python interpreter to access them). This is complicated, and we'll probably need help figuring out what this looks like for the actual use cases we care about here. I'm going to take a whack at https://github.com/pantsbuild/pants/issues/19552 soon and maybe https://github.com/pantsbuild/pex/issues/2097 and that should at least result in a couple of places you could make a

UnionRule

to support this for Lambda, and I could support this for RPM.