Hey there dear people! I am building a lambda zip, but unfortunately with all the libs unzipped it exceeds the 250mb limit. I tried to follow this https://github.com/pantsbuild/pants/discussions/18756 and add

dependencies=["!!requirements.txt#mypy", "!!requirements.txt#sqlalchemy-stubs"]

in my

python_awslambda

target, like:

python_awslambda(
    name="lambda",
    handler="app.py:lambda_handler",
    complete_platforms=["3rdparty/platforms:aws_lambda_python_3_10"],
    dependencies=["!!requirements.txt#mypy", "!!requirements.txt#sqlalchemy-stubs"]
)

This BUILD file is in

projects/tag

and

requirements.txt

in the repository root. Running

pants package

gives the error: `InvalidFieldException: projects/tag/BUILD24 Failed to get dependencies for projects/taglambda Addresses for generated first-party targets in the build root must include which target generator they come from, such as

requirements.txt:original_target

. However,

requirements.txt#mypy

from the

dependencies

field from the target projects/tag:lambda did not have a target name.` How can I link my

requirements.txt

in the root to my BUILD file in

projects/tag

? I would really appreciate the helps as always 🙏 thanks so much in advance!

Pietro are the dependency addresses exactly as they appear when you

pants list ::

?

You're great @late-advantage-75311 thanks a lot! now i see the right syntax

//:reqs#ansicolors
//:reqs#apispec
//:reqs#aws-lambda-context
//:reqs#aws-lambda-powertools
//:reqs#chalice
//:reqs#chalice-spec
//:reqs#jmespath
//:reqs#langchain
//:reqs#matplotlib
//:reqs#numpy
//:reqs#openai
//:reqs#opencv-python-headless
//:reqs#pillow
//:reqs#pydantic
//:reqs#pyspellchecker
//:reqs#pytest
//:reqs#python-dotenv
//:reqs#replicate
//:reqs#requests
//:reqs#setuptools
//:reqs#types-setuptools
//:reqs#webcolors

the thing is that now

mypy

doesn't appear here (because it's not an explicit dependency in my requirements.txt file) but I can see it in my .lock file (probably it is a transitory dependency) - and same for

sqlalchemy-stubs

is there a way to exclude them without having them as explicit dependencies?

oh ... interesting ...

yep, i can see them in my unzipped lambda folder, and they would get me right below the limit

Uff... I have to do some digging to be able to help. I hope maybe someone else already in-the-know can chime in meanwhile.

sounds good dear gautham thanks a lot for your help!

It's impossible to exclude transitive 3rd party deps.

it's a valid point dear john, I was following what was written in the issue: https://github.com/pantsbuild/pants/discussions/18756

@proud-byte-81916 you paint by numbers too much!

thanks a lot for claryfing, appreciate your patience and help dear john 🙂 how can i find these deps, if any?

Find what deps?

It seems Huon ran into the same issue with the same libraries: https://github.com/pantsbuild/pants/issues/15454#issuecomment-1185155133 . The reason why it is safe to remove the deps is because the kind of offending package here includes its type stubs and pulls mypy in, even though it is not needed at runtime.

Sure, that was the case for @broad-processor-92400. @proud-byte-81916 is that the case for you too or are you just blindly following the advice?

yes, Pietro did so

just curious - these two are very large libs that if you removed them would get you down below the 250 mb limit?

yep, i can see them in my unzipped lambda folder, and they would get me right below the limit

@late-advantage-75311 I don't think so

I checked it again, and I have specifically

sqlalchemy

SQLAlchemy-2.0.21.dist-info

that i believe are not needed

Why is mypy in your exclude list?

with mypy i have

mypy_extensions-1.0.0.dist-info

, I thought was the same as the issue above but probably not as it's becoming clear from our conversation

So, have you 1st tried manually

zip -d

deleting all

.pyi

files to see if your size reduction effort will even work to bring the zip under the limit?

I see your point dear John, but yes that works

Incidentally, the way Huon was able to work around it appeared to include declaring an explicit dependency on these distributions that were getting pulled in transitively and then excluding them explicitly in the package: https://github.com/huonw/pants-awslambda-UndefinedEnvironmentName/compare/182145d2[…]888cb0df74916589dc9a...94bdc6fe84a0dec263e81a6a13e4278352d77e34

That makes sense.

Thanks a lot dear Gautham! I was checking that as well, and added this:

python_requirement(name="sql-alchemy", requirements=["SQLAlchemy==2.0.21"])

python_awslambda(
    name="lambda",
    handler="app.py:lambda_handler",
    complete_platforms=["3rdparty/platforms:aws_lambda_python_3_10"],
    dependencies=["!!:sql-alchemy"]
)

In my zip file, I still get the folder

sqlalchemy

SQLAlchemy-2.0.21.dist-info

Is there something I am missing?

That gets you a bunch of breathing room without having to get hacky.

Yep that's a good solution dear John thanks a lot! Just wanted to check if it was possible also to solve it through zip as I'm very close to the limit, and also to get a better understanding in general of how this work

Pietro, you need to add explicit dependencies to the transitive dependency on

mypy

python_requirement(name="mypy", requirements=["mypy"])

. Then exclude it from the awslambda dependencies.

Ok, well @late-advantage-75311 unless things have changed, I think I'm right about transitive deps. You cannot exclude them. Pex will always add them; so Pants would have to be doing the

zip -d

I mention.

Thanks a lot guys for your help really 🙏 good point! it comes from langchain

So do you configure a DB for it to use or does it have a sqlite style embedded DB file?

I don't but probably the library uses it internally for some operation, possibly with vector databases - but that's not the case for what I am using langchain so should be safe to remove it

Absolutely.

I will do that then 🙂 thanks again for the help!

If and only if the dependency is only non-transitive.

Learned from this thread. John, thank you.

You're welcome. @proud-byte-81916 I'm hoping to get you to the point of doing some of this back and forth in your own head. You clearly know enough to answer yourself for some of this.

Yes thanks a lot dear John, and also thank you dear Gautham! I am definitely getting there, just I am still confused on these details and it's great to get clarified - every one of these threads bring me closer to the back and forth in my head for sure