Hi, I'm investigating using pants with a Python mo...
# general
r
Hi, I'm investigating using pants with a Python monorepo and have started by testing pants packaging for AWS lambda and deployment. However, I am encountering an issue when deploying due to the artifact ZIP created by pants being too large. Previously, I used serverless to package for lambda, and had settings to reduce the deployment size. How can I apply these similar settings to the pants packaging step?
h
Hi! Presumably it's the no-deploying of pandas and numpy part that matters for your case?
How are these being provided at runtime?
r
Hi Ben, thanks for responding so quickly! Numpy and pandas are being provided with a custom lambda layer
h
Aha, so Pants 2.18 will support building lambda layers. See here for how this works: https://www.pantsbuild.org/v2.18/docs/awslambda-python
There's an example there for how to have a third-party layer and a first-party layer
if you want something more nuanced (excluding specific third-party deps) then you can use
!!
to exclude a dep. See https://www.pantsbuild.org/v2.18/docs/targets (scroll down to "Ignore dependencies with ! and !!")
I suppose even in Pants 2.17 you could use
!!
to exclude numpy and pandas from your function, assuming you have some other way of building the underlying layer
r
Hi Ben, thanks for your help. I've tried to excluding numpy and pandas in the dependencies field:
Copy code
python_aws_lambda_function(
    name="x",
    output_path='api/x.zip',
    handler="handlers.py:*",
    dependencies=['api/x/y.py', '!!3rdparty/python:numpy', '!!3rdparty/python:pandas'],
    complete_platforms=["api/lambda_config:aws_lambda_python_3_10"]
)
But I get this error:
Copy code
InvalidFieldException: api/analytics/BUILD:3: Failed to get dependencies for api/analytics:x: The file or directory '3rdparty/python' does not exist on disk in the workspace, so the address '3rdparty/python:pandas' from the `dependencies` field from the target api/analytics:x cannot be resolved.
Also we are using pipenv
Actually nvm I got it to work!
Thank you
h
Cool
To emphasize something though: the
!!
syntax is intended to exclude a dep transitively, but that doesn't work through third-party deps, to which Pants is currently blind. So for example if you happened to depend on some third-party distribution foo that happened to depend on
pandas
, it would get pulled in that way regardless
work is underway to support more comprehensive exclusion, but sounds like you have a solution for now
I assume you realized that you want
!!3rdparty/python#numpy
etc?