Hello, I am using the

python_aws_lambda_layer

integration to make a layer that includes

redshift-connector

and it works great except that I can't seem to get it to not package boto3/botocore. I am trying like this

python_aws_lambda_layer(
    name="redshift_connector_layer",
    dependencies=[
        "!!//:reqs0#boto3",
        "!!//:reqs0#botocore",
        "//:reqs0#redshift-connector",
    ],
    include_sources=False,
)

I would like to exclude boto3/botocore since it's already in the lambda runtime

@broad-processor-92400 thoughts on this?

Yeah, excluding transitive dependencies was not supported by the underlying pex tool, so hasn't been implemented in Pants. Related issues: • https://github.com/pantsbuild/pants/issues/19256 • https://github.com/pex-tool/pex/issues/2097 • https://github.com/pantsbuild/pants/issues/12733 The first one has a comment indicating that potentially pex does have the

--exclude

flag required, so we could expose that. Are you interested in implementing it?

I am definitely willing to take a look. I must say that I haven't been in the pants code base but I'm taking a look around now. I shouldn't make any promises that I'll have the bandwidth either 😄

Yeah, I think it's a feature that applies to

pex_binary

etc. too. I'm not exactly sure the best way to expose it. Potentially 4 options for doing it in a dedicated way: 1. translate the

!!...

exclusions of

python_requirement

targets into

--exclude

arguments somehow 2. have an

exclude=["//:reqs0#boto3"]

field that takes target addresses, that are translated into PyPI names for

--exclude

3. have an

exclude=["boto3"]

that takes PyPI package names that just get passed to

--exclude

directly 4. both 2 and 3 (i.e.

exclude=["//:reqs0#boto3", "botocore"]

would pass

--exclude=boto3 --exclude=botocore

) An extra choice would be adding an escape hatch for passing arbitrary args to the underlying pex creation, similar to https://github.com/pantsbuild/pants/pull/20737 / https://github.com/pantsbuild/pants/pull/20237. This last one would be good as a stop-gap, to unlock this feature (and any future ones pex has) without having to solve "harder" design problems.

I like the escape hatch idea because of the additional flexibility you would get but as a user, I feel like options 2,3,4 are the nicer experience.

Yeah, I think we should offer both: escape hatch to give users/us arbitrary control + tailored helpers for "known" tasks (like this one). How do you feel about implementing the escape hatch as the very first step? maybe a new field

pex_build_extra_args

that is splatted at the end of https://github.com/pantsbuild/pants/blob/40e8bc94803a70f42b2c2889f73573c914c153dd/src/python/pants/backend/python/util_rules/faas.py#L454-L460 to be passed to the

PexFromTargetsRequest

below?

@rule
async def build_python_faas(
    request: BuildPythonFaaSRequest,
    pex_build_extra_args,
) -> BuiltPackage:
    additional_pex_args = (
        # Ensure we can resolve manylinux wheels in addition to any AMI-specific wheels.
        "--manylinux=manylinux2014",
        # When we're executing Pex on Linux, allow a local interpreter to be resolved if
        # available and matching the AMI platform.
        "--resolve-local-platforms",
    ) + pexbuild_extra_args

Would that be something as simple as that?

yeah, preferably added to

BuildPythonFaaSRequest

. I'd be expecting the changes to be very similar to https://github.com/pantsbuild/pants/pull/20237/files#diff-48c7912f30acce7d7d55171a4f13831981e2aaae3660cf853016dd622a13c5bf (and the rest of that PR), in terms of type structure and how things are set up, just the details of the docs, which invocation it is passed to and the tests that differ.

Ok. Let me poke around at that over this weekend. That PR is actually really helpful. If I get stuck or completely in over my head I'll reach out.

great, let me know how you go, more than happy to help (btw, I saw https://github.com/pantsbuild/example-serverless/pull/3, not 100% sure when I'll have a chance to look; thanks for that)

The type definitions throughout this repo 🤌

PR alone is fine. Looks great. There's one minor change: rather than the

if

+

+=

, I'd suggest using a

*

splat into the original tuple construction:

additional_pex_args = (
   ...
   *request.pex_build_extra_args.value,
)

I prefer your way as well but mypy doesn't seem to like that since

request.pex_build_extra_args.value

can be None.

Ah, sweet! It's... a bit annoying that pex would throw an error about a dependency that's been explicitly excluded 😢

Yeah. I suppose I could raise an issue in the pex repo for that because we probably don’t want to mask other potential errors.

Reviewed! looks great

I'll take care of those comments. Thanks for the review!