https://pantsbuild.org/ logo
Join Slack
Powered by
Hello everyone, Is it possible to exclude a 3rd pa...
# general
l
Hello everyone, Is it possible to exclude a 3rd party dependency that is used by another 3rd party dependency? For example, I'm using a 
boto3
and I want to exclude 
botocore
which is used by 
boto3
When I list my dependencies (
--transitive
flag is on), I only see 
boto3
so can't specify a correct target to exclude 
botocore
b
It's not currently easily possible. For context, is this for building a lambda or a docker image or something else?
l
I want to build a Lambda layer but it's getting too large because of 
numpy
dependency which I want to exclude and use it as separate layer
b
ah okay. yeah, that's definitely a thing. A few points: ā€¢ https://github.com/pantsbuild/pants/issues/19256 is a relevant issue about having this work smoothly ā€¢ I believe the 250MB unzipped limit includes any layers; are you hitting that limit or the 50MB zip-upload one? ā€¢ Botocore 1.32.1, released a few days ago, is like 20MB instead of 80MB: https://github.com/boto/botocore/issues/2365#issuecomment-1813447304
(and, side note, https://github.com/pantsbuild/pants/issues/12733 captures the behaviour of 
pants dependencies --transitive
not working well for transitive dependencies of PyPI packages.)
l
Thank you for the replay šŸ™Œ, Actually it's not really a size problem at this moment (but it can be soon if I add other dependencies) but a problem with packaging a 
numpy
itself. My lambda and layers are getting deployed but lambda fails to run with this error:
Copy code
Error importing numpy: you should not try to import numpy from its source directory; please exit the numpy source tree, and relaunch your python interpreter from there.
I'm aware of 
[python.resolves_to_only_binary]
but it's packaging it the same way and the problem remains I'm using pants version 
2.18.0a0
and 
python_aws_lambda_layer
to build this layer
So it though excluding this package would be easier
b
Ah. Hm, that's very weird and unexpected error. Can you provide your 
pants.toml
file and definition of the target you're building?
Also, can you upgrade to 
2.18.0
and see if that resolves the error, in case this is a problem we already fixed?
l
Copy code
[GLOBAL]
pants_version = "2.18.0a0"
pants_distdir = ""

backend_packages = [
    "pants.backend.awslambda.python",
    "pants.backend.python"
]


[python]
interpreter_constraints = ["CPython==3.9.*"]
enable_resolves = true

[python.resolves_to_only_binary]
python-default = ["numpy"]
__default__ = ["numpy"]
Copy code
python_requirements(name="reqs")

python_aws_lambda_layer(
    name="layer",
    dependencies=[":reqs"],
    include_sources=False,
    complete_platforms=["3rdparty/platforms:macos_python_3_9"],
    output_path="layers/yfinance/layer.zip"
)
b
Ah, hm, that 
3rdparty/platforms:macos_python_3_9
makes me suspicious: Lambda runs on Linux, so a complete platform file for macOS is a bit unexpected. Pants 2.18.0 comes with its own complete platforms (for lambdas); does it work if you remove that field?
l
Copy code
Successfully built the wheel peewee-3.17.0-cp39-cp39-macosx_11_0_arm64.whl from the sdist peewee-3.17.0.tar.gz but it is not compatible with the requested foreign target complete platform cp39-cp39-manylinux_2_26_x86_64.
this is without 
3rdparty/platforms:macos_python_3_9
Also, can you upgrade to 
2.18.0
and see if that resolves the error, in case this is a problem we already fixed?
This is a another problem I'm having right now. It's not updating but I was able to update form 
2.17.0
to 
2.18.0a0
before. I'm getting this:
mb I need to delete a cache or something?
b
Ah okay, so in that case, the error is genuine: it's not (easily) possible to get an artifact that's compatible with Lambda while on your platform, because it only comes in source form and seems to have some native code that needs to be compiled on the target platform. This is effectively a fundamental limitation. You have a few options: ā€¢ find or build a binary wheel for peewee and use that ā€¢ remove the peewee dependency ā€¢ build on the target platform directly, e.g. using an environment https://www.pantsbuild.org/docs/environments
This is a another problem I'm having right now. It's not updating but I was able to update form 
2.17.0
to 
2.18.0a0
before. I'm getting this:
Hm, that's very weird.
Oh, I think the 
export SCIE_BOOT=update pants
line isn't doing what you expect. It's setting two environment variables: 
SCIE_BOOT=update
and 
pants
.
ā€¢ To update the pants launcher run 
SCIE_BOOT=update pants
without the export ā€¢ To update the pants version (which the launcher coordinates installing and running), edit 
pants.toml
to set 
pants_version
l
Does it mean If I find a binary wheel for 
peewee
, I won't have a problem with 
numpy
missing binaries that I'm having right now? Just don't want to spend time for fixing peewee and eventually facing the same problem?
b
I suspect the numpy problems are coming from trying to use macOS numpy in the Linux lambda environment. One way to confirm this would be reduce things a lot, e.g. cut down a lambda/layer to depend only on 
numpy
and get that working, and then expand from there
l
Copy code
[GLOBAL]
pants_version = "2.18.0"
b
Just checking; do you have an existing build process for lambdas/layers that you're trying to switch to pants, or are you creating a new process starting in pants?
Ah, is in the same shell as the 
export
? You might need to unexport 
SCIE_BOOT
, e.g. 
unset SCIE_BOOT
or start a new shell.
šŸ‘€ 1
l
Yay šŸ™‚
šŸŽ‰ 1
Thank you for your time answering my questions, I appreciate it šŸ™Œ I'll try your suggestion with only 
numpy
dependency
5 Views
Open in Slack
PreviousNext
Powered by