https://pantsbuild.org/ logo
Join Slack
Powered by
I'm trying to ingore a transitive dependency in a ...
# general
c
I'm trying to ingore a transitive dependency in a 
python_awslambda
target (specifically botocore) but it seems to always include botocore in my zip files
e
You cannot exclude transitive dependencies. You can only exclude your own 1st party direct dependencies.
I think what you want is this feature: https://github.com/pantsbuild/pex/issues/2097
c
You can use the prefix 
!!
to transitively exclude a dependency, meaning that even if a target's dependencies include the bad dependency, the final result will not include the value.
Am I reading this incorrectly or does that not apply to awslambda targets?
e
It does not apply to any 3rdparty transitive dependency in any Python context.
If you depend on boto3 and you also depend on foo and foo depends on boto3, you cannot exclude boto3
👍 1
c
That makes sense. Thanks. Botocore is a dep of boto3 and 
!!
on boto3 worked but botocore is getting added via other deps.
e
Yeah. I'll be adding support for this in Pex, but hopefully you can see it's super dangerous.
How do you know the 3rdparty that needs it won't blow up?
c
Right.
e
Are you doing this to work around zip size? If so that's a losing battle that will just rage on - code always grows over time. Have you rejected switching to lambda support for docker images?
c
We have a mix of zip and docker. Docker is the default right now but there are many cases where I don't want the performance hit from docker
I had an api gateway endpoint with a lambda authorizer, so potentially two cold starts in a row. I was getting some 17 second responses
e
Gotcha. Well the Pex feature should ship this week. No clue how long it will take Pants folks to integrate.
c
is pants peek the best way to get similar info to this:
Copy code
boto3==1.26.61
    # via
    #   -r <http://requirements.in|requirements.in>
    #   moto
Like pip-tools does
e
But if you're hacking already, a PEX is just a zip. Just delete zip entries with 
zip -d
or other tooling / scripting.
💡 1
Well, Pants gets in the way here, but there are a few ways using Pex.
c
So just manually delete the extra deps as part of CI or something?
e
So just manually delete the extra deps as part of CI or something?
Yeah.
So, for the requirements.txt you can: 1. 
pex3 lock export-subset --lock <lockfile> (--platfrom <platform>|--complete-platform <complete platform JSON file>) [requirements ...]
-> that nets a requirerments.txt with pinned reqs + hashes 2. 
pex-tools cowsay.pex repository info -v --indent 2
+ 
jq
magic.
#1 is the most direct, but Pants gets in the way by adding invalid JSON (a somment header) to the top of Pex lockfiles.
So to use 1 you must temporarily strip the Pants headers from the lock file.
For example, in the https://github.com/pantsbuild/example-django repo:
Copy code
jsirois@Gill-Windows:~/dev/pantsbuild/example-django (main) $ pex3 lock export-subset --lock <(grep -v -E "^\s*//" lockfiles/python-default.lock) gunicorn
gunicorn==21.2.0 \
  --hash=sha256:3213aa5e8c24949e792bcacfc176fef362e7aac80b76c56f6b5122bf350722f0 \
  --hash=sha256:88ec8bff1d634f98e61b9f65bc4bf3cd918a90806c6f5c48bc5603849ec81033
packaging==23.1 \
  --hash=sha256:994793af429502c4ea2ebf6bf664629d07c1a9fe974af92966e4b8d2df7edc61 \
  --hash=sha256:a392980d2b6cffa644431898be54b0045151319d1e7ec34f0cfed48767dd334f
You can get the list of requirements from the PEX file itself created from the lock by:
Copy code
$ pex-tools dist/helloworld.service.welcome/gunicorn.pex info | jq -r '.requirements[]'
django-stubs==1.10.1
django<4,>=3.2.13
gunicorn>=20.1.0
requests>=2.25.1
types-requests>=2.25.1
Putting it all together:
Copy code
$ pex3 lock export-subset --lock <(grep -v -E "^\s*//" lockfiles/python-default.lock) $(pex-tools dist/helloworld.service.welcome/gunicorn.pex info | jq -r '.requirements[]')
django-stubs==1.10.1 \
  --hash=sha256:dfffa933e3c7f5cae92c860fbd6cf4ae58c4f78eed93843612f863207c9e0e1a \
  --hash=sha256:2ec21fc14dba392156e0ec8438e1863c86ddb295f1c8d88eecd7e0e04977c843
django==3.2.21 \
  --hash=sha256:d31b06c58aa2cd73998ca5966bc3001243d3c4e77ee2d0c479bced124765fd99 \
  --hash=sha256:a5de4c484e7b7418e6d3e52a5b8794f0e6b9f9e4ce3c037018cf1c489fa87f3c
gunicorn==21.2.0 \
  --hash=sha256:3213aa5e8c24949e792bcacfc176fef362e7aac80b76c56f6b5122bf350722f0 \
  --hash=sha256:88ec8bff1d634f98e61b9f65bc4bf3cd918a90806c6f5c48bc5603849ec81033
requests==2.31.0 \
  --hash=sha256:58cd2187c01e70e6e26505bca751777aa9f2ee0b7f4300988b709f44e013003f \
  --hash=sha256:942c5a758f98d790eaed1a29cb6eefc7ffb0d1cf7af05c3d2791656dbd6ad1e1
types-requests==2.31.0.4 \
  --hash=sha256:c7a9d6b62776f21b169a94a0e9d2dfcae62fa9149f53594ff791c3ae67325490 \
  --hash=sha256:a111041148d7e04bf100c476bc4db3ee6b0a1cd0b4018777f6a660b1c4f1318d
django-stubs-ext==4.2.2 \
  --hash=sha256:fdacc65a14d2d4b97334b58ff178a5853ec8c8c76cec406e417916ad67536ce4 \
  --hash=sha256:c69d1cc46f1c4c3b7894b685a5022c29b2a36c7cfb52e23762eaf357ebfc2c98
mypy==0.942 \
  --hash=sha256:a1b383fe99678d7402754fe90448d4037f9512ce70c21f8aee3b8bf48ffc51db \
  --hash=sha256:17e44649fec92e9f82102b48a3bf7b4a5510ad0cd22fa21a104826b5db4903e2
tomli==2.0.1 \
  --hash=sha256:939de3e7a6161af0c887ef91b7d41a53e7c5a1ca976325f429cb46ea9bc30ecc \
  --hash=sha256:de526c12914f0c550d15924c62d72abc48d6fe7364aa87328337a31007fe8a4f
types-pyyaml==6.0.12.12 \
  --hash=sha256:c05bc6c158facb0676674b7f11fe3960db4f389718e19e62bd2b84d6205cfd24 \
  --hash=sha256:334373d392fde0fdf95af5c3f1661885fa10c52167b14593eb856289e1855062
types-pytz==2023.3.1.1 \
  --hash=sha256:1999a123a3dc0e39a2ef6d19f3f8584211de9e6a77fe7a0259f04a524e90a5cf \
  --hash=sha256:cc23d0192cd49c8f6bba44ee0c81e4586a8f30204970fc0894d209a6b08dab9a
typing-extensions==4.8.0 \
  --hash=sha256:8f92fc8806f9a6b641eaa5318da32b44d401efaac0f6678c9bc448ba3605faa0 \
  --hash=sha256:df8e4339e9cb77357558cbdbceca33c303714cf861d1eef15e1070055ae8b7ef
asgiref==3.7.2 \
  --hash=sha256:89b2ef2247e3b562a16eef663bc0e2e703ec6468e2fa8a5cd61cd449786d4f6e \
  --hash=sha256:9e0ce3aa93a819ba5b45120216b23878cf6e8525eb3848653452b4192b92afed
pytz==2023.3.post1 \
  --hash=sha256:ce42d816b81b68506614c11e8937d3aa9e41007ceb50bfdcb0749b921bf646c7 \
  --hash=sha256:7b4fddbeb94a1eba4b557da24f19fdf9db575192544270a9101d8509f9f43d7b
sqlparse==0.4.4 \
  --hash=sha256:5430a4fe2ac7d0f93e66f1efc6e1338a41884b7ddf2a350cedd20ccc4d9d28f3 \
  --hash=sha256:d446183e84b8349fa3061f0fe7f06ca94ba65b426946ffebe6e3e8295332420c
packaging==23.1 \
  --hash=sha256:994793af429502c4ea2ebf6bf664629d07c1a9fe974af92966e4b8d2df7edc61 \
  --hash=sha256:a392980d2b6cffa644431898be54b0045151319d1e7ec34f0cfed48767dd334f
certifi==2023.7.22 \
  --hash=sha256:92d6037539857d8206b8f6ae472e8b77db8058fec5937a1ef3f54304089edbb9 \
  --hash=sha256:539cc1d13202e33ca466e88b2807e29f4c13049d6d87031a3c110744495cb082
charset-normalizer==3.2.0 \
  --hash=sha256:8e098148dd37b4ce3baca71fb394c81dc5d9c7728c95df695d2dca218edf40e6 \
  --hash=sha256:3bb3d25a8e6c0aedd251753a79ae98a093c7e7b471faa3aa9a93a81431987ace
idna==3.4 \
  --hash=sha256:90b77e79eaa3eba6de819a0c442c0b4ceefc341a7a2ab77d7562bf49f425c5c2 \
  --hash=sha256:814f528e8dead7d329833b91c5faa87d60bf71824cd12a7530b5526063d02cb4
urllib3==2.0.5 \
  --hash=sha256:ef16afa8ba34a1f989db38e1dbbe0c302e4289a47856990d0682e374563ce35e \
  --hash=sha256:13abf37382ea2ce6fb744d4dad67838eec857c9f4f57009891805e0b5e123594
types-urllib3==1.26.25.14 \
  --hash=sha256:9683bbb7fb72e32bfe9d2be6e04875fbe1b3eeec3cbb4ea231435aa7fd6b4f0e \
  --hash=sha256:229b7f577c951b8c1b92c1bc2b2fdb0b49847bd2af6d1cc2a2e3dd340f3bda8f
mypy-extensions==1.0.0 \
  --hash=sha256:4392f6c0eb8a5668a69e23d168ffa70f0be9ccfd32b5cc2d26a34ae5b844552d \
  --hash=sha256:75dbf8955dc00442a438fc4d0666508a9a97b6bd41aa2f0ffe9d2f2725af0782
Thats a pip-style lock file with just the deps needed by a particular PEX file, no more, no less.
c
On 2.17, is python_awslambda even using pex anymore?
We've switched over to 
layout = "zip"
e
Yes. And even more so.
c
oh haha
e
The end product is a zip but its created by Pex.
c
I was looking at the contents of the zip
Which is even easier to just drop deps out of but like you said... also scary
e
Yeah, its a few options to the 
pex-tools <PEX file> venv create
tool that make the zip.
Basically you can turn any PEX file into a venv. With the few extra options you can omit the venv interpreter and just export the site-packages effectively and prefix the xport with a dir name, etc.
I was wrong about 
pex-tools
, this is a 
pex3 venv create
feature. Pants is ~doing this:
Copy code
$ pex3 venv create --python python3.9 --prefix python --layout flat-zipped --lock <(grep -v -E "^\s*//" lockfiles/python-default.lock) $(pex-tools dist/helloworld.service.welcome/gunicorn.pex info | jq -r '.requirements[]') -d dist/lambda
$ zipinfo dist/lambda.zip | head
Archive:  dist/lambda.zip
Zip file size: 27578044 bytes, number of entries: 9415
drwxr-xr-x  2.0 unx        0 b- stor 80-Jan-01 00:00 python/Django-3.2.21.dist-info/
-rw-r--r--  2.0 unx    38712 b- defN 80-Jan-01 00:00 python/Django-3.2.21.dist-info/AUTHORS
-rw-r--r--  2.0 unx        4 b- defN 80-Jan-01 00:00 python/Django-3.2.21.dist-info/INSTALLER
-rw-r--r--  2.0 unx     1552 b- defN 80-Jan-01 00:00 python/Django-3.2.21.dist-info/LICENSE
-rw-r--r--  2.0 unx    13227 b- defN 80-Jan-01 00:00 python/Django-3.2.21.dist-info/LICENSE.python
-rw-r--r--  2.0 unx     4064 b- defN 80-Jan-01 00:00 python/Django-3.2.21.dist-info/METADATA
-rw-r--r--  2.0 unx   373663 b- defN 80-Jan-01 00:00 python/Django-3.2.21.dist-info/RECORD
-rw-r--r--  2.0 unx        0 b- defN 80-Jan-01 00:00 python/Django-3.2.21.dist-info/REQUESTED
$ zipinfo dist/lambda.zip | tail
-rw-r--r--  2.0 unx     1151 b- defN 80-Jan-01 00:00 python/yaml-stubs/loader.pyi
-rw-r--r--  2.0 unx     1032 b- defN 80-Jan-01 00:00 python/yaml-stubs/nodes.pyi
-rw-r--r--  2.0 unx     1672 b- defN 80-Jan-01 00:00 python/yaml-stubs/parser.pyi
-rw-r--r--  2.0 unx     1038 b- defN 80-Jan-01 00:00 python/yaml-stubs/reader.pyi
-rw-r--r--  2.0 unx     3363 b- defN 80-Jan-01 00:00 python/yaml-stubs/representer.pyi
-rw-r--r--  2.0 unx      787 b- defN 80-Jan-01 00:00 python/yaml-stubs/resolver.pyi
-rw-r--r--  2.0 unx     3573 b- defN 80-Jan-01 00:00 python/yaml-stubs/scanner.pyi
-rw-r--r--  2.0 unx      729 b- defN 80-Jan-01 00:00 python/yaml-stubs/serializer.pyi
-rw-r--r--  2.0 unx     1796 b- defN 80-Jan-01 00:00 python/yaml-stubs/tokens.pyi
9415 files, 81647571 bytes uncompressed, 25953272 bytes compressed:  68.2%
c
I feel like I need to learn way more about pex
Or at least I want to
I ultimately think I need to narrow down the source of some of the deps in my zips.
e
That info is in the lockfile.
c
There's something getting added in by importing something across our own code base... Like I do 
from .shared_code import helper_function
and 
helper_function
is doing something crazy with deps
e
Ok. Well just follow 
"requires_dists"
lists in the lockfile and you can track down how a dep gets pulled in by the root 
"requirements"
list.
👍 1
Or, 
pex-tools <pex> graph -h
and check out the options, but it'll open a browser with an svg of the dep graph.
👀 2
c
That sounds cool
e
Well, it'll open whatever you have associated with svg files - if anything.
And you'll need graphviz / dotty - but it will tell you that if it finds 
dot
is not installed.
For example - using the same django example PEX from above:
c
This is the first time I even installed pex so it's definitely going to take me a bit to get things figured out. Really appreciate the help though!
e
Well - there are ~0 docs, just `-h`; so it will take a while to figure things out. I took over as maintainer in ~2018 and ground out alot of features and fixes largely driven by Pants' needs, but have not yet had time to circle back and re-vamp / expand / make docs good.
2 Views
Open in Slack
PreviousNext
Powered by