https://pantsbuild.org/ logo
Join Slack
Powered by
Hi everyone. I'm trying to understand the followin...
# general
p
Hi everyone. I'm trying to understand the following error while trying to run a Django app via AWS Lambda:
Copy code
[ERROR] ResolveError: Failed to resolve requirements from PEX environment @ /app.
Needed cp38-cp38-manylinux_2_26_x86_64 compatible dependencies for:
 1: pillow>=6.2.0
    Required by:
      matplotlib 3.5.1
    But this pex had no ProjectName(raw='pillow', normalized='pillow') distributions.
 2: pillow~=9.2.0
    raise ResolveError(/pex/environment.py", line 589, in resolve_distsp_pex_envions.
I have 
pillow ~= 9.2.0
and 
matplotlib==3.5.1
as dependencies in my requirements.txt file. When I build my Lambda pacakge locally, I see 
Pillow-9.2.0-cp38-cp38-manylinux_2_28_x86_64.whl
in my .deps directory. I figured Pillow ~= 9.2.0 would match matplotlib's Pillow version requirements? What am I doing wrong?
e
Needed cp38-cp38-manylinux_2_26_x86_64 compatible dependencies
vs `Pillow-9.2.0-cp38-cp38-manylinux_2_28_x86_64.whl`sounds like a prolem to me. That's needs glibc 2.26 compatibility but has glibc 2.28 compatibility.
What is your configuration for the relevant target? Is it a 
python_awslambda
target?
p
Yes, it's this:
Copy code
python_awslambda(
    name="lambda",
    runtime="python3.8",
    handler="handler.py:handler",
)
e
Yeah, so since this appears to be glibc titchy and there is a compatible wheel out there (this one: https://files.pythonhosted.org/packages/20/cb/261342854f01ff18281e97ec8e6a7ce3beaf8e1091[…]-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl), you'll need to remove the 
runtime
parameter and use `complete_platforms`instead: https://www.pantsbuild.org/docs/reference-python_awslambda#codecomplete_platformscode That means you'll need to: 1. Get your hands on the target AWS image so you can run some commands to come up with a complete platform description of the machine. 2. Then take the JSON blob from 2 and check it in to your repo with a 
file
target owning it that the `python_awslambda`can refer to.
That step 1 is described in more detail (as linked in the Pants docs) over in the Pex docs: https://pex.readthedocs.io/en/latest/buildingpex.html#complete-platform
A bit of work to do, but speak up if you get stuck.
It would probably be great for Pants to ship some known complete platforms for AWS targets of course!
👍 1
p
Sorry, do you mind elaborating a bit? Maybe an example you can point to for setting 
complete_platforms
?
e
That's pretty elaborate! How about let me know which step is confusing.
p
Is this in the right ballpark?
Copy code
python_awslambda(
    name="lambda",
    # runtime="python3.8",
    handler="handler.py:handler",
    complete_platforms=['{"marker_environment": "python_version >= '3.8'","compatible_tags": ["cp38-cp38-manylinux_2_26_x86_64"]}'],
)
e
No. So, the 1st step is the most involved. The actual complete platforms really should be made via program and not by hand. That's the 
pex3
command described in https://pex.readthedocs.io/en/latest/buildingpex.html#complete-platform
You'll need to get pex (and thus pex3) installed in a venv on the targeted AWS image using its Python 3.8.
Once you've done that, you'll have a very much bigger JSON blob, hundreds of object fields - nearly a thousand. Save that in a file next to your BUILD for simplicity and then something like:
Copy code
file(name="aws-3.8-cp", source="aws-3.8-cp.json")

python_awslambda(
    ...
    complete_platforms=[":aws-3.8-cp"],
)
p
Ah okay. So I generated that JSON file called aws-38-cp.json to get the markers, saved it locally:
Copy code
{
  "path": "/var/lang/bin/python3.8",
  "compatible_tags": [
    "cp38-cp38-manylinux_2_26_x86_64",
    "cp38-cp38-manylinux_2_25_x86_64",
    "cp38-cp38-manylinux_2_24_x86_64",
    "cp38-cp38-manylinux_2_23_x86_64",
    "cp38-cp38-manylinux_2_22_x86_64",
    "cp38-cp38-manylinux_2_21_x86_64",
    "cp38-cp38-manylinux_2_20_x86_64",
    "cp38-cp38-manylinux_2_19_x86_64",
    ...
I updated the BUILD file accordingly:
Copy code
python_awslambda(
    name="lambda",
    handler="handler.py:handler",
    complete_platforms=[":aws-38-cp"],
)

...

file(name="aws-38-cp", source="aws-38-cp.json")
But I'm still seeing the same issue unfortunately. I see that the highest-compatible tag is 
cp38-cp38-manylinux_2_26_x86_64
in the JSON, while Pillow expects 2.28. Is there something I can do to resolve this?
e
That's the only wheel out on PyPI that fits the tags. You could confirm that the 
cp38-cp38-manylinux_2_17_x86_64
and / or 
cp38-cp38-manylinux2014_x86_64
tags are in your JSON 1st.
p
Yeah, they're both in there:
Copy code
{
  "path": "/var/lang/bin/python3.8",
  "compatible_tags": [
    "cp38-cp38-manylinux_2_26_x86_64",
    "cp38-cp38-manylinux_2_25_x86_64",
    "cp38-cp38-manylinux_2_24_x86_64",
    "cp38-cp38-manylinux_2_23_x86_64",
    "cp38-cp38-manylinux_2_22_x86_64",
    "cp38-cp38-manylinux_2_21_x86_64",
    "cp38-cp38-manylinux_2_20_x86_64",
    "cp38-cp38-manylinux_2_19_x86_64",
    "cp38-cp38-manylinux_2_18_x86_64",
    "cp38-cp38-manylinux_2_17_x86_64",
    "cp38-cp38-manylinux2014_x86_64",
    "cp38-cp38-manylinux_2_16_x86_64",
    "cp38-cp38-manylinux_2_15_x86_64",
    "cp38-cp38-manylinux_2_14_x86_64",
    "cp38-cp38-manylinux_2_13_x86_64",
    "cp38-cp38-manylinux_2_12_x86_64",
    "cp38-cp38-manylinux2010_x86_64",
    "cp38-cp38-manylinux_2_11_x86_64",
    "cp38-cp38-manylinux_2_10_x86_64",
Do I need to destroy a cache somewhere in order for this change to propagate?
e
You should not need to, no.
You could try adding 
--no-local-cache
though - it will be slow.
p
Darn, unfortunately didn't work. I also tried moving up "cp38-cp38-manylinux_2_17_x86_64" to the top of the 
compatible_tags
list just in case order matters (in a truly desperate move)
e
This is making 0 sense. So the resulting PEX in dist/ still has a 2_28 glibc dep? You definitely deleted the runtime field in favor of complete platforms?
p
Yes, it unfortunately still has 
Pillow-9.2.0-cp38-cp38-manylinux_2_28_x86_64.whl
. I also tried settings 
compatible_tags
to just one value, 
cp38-cp38-manylinux_2_17_x86_64
, with hope that it would "force-pull" the compatible Pillow whl, but it still resulted in 2_28.
e
Hm, just 1 value should definitely fail the build; so that implies complete_platforms is not being used at all. Two questions: 1. What version of Pants is this? 2. WHat is the full output of the 
./pants ...
command if you add 
-ldebug
- if you can share that.
p
1. I'm using 2.12.0 For 2:
Copy code
17:44:27.30 [DEBUG] acquiring lock: <pants.pantsd.lock.OwnerPrintingInterProcessFileLock object at 0x7f8d726c0df0>
17:44:27.30 [DEBUG] releasing lock: <pants.pantsd.lock.OwnerPrintingInterProcessFileLock object at 0x7f8d726c0df0>
17:44:27.30 [DEBUG] Connecting to pantsd on port 37733
17:44:27.30 [DEBUG] Connecting to pantsd on port 37733 attempt 1/3
17:44:27.30 [DEBUG] Connected to pantsd
17:44:27.31 [DEBUG] Launching 1 roots (poll=false).
17:44:27.31 [DEBUG] Dependency SessionValues of Some("@rule(pants.engine.internals.options_parsing.parse_options())") changed.
17:44:27.32 [DEBUG] Dependency @rule(pants.engine.internals.options_parsing.parse_options()) of Some("@rule(pants.engine.internals.options_parsing.scope_options())") changed.
17:44:27.32 [DEBUG] computed 1 nodes in 0.008025 seconds. there are 7 total nodes.
17:44:27.35 [DEBUG] Launching 1 roots (poll=false).
17:44:27.35 [DEBUG] computed 1 nodes in 0.000466 seconds. there are 7 total nodes.
17:44:27.38 [DEBUG] specs are: Specs(address_specs=AddressSpecs(literals=(), globs=(), filter_by_global_options=True), filesystem_specs=FilesystemSpecs(file_includes=(FileLiteralSpec(file='src/cort/handler.py'),), dir_includes=(), ignores=()), from_change_detection=False)
17:44:27.38 [DEBUG] changed_options are: ChangedOptions(since=None, diffspec=None, dependees=<DependeesOption.NONE: 'none'>)
17:44:27.38 [DEBUG] Launching 1 roots (poll=false).
17:44:27.38 [DEBUG] Dependency SessionValues of Some("@rule(pants.engine.internals.options_parsing.parse_options())") changed.
17:44:27.38 [DEBUG] Dependency @rule(pants.engine.internals.options_parsing.parse_options()) of Some("@rule(pants.engine.internals.options_parsing.scope_options())") changed.
17:44:27.38 [DEBUG] Dependency @rule(pants.engine.internals.options_parsing.parse_options()) of Some("@rule(pants.engine.internals.options_parsing.scope_options())") changed.
17:44:27.38 [DEBUG] computed 1 nodes in 0.001720 seconds. there are 19920 total nodes.
17:44:27.38 [DEBUG] requesting <class 'pants.core.goals.package.Package'> to satisfy execution of `package` goal
17:44:27.38 [DEBUG] Launching 1 roots (poll=false).
17:44:27.38 [DEBUG] Dependency @rule(pants.engine.internals.options_parsing.parse_options()) of Some("@rule(pants.engine.internals.options_parsing.scope_options())") changed.
17:44:27.38 [DEBUG] Dependency @rule(pants.engine.internals.options_parsing.parse_options()) of Some("@rule(pants.engine.internals.options_parsing.scope_options())") changed.
17:44:27.40 [DEBUG] Dependency @rule(pants.engine.internals.options_parsing.parse_options()) of Some("@rule(pants.engine.internals.options_parsing.scope_options())") changed.
17:44:27.40 [DEBUG] Dependency @rule(pants.engine.internals.options_parsing.parse_options()) of Some("@rule(pants.engine.internals.options_parsing.scope_options())") changed.
17:44:27.40 [DEBUG] Dependency @rule(pants.engine.internals.options_parsing.parse_options()) of Some("@rule(pants.engine.internals.options_parsing.scope_options())") changed.
17:44:27.40 [DEBUG] Dependency @rule(pants.engine.internals.options_parsing.parse_options()) of Some("@rule(pants.engine.internals.options_parsing.scope_options())") changed.
17:44:27.40 [DEBUG] Dependency @rule(pants.engine.internals.options_parsing.parse_options()) of Some("@rule(pants.engine.internals.options_parsing.scope_options())") changed.
17:44:27.41 [DEBUG] Dependency @rule(pants.engine.internals.options_parsing.parse_options()) of Some("@rule(pants.engine.internals.options_parsing.scope_options())") changed.
17:44:27.41 [DEBUG] Dependency @rule(pants.engine.internals.options_parsing.parse_options()) of Some("@rule(pants.engine.internals.options_parsing.scope_options())") changed.
17:44:27.41 [DEBUG] Dependency @rule(pants.engine.internals.options_parsing.parse_options()) of Some("@rule(pants.engine.internals.options_parsing.scope_options())") changed.
17:44:27.41 [DEBUG] Dependency @rule(pants.engine.internals.options_parsing.parse_options()) of Some("@rule(pants.engine.internals.options_parsing.scope_options())") changed.
17:44:27.41 [DEBUG] Dependency SessionValues of Some("@rule(pants.engine.environment.environment_subset())") changed.
17:44:27.41 [DEBUG] Dependency SessionValues of Some("@rule(pants.engine.environment.environment_subset())") changed.
17:44:27.73 [INFO] Wrote dist/src.cort/lambda.zip
    Complete platform: src/cort/aws-38-cp.json
              Handler: lambdex_handler.handler
17:44:27.73 [DEBUG] Completed: `package` goal
17:44:27.73 [DEBUG] computed 1 nodes in 0.353665 seconds. there are 19922 total nodes.
e
Right, if you can possibly re-run with 
-ldebug --no-local-cache
- the interesting bits are skipped there.
p
e
Ok, so that's using a `--lock`of 
requirements-lock.txt
- what hashes of pillow does it have locked?
Is that a hand-rolled lock or a Poetry exported lock?
p
Yeah, I used 
./pants generate-lockfiles
to generate the lockfile. No Poetry on our end. This is what our pants.toml looks like:
Copy code
...

[python]
enable_resolves = true
interpreter_constraints = ['CPython>=3.8,<4']
default_resolve = 'python-default'

[python.resolves]
python-default = "requirements-lock.txt"

...
Looking at requirements-lock.txt, we see Pillow as a dependency to imageio (pillow >= 8.3.2) and as a dependency to matplotlib (pillow >= 6.2.0) on top of our own explicit dependency (~9.2.0).
e
Ok, I think Pants 2.12 still defaults to using Poetry under the covers for locking. Are you in a position to try switching to the Pex locker? Poetry locker support goes away in 2.15, so eventually the switch will happen anyhow.
Hrm, the 2.12 docs claim is Pex is the default locker: https://www.pantsbuild.org/docs/reference-python#section-lockfile-generator
p
Oh, but I thought we were already using Pex locker?
Untitled.txt
Untitled.txt
Here's what our requirements.txt file looks like, for reference ^
e
Aha - ok, you showed you used this above - the .txt name threw me off.
Copy code
[python.resolves]
python-default = "requirements-lock.txt"
Looking at your lock now...
Ok, yeah, this looks like it has to be a Pex lockfile reading bug. It's not appearing to honor --complete-platforms. Let me check the changelog though, that's a fairly old Pex by now. Its 2.1.90 and 2.1.104 is out there.
Do you mind sharing your complete platform file for a repro attempt?
Actually, I think --resolve-local-platforms is the problem here. You appear to be on a linux machine running this and that linux machine has a CPython 3.8 on the PATH - is that correct?
I think it must be.
Ah, yeah, WSL - so Ubuntu on Windows.
So, if you were to take Python 3.8 off your PATH for the command run, you should get a different lambda.zip this time. IOW, the `--resolve-local-platforms`will not find a matching local interpreter for the `--complete-platform`and will be forced to used the 
--complete-platform
description.
So, in short, this looks like a Pex bug either way, but if you get a chance to provide me with your complete platform json, I can confirm it and file the appropriate Pex bug.
p
Ok will do - I'll send it to you ASAP. Thank you so much for your help so far!
Hi John, sorry for the delay. Here is the platforms JSON as requested.
Untitled.txt
e
No worries at all. You've been a great debugging partner. So that confirms the issue is 
--resolve-local-platforms
. On WSL if I use the pex command from the 
-ldebug
logs you provided I too get a lambda.zip where:
Copy code
(pex.venv) jsirois@Gill-Windows:~$ pex-tools ./lambda.zip info -i2 | grep -i pillow
    "Pillow-9.2.0-cp38-cp38-manylinux_2_28_x86_64.whl": "d0c4bca60376f84608c7e5ed5d261c4c937cbf1195fcf259f2de49910d56d9d6",
    "pillow~=9.2.0",
Now if I re-run but drop the `--resolve-local-platforms`I get:
Copy code
(pex.venv) jsirois@Gill-Windows:~$ pex "Django<4" "architect==0.6.0" "boto3==1.24.36" "clickhouse_driver==0.2.4" "dill==0.3.3" "django-admin-rangefilter==0.8.1" "django-admin-row-actions==0.0.5" "django-cors-headers==3.11.0" "django-currentuser<0.6.0" "django-debug-toolbar==3.2.4" "django-filter==2.4.0" "django-json-widget==1.1.1" "django-jsonschema-form==1.0.3" "django-opensearch-dsl==0.2.0" "django-phonenumber-field==6.1.0" "django-redis-admin" "django-redis<6" "django-rest-hooks==1.5.0" "django-simple-history==3.0.0" "django-structlog>=2.1.0<2.2" "django-taggit-serializer==0.1.7" "django-taggit==3.0.0" "djangorestframework==3.13.1" "drf-api-tracking" "drf-spectacular==0.22.1" "drf-yasg<=1.21" "dropbox==10.6.0" "environs~=9.5.0" "exif==1.3.5" "geopandas" "h5py" "httpx==0.23.0" "humanize>=3.14.0<4.0.0" "infi.clickhouse_orm==2.1.1" "jsonmerge~=1.8.0" "jsonschema==3.2.0" "matplotlib==3.5.1" "natsort==7.1.1" "numpy~=1.22.0" "pandas~=1.3.3" "phonenumbers==8.12.20" "pillow~=9.2.0" "pyjwt~=2.3.0" "pylas==0.4.3" "python_dateutil>=2.5.3" "pytz==2022.1" "requests" "sentry-sdk<1.6.0" "serverless-wsgi>=3.0.0" "structlog~=22.1.0" "urllib3>=1.25.3" "--lock" "lock.json" --output-file lambda-no-rlp.zip --manylinux manylinux2014 --complete-platform aws-38-cp.json
Traceback (most recent call last):
  File "/home/jsirois/pex.venv/bin/pex", line 8, in <module>
    sys.exit(main())
  File "/home/jsirois/pex.venv/lib/python3.8/site-packages/pex/bin/pex.py", line 762, in main
    do_main(
  File "/home/jsirois/pex.venv/lib/python3.8/site-packages/pex/bin/pex.py", line 783, in do_main
    pex_builder = build_pex(
  File "/home/jsirois/pex.venv/lib/python3.8/site-packages/pex/bin/pex.py", line 629, in build_pex
    result = try_(
  File "/home/jsirois/pex.venv/lib/python3.8/site-packages/pex/result.py", line 84, in try_
    raise ResultError(error=result)
pex.result.ResultError: Failed to resolve compatible artifacts from lock lock.json for 1 target:
1. cp38-cp38-manylinux_2_26_x86_64:
    Failed to resolve all requirements for complete platform cp38-cp38-manylinux_2_26_x86_64 from lock.json:

Configured with:
    build: False
    use_wheel: True

Dependency on django-admin-row-actions not satisfied, 1 incompatible candidate found:
1.) django-admin-row-actions 0.0.5 (via: django-admin-row-actions==0.0.5) does not have any compatible artifacts:
    <https://files.pythonhosted.org/packages/b9/23/bdc09b4ac29ad72e3694886ebfc5720083388cb6669d39dcff00ebe889f2/django-admin-row-actions-0.0.5.tar.gz>

Dependency on django-json-widget not satisfied, 1 incompatible candidate found:
1.) django-json-widget 1.1.1 (via: django-json-widget==1.1.1) does not have any compatible artifacts:
    <https://files.pythonhosted.org/packages/c0/13/f256f113ab2a0828e0c5992d2f2b7c6fd377ee9e9918d6d7cb14a1b2e706/django-json-widget-1.1.1.tar.gz>

Dependency on django-opensearch-dsl not satisfied, 1 incompatible candidate found:
1.) django-opensearch-dsl 0.2 (via: django-opensearch-dsl==0.2.0) does not have any compatible artifacts:
    <https://files.pythonhosted.org/packages/cf/f3/1c95c57ceea474eba3c53847d50edfcee8d246bf9319935d8a60bca11076/django-opensearch-dsl-0.2.0.tar.gz>

Dependency on django-taggit-serializer not satisfied, 1 incompatible candidate found:
1.) django-taggit-serializer 0.1.7 (via: django-taggit-serializer==0.1.7) does not have any compatible artifacts:
    <https://files.pythonhosted.org/packages/68/c4/9737e1df18ca8af0f40f46a54af08a15d75eb7afd7805ba94c87fc042ae3/django-taggit-serializer-0.1.7.tar.gz>

Dependency on jsonmerge not satisfied, 1 incompatible candidate found:
1.) jsonmerge 1.8 (via: jsonmerge~=1.8.0) does not have any compatible artifacts:
    <https://files.pythonhosted.org/packages/37/1b/05dd6fd53a76a73e898f3d2b60d02ee90af9e11de2f8e3802a6262fa1ca9/jsonmerge-1.8.0.tar.gz>

Dependency on pylas not satisfied, 1 incompatible candidate found:
1.) pylas 0.4.3 (via: pylas==0.4.3) does not have any compatible artifacts:
    <https://files.pythonhosted.org/packages/df/91/22702de36464dac4dac89e876f59b8d5ef3f6fdaeccbf24dc68aa77f4d9b/pylas-0.4.3.tar.gz>

Dependency on future not satisfied, 1 incompatible candidate found:
1.) future 0.18.2 (via: django-json-widget==1.1.1 -> future) does not have any compatible artifacts:
    <https://files.pythonhosted.org/packages/45/0b/38b06fd9b92dc2b68d58b75f900e97884c45bedd2ff83203d933cf5851c9/future-0.18.2.tar.gz>

Dependency on coreschema not satisfied, 1 incompatible candidate found:
1.) coreschema 0.0.4 does not have any compatible artifacts:
    <https://files.pythonhosted.org/packages/93/08/1d105a70104e078718421e6c555b8b293259e7fc92f7e9a04869947f198f/coreschema-0.0.4.tar.gz>
    requirers:
    via: drf-yasg<=1.21 -> coreschema>=0.0.4
    via: drf-yasg<=1.21 -> coreapi>=2.3.3 -> coreschema
(pex.venv) jsirois@Gill-Windows:~$
It may not look like it, but that's what you want! The complete platform description is being used to build for the foreign AWS platform description and the PEX build is failing because only wheels are accepted for foreign platform PEX builds (PEX isn't smart enough to do cross-builds) and 8 of your dependencies are only available in sdist form.
So, to fix that you currently need to pre-build those wheels (roughly run 
pip wheel django-admin-row-actions==0.0.5 django-json-widget==1.1.1 django-opensearch-dsl==0.2.0 django-taggit-serializer==0.1.7 jsonmerge==1.8.0 pylas==0.4.3 future==0.18.2 coreschema==0.0.4 --no-deps --wheel-dir wheels/
on the target AWS image) and then serve those wheels up in a flat directory over http or via a netwrok share and the use 
[python-repos] repos
over in Pants to point at the extra source of pre-built wheels: https://www.pantsbuild.org/docs/reference-python-repos#section-repos
And, the Pex issue tracking this is here: https://github.com/pantsbuild/pex/issues/1899 With that fixed, even though Pants passes 
--resolve-local-platforms
, the local CPython 3.8 in WSL's Ubuntu would no-longer resolve as a substitute for the AWS CPython 3.8 described by your complete platform json and you'd hit the error above which is what you want. In the meantime, if you want to work around, you'll need to eliminate the WSL CPython 3.8 from your PATH when building the lambda.
i
Hi John. Thanks for your help. Is the workaround for getting the incompatibility error to pop up?
e
Yes, exactly. You'll then need to fix that problem by pre-building wheels as described.
i
Got it. We went ahead and added uploaded the whls so they are publicly accessible, and added a new repo to 
[python-repos] repos
. Will pants first try to pull from our custom repo before going to pypi?
e
It's undefined. Pants uses Pex which uses Pip and Pip has historically changed preference order without warning.
i
Ah. So will adding the repo will only work if we have the correct complete platforms? Otherwise it might pull for pypi?
Hey John. Sorry to bother you again. But we are still having the same issue. Do we need to create an updated json to pass into 
complete_platforms
? Or is it enought to just add the new repo with the generated whls?
e
No, a complete platform is fully determined by an interpreter and an interpreter alone. If the targeted foreign interpreter has not changed then neither has its complete platforms. You need two things to get this working: 1. On the Linux x86_64 machine where you run `./pants package ...
for the lambda, there must be no Python 3.8 interpreter visible (on the PATH) to Pants. That or run the
./pants package ...`command on a Mac. This step works around a bug in Pex filed above and can go away once that bug is fixed and Pants incorporates it. 2. A `[python-repos] repos = ...
Copy code
that points to a find-links repo containing all your pre-built wheels.
With all this set up, you can re-generate your lockfiles with
./pants generate-lockfiles`and then proceed.
When you re-generate the lock files you should see new wheel entries picked up from your newly added find-links repo.
Pants should recognize you modified 
[python-repos] repos
since the last time you generated lock files and force you to re-generate them, but Pants 2.12.x has a bug and does not do this, so you need to remember to re-gen on your own. This is fixed in Pants 2.14.0a0 by https://github.com/pantsbuild/pants/pull/16525
i
Thanks John. So we can remove complete platforms in favor of runtime?
e
No. 
runtime
is strictly worse - it underspecifies the foreign platform and thus allows in bad ABI wheels.
In short, runtime is "covenient" but almost always a lie and very often wrong. Using complete platforms is strictly correct always.
What it boils down to is that resolving a wheel in Python - whether via Pip, Poetry, or any tool, requires 3 things: 1. The full list of tags supported by the interpreter you want to install the wheels to work with. That's O(100) tags and allows selection of binary compatible wheels: https://peps.python.org/pep-0425/ 2. The complete marker environment for the interpreter you want to install the wheels to work with: https://peps.python.org/pep-0508/#environment-markers - This allows handling requirements with 
;
in them like 
pywinpty; sys_platform == "win32"
3. The Python interpreter full version for selecting applicability based on `Python-Requires`metadata (i.e.: `>=3.7`for a project that only works with Python 3.7+ A complete platform gives you all that data. A runtime / abbreviated platform gives you almost none of that data - really just #3
i
Hi John. Sorry for the questions. What is the reasoning behind running 
pants package
, and then generate the lockfile? How does packaging impact lockfile generation?
e
Its the opposite. You must 1st re-generate the lockfile, and only then can you run any other Pants command that needs to resolve distributions. The 
./pants package
goal is just one of those.
To be clear - re-generating the lock file is a 1 time step. Its required once whenever you update requirements or update 
[python-repos]
- but that's the only time it's required. Once you've done it, you can check in the updated lockfile and then proceed to run other Pants goals like 
./pants test
, 
./pants package
, etc.
Normally, if there is a change that affects the current lockfile such that it need to be re-generated, when you run ~any Pants goal, it will fail and tell you to 
./pants generate-lockfiles
. In this case though you hit another bug where updating `[python-repos] repos`slips by Pants defenses and you are not warned.
Does that make sense?
i
the lockfile generation part makes sense. The part that does not make sense is the PATH manipulation part. Why does generating on the lockfile on a mac work, but on WSL not work? Also when we generate the 
complete_platform
we are running the command inside of a lambda docker container, so how does it know we are in WSL? Also by removing python3.8 from our path, what is pants going to use instead?
e
So, on WSL you have a CPython 3.8 just like the lambda runtime does. However, unlike the lambda runtime which uses glibc 2.26 , the WSL uses glibc 2.28. This means any wheels containing the tag fragments 
__2__28_
or _`_2_27_`_ will work on WSL but will not work in the lambda runtime. The lambda runtime can only work with wheels built for 
_2_26_
or lower. Now, Pants passes Pex a flag `--resolve-local-platforms`which tells Pex: "please try and find a local interpreter that matches the --platform and --complete-platform values I've passed you if possible and use that instead if successful". That flag is there to allow building sdists into wheels when possible, but it is predicated on finding a local interpreter that exactly matches the foreign platform. On a Mac you will never find such an interpreter since the foreign platform is Linux. Under WSL Pex currently has a bug (https://github.com/pantsbuild/pex/issues/1899) that decides the foreign `_2_26_`platform is satisifed by the local 
_2_28_
interpreter. As such, you have to hide that interpreter from Pants to work around that bug for now.
This is definitely complicated, but I want to make sure you understand what's going on.
i
ah. I got it now. So would an easier solution be to generate the lockfile in a linux docker container? (edit I assume it all has to do with the version of glibc version on the docker container? Maybe I can downgrade glibc)
update: we opted to generate the lockfile in a docker container with a lambda base image so the environment matches perfectly.
e
Sounds good.
In that case though the old lockfile would have worked.
i
we are in the process of generating it now for the first time in this lambda docker image. Will keep you updated on how it goes. thanks.
hmm...looks like generating the lockfile inside a lambda docker container with glibc 2.26 didn't make any difference to the lockfile vs generating in a glibc 2.28 environment
update: looks like the only thing that changed in the updated lockfile was the platform tag to 
manylinux_2_26_x86_64
so now we get the incompatibility error!
now it looks like the repo we added with our generated whls is not being used
e
Exactly.
i
We are now hardcoding the url to our generated whls since adding the repo location to 
python-repos
isn't working
e
When 
--resolve-local-platforms
correctly picks a compatible CPython 3.8 
2_26
interpreter in the image, it can use the old lockfile and build sdists into wheels. The wheels you prebuilt and made available via the find links repo are not needed.
We are now hardcoding the url to our generated whls since adding the repo location to 
python-repos
isn't working
That does not make sense. I want to put on the brakes if possible.
You no longer need 
[python-repos] repos
at all.
You should not need to hand edit any URLs.
i
aw man. there is now a team of 4 involved in this endeavor. we wait on your every word. 😂
we will remove the urls
But we are confused why we are getting the incompatibility errors.
e
You'll need to fill me in on both the complete set of changes you're now poerating under and the error you are getting.
If you are running 
./pants package
in a docker container that matches the AWS lambda runtime, you should need no changes to your Pants configuration (re-wind a ~day) and the resulting lambda should just work.
Everything I coached you through was to get Pants running outside of any specialized container and still producing a PEX (lambdex) that would work out on AWS.
i
ah. got it. makes sense. We are trying to streamline everything as much as possible, so I should have clarified that we have no issues making a specialized docker container 😅
e
Ok. Well the whole thing started off with a different person altogether - @plain-summer-72727 - unless you are both of those accounts.
i
We have been sitting next to eachother this whole time
p
Yep, I'm here! Hahah
e
Aha, OK.
i
We are removing the URLs, and 
repo
e
The moral of the story is - when you can - always develop on == production. Using Mac or a different Linux etc. has a very high cost.
We try to support using not production, since everyone seems to love to do this for reasons I do not understand - but its tricky.
I mean, I'm not advocating developing in production, just on the production OS.
i
ya. for sure. We get that. deff best practice
It's just aws lambda container needs a bit of TLC so it's ready for development
Still a bit odd how we can 
package
locally on a 
2.28
machine after generating the lockfile on a 
2.26
. Maybe our local machines also have 
2.26
? Either way, we are understanding this better I think
Deff going to do development on something closer to production
e
Lockfiles are universal. They lock all artifacts available for the solved versions.
The only issue on the 
2.28
machine was that it picked the `2.28`wheel, it should have picked the `2.17`wheel.
That latter wheel is compatible with glibc 
2.26
You may be quite frustrated with Pants & Pex after all this, but I encourage you to express some of your pain by taking the survey linked in the banner on PyPI while the pain is still fresh: https://pypi.org/ I did this this morning myself.
Those folks are working hard on making things better and you can help steer them in the right direction.
i
Got it! We will for sure take the survey. Python needs something like this so we are happy to contribute
Open in Slack
PreviousNext
Powered by