https://pantsbuild.org/ logo
Join Slack
Powered by
Hopefully simple and a quick check: trying to buil...
# general
e
Hopefully simple and a quick check: trying to build/deploy/test the awslambda example from https://github.com/pantsbuild/example-python in prep for starting to use it in our product (hopefully). But while building and deploying (via Terraform) seemed to work ok, when executing it I got warnings about "no importlib-metadata distribution" and after "fixing" that, "no typing-extensions distribution"... etc. My "fix" was to add 
importlib-metadata==3.10.0
and 
typing-extensions==3.7.4.3
to the 
requirements.txt
file and manually add 
dependencies=["//:importlib-metadata", "//:typing-extensions"]
to the 
python_awslambda
target. That works, and it executes... but was that correct and/or is there a better way to resolve that?
h
Hello! You're right that you probably do need to add both of those libraries to your requirements.txt if you did not have them. Ideally though, Pants would have inferred the dependencies. What happens if you remove the explicit dependencies field and keep the requirements.txt change? You can run ./pants dependencies --transitive path/to/lambda.py
h
Nothing in that repo imports 
importlib_metadata
, but 
importlib-metadata
is in the 
constraints.txt
in that example repo, so something transitively needs it.
But it should be brought in by pip, so I'm not sure what's up with that
And I certainly can't see what needs 
typing-extensions
Can you post stacktraces to each of those errors? Feel free to redact as needed
e
Hm; it did not infer the dependencies. I did have to take out the 
interpreter_constraints
line in 
pants.toml
(it was failing trying to get pyyaml on that version and disabling constraints but using 
requirements.txt
allowed it to build; I probably should have tried updating the constraints but another time) In other words, with those two in requirements.txt and with 
dependencies=["//:importlib-metadata", "//:typing-extensions"],
in the BUILD for lambda, I get
Copy code
...
//:importlib-metadata
...
//:typing-extensions
...but if I take out the 
dependencies=
line, those two are indeed missing. Let me take it out, redeploy, and look at the error traces
With both dependencies missing:
Copy code
START RequestId: e4b8b696-0207-4950-93ab-5779fc87d236 Version: $LATEST
[ERROR] ResolveError: Failed to resolve requirements from PEX environment @ /var/task.
Needed manylinux_2_17_x86_64-cp-37-cp37m compatible dependencies for:
 1: importlib-metadata>=0.12; python_version < "3.8"
    Required by:
      virtualenv 20.4.3
      pluggy 0.13.1
      tox 3.23.0
    But this pex had no 'importlib-metadata' distributions.
 2: importlib-metadata; python_version < "3.8"
    Required by:
      pre-commit 2.12.0
    But this pex had no 'importlib-metadata' distributions.
Traceback (most recent call last):
  File "/var/lang/lib/python3.7/imp.py", line 234, in load_module
    return load_source(name, filename, file)
  File "/var/lang/lib/python3.7/imp.py", line 171, in load_source
    module = _load(spec)
  File "<frozen importlib._bootstrap>", line 696, in _load
  File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 728, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/var/task/lambdex_handler.py", line 30, in <module>
    bootstrap_pex_env(__entry_point__)
  File "/var/task/.bootstrap/pex/pex_bootstrapper.py", line 495, in bootstrap_pex_env
    PEXEnvironment(entry_point, pex_info).activate()
  File "/var/task/.bootstrap/pex/environment.py", line 366, in activate
    self._activated_dists = self._activate()
  File "/var/task/.bootstrap/pex/environment.py", line 696, in _activate
    resolved = self.resolve()
  File "/var/task/.bootstrap/pex/environment.py", line 522, in resolve
    self._resolved_dists = self.resolve_dists(all_reqs)
  File "/var/task/.bootstrap/pex/environment.py", line 607, in resolve_dists
    "{items}".format(pex=self._pex, platform=self._platform, items="\n".join(items))
END RequestId: e4b8b696-0207-4950-93ab-5779fc87d236
REPORT RequestId: e4b8b696-0207-4950-93ab-5779fc87d236	Duration: 9036.00 ms	Billed Duration: 9036 ms	Memory Size: 128 MB	Max Memory Used: 33 MB	
Unknown application error occurred
With `dependencies=["//:importlib-metadata"]`:
Copy code
START RequestId: dfda5498-7991-4232-8099-45c601d9b712 Version: $LATEST
[ERROR] ResolveError: Failed to resolve requirements from PEX environment @ /var/task.
Needed manylinux_2_17_x86_64-cp-37-cp37m compatible dependencies for:
 1: typing-extensions>=3.6.4; python_version < "3.8"
    Required by:
      importlib-metadata 3.10.0
    But this pex had no 'typing-extensions' distributions.
Traceback (most recent call last):
  File "/var/lang/lib/python3.7/imp.py", line 234, in load_module
    return load_source(name, filename, file)
  File "/var/lang/lib/python3.7/imp.py", line 171, in load_source
    module = _load(spec)
  File "<frozen importlib._bootstrap>", line 696, in _load
  File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 728, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/var/task/lambdex_handler.py", line 30, in <module>
    bootstrap_pex_env(__entry_point__)
  File "/var/task/.bootstrap/pex/pex_bootstrapper.py", line 495, in bootstrap_pex_env
    PEXEnvironment(entry_point, pex_info).activate()
  File "/var/task/.bootstrap/pex/environment.py", line 366, in activate
    self._activated_dists = self._activate()
  File "/var/task/.bootstrap/pex/environment.py", line 696, in _activate
    resolved = self.resolve()
  File "/var/task/.bootstrap/pex/environment.py", line 522, in resolve
    self._resolved_dists = self.resolve_dists(all_reqs)
  File "/var/task/.bootstrap/pex/environment.py", line 607, in resolve_dists
    "{items}".format(pex=self._pex, platform=self._platform, items="\n".join(items))
END RequestId: dfda5498-7991-4232-8099-45c601d9b712
REPORT RequestId: dfda5498-7991-4232-8099-45c601d9b712	Duration: 9917.03 ms	Billed Duration: 9918 ms	Memory Size: 128 MB	Max Memory Used: 34 MB	
Unknown application error occurred
So it looks like importlib-metadata is at fault for wanting typing-extensions. Could be much of this was caused by not using the existing constraints, too.
h
Possibly, although some version of importlib-metadata (and its deps, including typing-extensions) should have been resolved just by normal pip behavior.
Are you literally building just the example-python example as-is, or had you modified it in some way? If not then it sounds like that repo's constraints.txt is broken
👍 1
e
Sorry for the delay. Literally was building it as-is (at least that was the intent; I did not modify anything that I'm aware of except the 
pants.toml
file to remove the constraints when the resolution of pyyaml failed. So if I enable the constraints.txt again and take out my modifications to requirements.txt and BUILD (so I think this is where I started), 
./pants package helloworld:awslambda
got me
Copy code
...
stderr:
ERROR: Could not find a version that satisfies the requirement pyyaml>=5.1 (from pre-commit)
ERROR: No matching distribution found for pyyaml>=5.1
...
...which surprised me as well. So I'm not sure if constraints.txt is broken or... well, not sure really. But my woes started with the pyyaml error above which is why I disabled constraints, which then ended with the importlib-metadata and typing-extensions issues.
h
No matching distribution found for pyyaml>=5.1
is pretty surprising, given https://pypi.org/project/PyYAML/
Are you using a private repository instead of public pypi or something?
e
Awslambda resolves for Linux using --platform iirc? And so if you're on mac and do not have a pre-built Linux wheel you'd get something like this.
👍 1
It looks though like the prebuilts are there on PyPI so probably a dead end.
h
I reproduce the "Could not find a version that satisfies the requirement pyyaml>=5.1" error
so will start from that
Weird, if I run standalone 
pex --output-file=foo.pex protobuf>=3.11.3 setuptools>=42.0.0 translate>=3.2.1
it works, and I get pyyaml in 
.deps/PyYAML-5.4.1-cp36-cp36m-macosx_10_9_x86_64.whl/
But if I run 
__run.sh
in the Pants process execution sandbox it fails with
Copy code
ERROR: Could not find a version that satisfies the requirement pyyaml>=5.1 (from pre-commit)
ERROR: No matching distribution found for pyyaml>=5.1
And specifically if I turn on verbose logging, I see
Copy code
Skipping link: none of the wheel's tags match: cp36-cp36m-macosx_10_9_x86_64: <https://files.pythonhosted.org/packages/ef/e9/d62912119552b157ed66dc8297ae6ac08629d7d5c497d4faa26b0c3a4efe/PyYAML-5.4.1-cp36-cp36m-macosx_10_9_x86_64.whl#sha256=6c78645d400265a062508ae399b60b8c167bf003db364ecb26dcab2bda048253> (from <https://pypi.org/simple/pyyaml/>) (requires-python:>=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*, !=3.5.*)
Even though that tag should match
And the same thing happens if I build with python 3.7 or 3.8
Oh wait...
Silly me, forgot to add 
--platform
in my local runs
But with that it still works, on macos, with pyyaml in 
.deps/PyYAML-5.4.1-cp37-cp37m-manylinux1_x86_64.whl/
So the prebuilts on pypi should work
And in the failed 
__run.sh
run it does find the prebuilt with matching tags:
Copy code
Found link <https://files.pythonhosted.org/packages/7a/a5/393c087efdc78091afa2af9f1378762f9821c9c1d7a22c5753fb5ac5f97a/PyYAML-5.4.1-cp37-cp37m-manylinux1_x86_64.whl#sha256=e1d4970ea66be07ae37a3c2e48b5ec63f7ba6804bdddfdbd3cfd954d25a82e63> (from <https://pypi.org/simple/pyyaml/>) (requires-python:>=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*, !=3.5.*), version: 5.4.1
Ok, this is because of the bad 
constraints.txt
file
👀 1
It pins to 5.3.1
e
Aha. A requirements <-> constraints self conflict / forgotten constraints.txt re-gen?
If so, I still haven't found time to think about / work on https://github.com/pantsbuild/pex/issues/1200 but if anyone does have time, obviously yay.
h
Well it might be
I fixed up the constraints file and then hit an issue with the fact that we constrain to an old narrow range of setuptools versions in the 
extra_requirements
of lambdex
And we currently resolve tools against the repo's constraints file, which doesn't really make sense.
👍 1
@echoing-farmer-15630 Gotta stop for the night, but I see how to fix this, hopefully tomorrow I can get the example-python repo stable so you can crib off it for your own lambdas. What Python version are you targeting in your own code?
But anyway, this is our breakage, sorry it's causing you trouble, will fix ASAP.
e
No trouble; I learned what I needed to and got a good first-approximation lambda running for future production, so I'm greatly appreciative of the example in any case (probably learned a little getting it working)--just wanted to bring it up for the next person who may hit it. Nice work!
❤️ 1
h
Update: haven't forgotten about this, just haven't gotten around to it yet...
OK confirmed that example-python seems to work fine on Ubuntu
but on MacOS I see many issues
All relating to the fact that it's not really possible to build lambdas on MacOS
I mean, it could hypothetically be done if all your dependencies have prebuild linux wheels, but that maybe be unlikely in practice, and in any case there are complexities with things like interpreter selection that make it fairly impractical.
What we definitely should do is show a sensible error message in this case
5 Views
Open in Slack
PreviousNext
Powered by