https://pantsbuild.org/ logo
#general
Title
# general
n

nice-florist-55958

09/21/2022, 1:22 PM
Not sure what changed on our end yet, but up to as recently as yesterday morning generate-lockfiles worked just fine, but now no matter what I do I get a KeyError for any input requirement that is pinned in some way, e.g., ipywidgets<8.0 — stacktrace below. I've tried nuking the cache, switching between 13 and 14 pants versions, and no luck. We can still build pex_binarys from the current lock file. Just wondering if anything immediately comes to mind that would cause this behavior.
pex: Resolving interpreters :: Indexing downloads
Traceback (most recent call last):
File "/d/d1/user/taymarti/.cache/pants/named_caches/pex_root/installed_wheels/dd24a2333561b849fbebe03fe7319585ddb9a8ad512920b6bb2d9249a810cb71/pex-2.1.104-py2.py3-none-any.whl/pex/result.py", line 105, in catch
return func(*args, **kwargs)
File "/d/d1/user/taymarti/.cache/pants/named_caches/pex_root/installed_wheels/dd24a2333561b849fbebe03fe7319585ddb9a8ad512920b6bb2d9249a810cb71/pex-2.1.104-py2.py3-none-any.whl/pex/cli/command.py", line 84, in run
return subcommand_func(self)
File "/d/d1/user/taymarti/.cache/pants/named_caches/pex_root/installed_wheels/dd24a2333561b849fbebe03fe7319585ddb9a8ad512920b6bb2d9249a810cb71/pex-2.1.104-py2.py3-none-any.whl/pex/cli/commands/lock.py", line 432, in _create
create(
File "/d/d1/user/taymarti/.cache/pants/named_caches/pex_root/installed_wheels/dd24a2333561b849fbebe03fe7319585ddb9a8ad512920b6bb2d9249a810cb71/pex-2.1.104-py2.py3-none-any.whl/pex/resolve/lockfile/create.py", line 311, in create
create_lock_download_manager = CreateLockDownloadManager.create(
File "/d/d1/user/taymarti/.cache/pants/named_caches/pex_root/installed_wheels/dd24a2333561b849fbebe03fe7319585ddb9a8ad512920b6bb2d9249a810cb71/pex-2.1.104-py2.py3-none-any.whl/pex/resolve/lockfile/create.py", line 83, in create
artifact = file_artifacts_by_filename.get(f) or source_artifacts_by_pin[pin]
KeyError: Pin(project_name=ProjectName(raw='ipywidgets', normalized='ipywidgets'), version=Version(raw='7.7.2', normalized='7.7.2'))
Pin(project_name=ProjectName(raw='ipywidgets', normalized='ipywidgets'), version=Version(raw='7.7.2', normalized='7.7.2'))
e

enough-analyst-54434

09/21/2022, 2:16 PM
@nice-florist-55958 what version of Pants are you using? 2.13.0 Uses Pex 2.1.102 and 2.14.x and main (2.15.x) use 2.1.103. You appear to be using Pex 2.1.104 which would require a manual override.
n

nice-florist-55958

09/21/2022, 2:36 PM
Yes 104. I can't remember the purpose of the manual override anymore, but presumably to overcome an issue in one of our prior threads here that had been fixed in a later pex release.
e

enough-analyst-54434

09/21/2022, 2:40 PM
Ok. As always, if you can re-run with
-ldebug
and hand over the failing
pex
command line noting any redactions if redactions are needed, that would be useful to establish a repro.
I am very happy to get testing on 104, but the only reason to use 104 is the new `--pip-version 22.2.2`which you can't do as a Pants user. Pants will need to plumb that option. 104 is very new (3 weeks old) so any recollection you can dredge up why the switch would be useful. Perhaps better would just be what version you switched from.
I'm going to be OOO ~presently through the 28th, so you may need to ping someone else for help with this if you can't resolve the issue.
n

nice-florist-55958

09/24/2022, 5:26 PM
@enough-analyst-54434 no worries, let’s pick up when you are back. I remember the reason is that we were on 2.13/99 and you fixed some issues related to os environment marker tags (JupyterLab expressing extra ‘nt’ requirements if you recall) — 104 just happened to be the latest available release on GitHub at the time we made the change.
So I’ve narrowed the issue to matplotlib. It’s unconstrained in our input and the recent version 3.6.0 fails to install (when done directly with pip install) because it complains it could not find setuptools_scm >= 7.0 on our repo (it’s there). After installing that directly, the next error was contourpy>=1.05 not found, which was true. After adding it to our repo, I could install matplotlib==3.6.0 and PEX could lock it. I think this is just a replay of a similar class of issues where PEX delegates to PIP but does not relay meaningful information back to the user when the lockgen fails outright (the case here) or hangs indefinitely (as was the case for the jupyterlab issue). I know from experience in most cases the reported error is meaningful, so I don’t mean to imply things are totally broken or anything. Besides improving error reporting when things go wrong, these types of issues would be prevented by implementing the incremental resolve feature others have mentioned (I.e., implicitly pin all unchanged requirements based on the current resolve and only let the new requirements’ versions be free variables). Of course, when you do want to finally regen the resolve without this feature on, having detailed error reporting will be helpful, especially when you have lots of requirements that are only loosely pinned thereby creating multiple opportunities for the lockgen process to fail. (Summary of the behavior from client’s pov, which isn’t helpful for debugging and makes on suspect unrelated issues. Reported error is a KeyError of a Pin in the following order: pinned requirements in alphabetical order (and then if those are removed so everything is unconstrained) followed by unpinned requirements in alphabetical order.)
Is it possible to annotate an input requirement such that the —only-binary=<requirement> is ultimately passed to pip? That could be helpful for very complex libs that private repos may not have wheels for yet.
e

enough-analyst-54434

10/03/2022, 7:36 PM
Sorry for the delay getting back to you here. 1st the easy part
--only-binary
is supported. Use this: https://www.pantsbuild.org/docs/reference-python#only_binary
In terms of visibility into lock creation - agreed, That's a problem that needs to be fixed. Here is a concrete example: https://github.com/pantsbuild/pex/issues/1927. I think Pex will simply always have to parse the Pip debug log and use heuristics to try and present some actionable information. The `--preserve-pip-download-log`option exists today, but Pants does not use it and most users would not really want to read the full Pip verbose log to try to figure out what's going wrong.
The incremental resolve feature has indeed been coming up more and more. Again here: https://github.com/pantsbuild/pants/issues/15607#issuecomment-1265840306 I'm currently working on adding Windows support to Pex and then to Pants (real support, not WSL which already works). Next step though may be to focus on Pex lock error improvement and incremental resolve support.
7 Views