I am very happy to get testing on 104, but the only reason to use 104 is the new `--pip-version 22.2.2`which you can't do as a Pants user. Pants will need to plumb that option. 104 is very new (3 weeks old) so any recollection you can dredge up why the switch would be useful. Perhaps better would just be what version you switched from.

I'm going to be OOO ~presently through the 28th, so you may need to ping someone else for help with this if you can't resolve the issue.

So I’ve narrowed the issue to matplotlib. It’s unconstrained in our input and the recent version 3.6.0 fails to install (when done directly with pip install) because it complains it could not find setuptools_scm >= 7.0 on our repo (it’s there). After installing that directly, the next error was contourpy>=1.05 not found, which was true. After adding it to our repo, I could install matplotlib==3.6.0 and PEX could lock it. I think this is just a replay of a similar class of issues where PEX delegates to PIP but does not relay meaningful information back to the user when the lockgen fails outright (the case here) or hangs indefinitely (as was the case for the jupyterlab issue). I know from experience in most cases the reported error is meaningful, so I don’t mean to imply things are totally broken or anything. Besides improving error reporting when things go wrong, these types of issues would be prevented by implementing the incremental resolve feature others have mentioned (I.e., implicitly pin all unchanged requirements based on the current resolve and only let the new requirements’ versions be free variables). Of course, when you do want to finally regen the resolve without this feature on, having detailed error reporting will be helpful, especially when you have lots of requirements that are only loosely pinned thereby creating multiple opportunities for the lockgen process to fail. (Summary of the behavior from client’s pov, which isn’t helpful for debugging and makes on suspect unrelated issues. Reported error is a KeyError of a Pin in the following order: pinned requirements in alphabetical order (and then if those are removed so everything is unconstrained) followed by unpinned requirements in alphabetical order.)

Is it possible to annotate an input requirement such that the —only-binary=<requirement> is ultimately passed to pip? That could be helpful for very complex libs that private repos may not have wheels for yet.

In terms of visibility into lock creation - agreed, That's a problem that needs to be fixed. Here is a concrete example: https://github.com/pantsbuild/pex/issues/1927. I think Pex will simply always have to parse the Pip debug log and use heuristics to try and present some actionable information. The `--preserve-pip-download-log`option exists today, but Pants does not use it and most users would not really want to read the full Pip verbose log to try to figure out what's going wrong.

The incremental resolve feature has indeed been coming up more and more. Again here: https://github.com/pantsbuild/pants/issues/15607#issuecomment-1265840306 I'm currently working on adding Windows support to Pex and then to Pants (real support, not WSL which already works). Next step though may be to focus on Pex lock error improvement and incremental resolve support.