https://pantsbuild.org/ logo
Join Slack
Powered by
Is is possible to use pip cli options with Pants? ...
# general
r
Is is possible to use pip cli options with Pants? Is there any possibility to install 
aioeventlet
pip package with Pants? More details in the 🧵.
In our project 
aioeventlet
package is used. Providing this package as dependency with
Copy code
python_requirement(
    name="aioeventlet",
    requirements=["aioeventlet"],
)
will fail due to inconsistent versions specified. Using pip in a virtualenv directly, the error can be reproduced with 
pip install aioeventlet
. However, a workaround for direct pip usage is 
pip install --use-deprecated=legacy-resolver aioeventlet
. Is there any possibility to give command line options to pip with Pants? Is there any possibility to install 
aioeventlet
pip package with Pants?
h
Hey Fritz, welcome! Pants uses Pex for dependency resolution, which itself vendors pips. Certain options can be passed to pip via Pants options like 
[python-repos].indexes
, but there is no generic way to pass arbitrary options In this particular case, have you read about lockfiles? I believe that should work around your issue - if you tell Pants/Pex/Pip to use a lockfile that correctly resolved 
aioeventlet
, then you can force it to use what you want rather than trying to resolve from scratch. https://www.pantsbuild.org/docs/python-third-party-dependencies#user-lockfile
r
Hi @hundreds-father-404, thanks for your quick reply and suggestion regarding lockfiles! I did not employ them for my first evaluation but will try them next. 🙂
🤞 1
👍 1
e
Pants used to plumb Pex's 
--resolver-version {pip-legacy-resolver,pip-2020-resolver}
which corresponds to the Pip option you mention. That was deliberately removed though in Pants 2.5 here: https://github.com/pantsbuild/pants/pull/11819 If the only way to get aioeventlet resolved is with the Pip legacy resolver ... we'll have to think harder.
Yeah, this works fine (defaults to 
--resolver-version=pip-legacy-resolver
): 
$ pex aioeventlet -oaioeventlet.pex
and this fails:
Copy code
$ pex --resolver-version=pip-2020-resolver aioeventlet -oaioeventlet.pex
WARNING: Discarding <https://files.pythonhosted.org/packages/ee/1a/1573ef35a49bfe0144a98a9bdd40ac5a692e9a117ac8735e682e03dc39ce/aioeventlet-0.5.2.tar.gz#sha256=cecb51ea220209e33b53cfb95124d90e4fcbee3ff8ba8a179a57120b8624b16a> (from <https://pypi.org/simple/aioeventlet/>). Requested aioeventlet from <https://files.pythonhosted.org/packages/ee/1a/1573ef35a49bfe0144a98a9bdd40ac5a692e9a117ac8735e682e03dc39ce/aioeventlet-0.5.2.tar.gz#sha256=cecb51ea220209e33b53cfb95124d90e4fcbee3ff8ba8a179a57120b8624b16a> has different version in metadata: '0.5.1'
ERROR: Could not find a version that satisfies the requirement aioeventlet
ERROR: No matching distribution found for aioeventlet
pid 8358 -> /home/jsirois/.pex/venvs/a9ec88dcb420b50984f68e2ee107104cfd87c83b/9148e449944f6676b304f8f06462e05782fdb928/pex --disable-pip-version-check --no-python-version-warning --exists-action a --isolated -q --cache-dir /home/jsirois/.pex --log /tmp/tmploxsa78u/pip.log download --dest /tmp/tmpgpxxhaks/home.jsirois..venv.pex.bin.python3 aioeventlet --index-url <https://pypi.org/simple> --retries 5 --timeout 15 exited with 1 and STDERR:
None
That failure in particular (published distribution is aioeventlet-0.5.2.tar.gz but metadata inside claims 0.5.1 (when you crack open the tarball you'll find:
Copy code
$ grep -E "^Version:" PKG-INFO 
Version: 0.5.1
) combined with the fact that there is exactly 1 dist ever published for this back in 2017 with a non-functioning readthedocs and a dead bitbucket repo all say steer clear of this library. I'm assuming though you have ~no choice. If so, your best bet is to simply download the tarball, extract it and re-create the sdist. Or, simpler, just build a wheel:
Copy code
$ python3.10 -mvenv fix-aioeventlet
$ fix-aioeventlet/bin/pip install wheel setuptools
$ fix-aioeventlet/bin/pip wheel --wheel-dir fixed/ --use-deprecated=legacy-resolver --no-deps aioeventlet
That will give you:
Copy code
$ ls fixed/
aioeventlet-0.5.1-py3-none-any.whl
You can then point Pex (or Pants or Pip) at that local directory and have success with the pip-2020-resolver:
Copy code
$ pex -f fixed/ aioeventlet==0.5.1 -o aioeventlet.pex
The key will be placing the 
fixed
directory in a shared location or on a shared web-server with an index page and pointing Pants to it with 
[python-repos] repos
https://www.pantsbuild.org/docs/reference-python-repos#section-repos
r
Thank you very much for your detailed insights, @enough-analyst-54434! What I understand is, that you said, that the suggestion using lockfiles will fail and I must use the more elaborate solution you described. Is that correct? Or was it more along the lines of "if the lock-file suggestion doesn't work, you need to..."
I also noticed, that 
aioeventlet
is an rather old, outdated and unfavourable package, you would not want to use or have to manage at all (with all the symptoms you described). But as you very well assumed, there are limitations to what is in my power, as I am "only the infrastructure guy" designated to evaluate an possibly migrate an existing code base to a new build system.
Thanks again @enough-analyst-54434! I just tried your suggestion and had success. For my first tests I did not upload to a remote server though, but used the PEP 440 direct reference
Copy code
"aioeventlet @ file:///absolute/path/to/my/folder/fixed/aioeventlet-0.5.1-py3-none-any.whl",
within the requirements block.
I know, of course, that this probably needs to be elaborated on in the regard you described, using a remote server for download, when using in production.
e
I think the lock file suggestion was never relevant to this particular case. The 
aioeventlet
distribution has this unexpected property of misrepresenting its version. Combined with there being exactly one published version ever, a lockfile (presumably to pin to some known good version) is of no use.
👍 1
🙏 1
11 Views
Open in Slack
PreviousNext
Powered by