Hi, I am trying to use different versions of the s...
# general
w
Hi, I am trying to use different versions of the same python module for different interpreters, as described in https://www.pantsbuild.org/docs/reference-python_requirement#coderequirementscode :
You can specify multiple requirements for the same project in order to use environment markers, such as
["foo>=1.2,<1.3 ; python_version>'3.6'", "foo==0.9 ; python_version<'3'"]
.
Practically my BUILD contains this instruction :
Copy code
python_requirement(
    name="streamlit",
    requirements=[
        "streamlit>1.12 ; python_version>'3.9'",
        "streamlit==1.12 ; python_version<='3.9'"
    ],
)
which gives me the following error :
Copy code
ERROR: Cannot install streamlit==1.12 and streamlit>1.12 because these package versions have conflicting dependencies.
how should I approach this specific problem ?
e
If you take everything you have written down there seriously, you'd want to look for where you're using >1 Python interpreter at once, since that's the only way to get a conflict. Detour to prove that follows: Pants uses Pex which uses Pip and this works just fine with 1 interpreter:
Copy code
$ PEX_TOOLS=1 ./example.pex info -i2 | grep streamlit
    "streamlit-1.12.0-py2.py3-none-any.whl": "6bf86911f48e207888e7dfdf7f51e3bcb21d4c32e9b05093c6800be11157dc39",
    "streamlit==1.12; python_version <= \"3.9\""
$ pex --python python3.9 "streamlit>1.12 ; python_version>'3.9'" "streamlit==1.12 ; python_version<='3.9'" -o example.pex --include-tools
$ PEX_TOOLS=1 ./example.pex info -i2 | grep streamlit
    "streamlit-1.12.0-py2.py3-none-any.whl": "6bf86911f48e207888e7dfdf7f51e3bcb21d4c32e9b05093c6800be11157dc39",
    "streamlit==1.12; python_version <= \"3.9\""
$ pex --python python3.10 "streamlit>1.12 ; python_version>'3.9'" "streamlit==1.12 ; python_version<='3.9'" -o example.pex --include-tools
$ PEX_TOOLS=1 ./example.pex info -i2 | grep streamlit
    "streamlit-1.24.0-py2.py3-none-any.whl": "2f03f90699d88d902911749ba793d7b52adde72ffb9f4a5cd3b204dc9eab5c9a",
    "streamlit>1.12; python_version > \"3.9\""
So - back to how >1 interpreter could be used: when creating a lock file. So - you left out the command you ran to get that error, but I'd guess it was
pants generate-lockfiles
and further guess your interpreter constraints allow for Python versions that straddle the divide. In that case the lock is impossible - locks can only lock 1 version for any given project (streamlit here). So, if I'm right here, in short, the advice you quote is stale in the world of lock file use. To support two different locks (aka resolves), you need two different resolves. For more on that, see here: https://www.pantsbuild.org/docs/python-lockfiles#multiple-lockfiles
The repro for my guess using the underlying tools is:
Copy code
$ pex3 lock create --interpreter-constraint ">=3.8,!=3.9.7,<=3.11" --style universal "streamlit>1.12 ; python_version>'3.9'" "streamlit==1.12 ; python_version<='3.9'" --pip-version latest
 --resolver-version pip-2020-resolver
pid 2194 -> /home/jsirois/.pex/venvs/29a63f13457fc3d6f8c222205ad9dbb084735c8c/5985ed09b49a653d6596b0e14d134c5456cf1a9f/bin/python -sE /home/jsirois/.pex/venvs/29a63f13457fc3d6f8c222205ad9dbb084735c8c/5985ed09b49a653d6596b0e14d134c5456cf1a9f/pex --disable-pip-version-check --no-python-version-warning --exists-action a --no-input --isolated -q --cache-dir /home/jsirois/.pex/pip/23.1.2/pip_cache --log /tmp/pex-pip-log.j5nmf16m/pip.log download --dest /tmp/tmpnfzbsexf/home.jsirois.bin.pex.venv.bin.python streamlit>1.12 ; python_version>'3.9' streamlit==1.12 ; python_version<='3.9' --index-url <https://pypi.org/simple> --retries 5 --timeout 15 exited with 1 and STDERR:
ERROR: Cannot install streamlit==1.12 and streamlit>1.12 because these package versions have conflicting dependencies.
ERROR: ResolutionImpossible: for help visit <https://pip.pypa.io/en/latest/topics/dependency-resolution/#dealing-with-dependency-conflicts>

 The conflict is caused by:
     The user requested streamlit>1.12
     The user requested streamlit==1.12

 To fix this you could try to:
 1. loosen the range of package versions you've specified
 2. remove package versions to allow pip attempt to solve the dependency conflict
w
thanks, your speculations are correct. Supposedly, using multiple lock files would force me to duplicate a lot of targets considering that transitive dependencies must use the same resolve.
e
I am not an expert in multiple lockfiles, but if you read there, there is a
parameterize(...)
gizmo to help with this and I think folks also use macros / (and
__defaults__
too?) to further ameliorate boilerplate. But, as I said, I really have no business talking about that feature - I don't know it well at all. And, yeah. if there is any way to not use multiple resolves, that is always better. It's just sometimes unavoidable.
w
using
parameterize
is a convenient way to solve this, thanks !