https://pantsbuild.org/ logo
l

lively-dusk-46231

10/23/2022, 4:47 AM
I found an issue with
bandit
when debugging some version issues with
pants
https://github.com/PyCQA/bandit/issues/960 Do you think it makes sense to change the default
[bandit].version
from
bandit<1.8,>=1.7.0
->
bandit<1.8,>=1.7.0,!=1.7.2
? If yes - I can create an issue in github
e

enough-analyst-54434

10/23/2022, 6:18 AM
That does not look like a bug to me:
bandit requires Python '>=3.7' but the running Python is 3.6.5
That's just bandit saying they dropped older Python support. Projects do this all the time. You might quibble that they should have only done this in a minor or major version bump and not in a patch version bump, but the ship has sailed.
If you look at latest bandit, this is still the case; so your proposed Pants fix won't work: https://pypi.org/project/bandit/1.7.4/#data
bandit>=1.7.2
will simply never work for Python 3.6.
l

lively-dusk-46231

10/23/2022, 7:01 AM
Hi John, I understand dropping support for older pythons. But that is not the case here When I read bandit==1.7.2 's pypi page. It does not have any "python requires" on the left side. So when I do
pip3.6 install bandit
it is fetching bandit 1.7.2 If bandit 1.7.2 does not support py3.6 It should have fetched bandit=1.7.1 I expect pants to take bandit 1.7.1 if I have interpreter constraints as py3.6 When I tried it without pants, I saw the issue with even pip - pip3.6 should have taken bandit 1.7.1 Similar issues don't occur with other libraries. If I do
pip3.6 install flask
it installs the last known version of flask that works with py3.6
e

enough-analyst-54434

10/23/2022, 12:22 PM
@lively-dusk-46231 I think the issue here is you have such an old version of Pip that it does not understand
Requires-Python
metadata. Here's where PBR did the right things in
setup.cfg
and
setup.py
to drop support: https://github.com/PyCQA/bandit/pull/777 And here's the correct result:
Copy code
unzip -qc bandit-1.7.2-py3-none-any.whl bandit-1.7.2.dist-info/METADATA | grep Requires-Python
Requires-Python: >=3.7
Aha, the issue is too new a Pip: https://github.com/pypa/pip/issues/11142
l

lively-dusk-46231

10/23/2022, 1:56 PM
Hmm - you're right. When using
pip
directly - I can fix the issue by using ~21 Any way for me to check the pip version that pants is using ? I think it is creating a pex file based on my interpreter constraints ?
e

enough-analyst-54434

10/23/2022, 1:57 PM
It uses a vendored version of 20.3.4.
l

lively-dusk-46231

10/23/2022, 1:59 PM
😕 I see - looks like 20.3.4 is known to have this bug: https://github.com/pypa/pip/issues/9925#issuecomment-830512306
actually - I'm not sure. Looks like this was just an optimization not a bug ... Hmm. I can't figure out what is going wrong
e

enough-analyst-54434

10/23/2022, 2:10 PM
The short story is this does not seem like a bandit bug.
👍 1
l

lively-dusk-46231

10/23/2022, 2:20 PM
Makes sense - not a bandit bug. Thanks for the to and fro to help me debug this🙂 But I feel like the pants or pex or pip behavior somewhere is not correct. Will redo my initial test and report the issue
This is reproducible:
Copy code
$ cat pants.toml
[GLOBAL]
pants_version = "2.13.0"
backend_packages = [
  "pants.backend.python",
  "pants.backend.python.lint.bandit",
]

[python]
interpreter_constraints = ['CPython>=3.6,<4']

[bandit]
lockfile = 'bandit.lock'

$ cat BUILD
python_sources(name="root")
Now when I do:
Copy code
./pants generate-lockfiles
It creates a lock file with
bandit==1.7.2
Now when I run:
Copy code
./pants lint ::
It raises:
Copy code
Dependency on bandit not satisfied, 1 incompatible candidate found:
1.) bandit 1.7.2 (via: bandit<1.8,>=1.7.0) requires Python >=3.7
This is not the behavior I would have expected. Shouldn't "something" have chosen bandit==1.7.1 in the lockfile to satisfy my interpreter constraints ?
e

enough-analyst-54434

10/23/2022, 2:30 PM
Yeah, that's a bug somewhere in Pants / Pex / Pip but not in bandit as determined above. I can take a look here shortly.
l

lively-dusk-46231

10/23/2022, 2:32 PM
Thanks !
Just a FYI - using
[bandit].version = 'bandit<1.8,>=1.7.0,!=1.7.2
- made it work (no clue why anymore).
w

wide-midnight-78598

10/23/2022, 2:43 PM
Would any of this ticket you created be relevant in Pants? https://github.com/PyCQA/bandit/issues/960
🤷‍♂️ 1
b

bitter-ability-32190

10/23/2022, 2:45 PM
I can't remember if the ICs make it into the lockfile. If you look at the lockfile, does it list the ICs it was generated with?
l

lively-dusk-46231

10/23/2022, 2:46 PM
Yes, the interpreter constraints are mentioned:
Copy code
$ head -20 bandit.lock
// This lockfile was autogenerated by Pants. To regenerate, run:
//
//    ./pants generate-lockfiles --resolve=bandit
//
// --- BEGIN PANTS LOCKFILE METADATA: DO NOT EDIT OR REMOVE ---
// {
//   "version": 2,
//   "valid_for_interpreter_constraints": [
//     "CPython<4,>=3.6" <------------------------------------------ HERE?
//   ],
//   "generated_with_requirements": [
//     "GitPython==3.1.18",
//     "bandit<1.8,>=1.7.0",
//     "setuptools"
//   ]
// }
// --- END PANTS LOCKFILE METADATA ---
b

bitter-ability-32190

10/23/2022, 2:48 PM
What happens if you pin 3.7 in your pants.toml and relock?
l

lively-dusk-46231

10/23/2022, 2:50 PM
Using
[python].interpreter_constraints = ['CPython>=3.7,<=3.8']
- makes it work (uses bandit 1.7.4)
Using
[python].interpreter_constraints = ['CPython>=3.6,<=3.7']
- FAILS (uses bandit 1.7.2 - which is meant for py37+)
w

wide-midnight-78598

10/23/2022, 3:02 PM
What does
python3.6 -m pip install bandit==1.7.2
do?
l

lively-dusk-46231

10/23/2022, 3:31 PM
I have 2 pythons installed on my system that are v3.6 Using anaconda:
Copy code
$ ~/python36/bin/python -V
Python 3.6.5 :: Anaconda, Inc.
$ ~/python36/bin/python -m pip -V
pip 21.0.1
$ ~/python36/bin/python -m pip install bandit==1.7.2
ERROR: Package 'bandit' requires a different Python: 3.6.5 not in '>=3.7'
$ ~/python36/bin/python -m pip install 'bandit<=1.7.2'
Successfully installed bandit-1.7.1
System python in Ubuntu 18.04 (WSL2 in Windows 10):
Copy code
$ /usr/bin/python3 -V
Python 3.6.9
$ /usr/bin/python3 -m pip -V
pip 9.0.1
$ /usr/bin/python3 -m pip install bandit==1.7.2
bandit requires Python '>=3.7' but the running Python is 3.6.9
$ /usr/bin/python3 -m pip install 'bandit<=1.7.2'
bandit requires Python '>=3.7' but the running Python is 3.6.9
e

enough-analyst-54434

10/23/2022, 4:23 PM
I've played around with this and it seems to be this all boils down to the Requires-Python metadata being present for 1.7.2 but not published on PyPI. Neither 1.7.1 nor 1.7.3 have this problem. FWICT there is a known issue that Warehouse (the PyPI implementation), takes metadata from the 1st published dist for a given release version. If twine (used to upload dists) uploads the sdist 1st this can foil things since sdists rarely have full valid metadata in their PKG-INFO. This is unlike wheels which do have full valid metadata in their METADATA dist-info file.
It appears to be the case that once set for a release, metadata is immutable. I've tried a few things with a small test project of mine.
I need to dig more here into the twine / upload order issue thing and to see whether or not PyPI accepts requests to purge / re-create metadata fron existing uploaded files, but @lively-dusk-46231 your `!=1.72`clause is the right way to fix this. As to whether Pants should include that in its stock requirement string - I'm not sure. I can't see it hurting, but this only affects folks who use Python 3.5 and Python 3.6 fwict since 1.7.1 has
Requires-Python >=3.5
.
l

lively-dusk-46231

10/23/2022, 4:36 PM
Yikes - scary I'll go ahead with my workaround for now I'm going to create a ticket in the pants github so we can document this issue in case other people face it
e

enough-analyst-54434

10/23/2022, 4:36 PM
Thanks - that would be good.
l

lively-dusk-46231

10/23/2022, 4:40 PM
Feel free to add any notes from your side or close as you think best: https://github.com/pantsbuild/pants/issues/17329
e

enough-analyst-54434

10/23/2022, 4:46 PM
Thank you. Yeah, it turns out the scary is in fact scary. Added data to the ticket, but for 1.7.2 and 1.7.2 only, the sdist was uploaded 1st.
7 Views