https://pantsbuild.org/ logo
Join Slack
Powered by
I found an issue with `bandit` when debugging some...
# general
l
I found an issue with 
bandit
when debugging some version issues with 
pants
https://github.com/PyCQA/bandit/issues/960 Do you think it makes sense to change the default 
[bandit].version
from 
bandit<1.8,>=1.7.0
-> 
bandit<1.8,>=1.7.0,!=1.7.2
? If yes - I can create an issue in github
e
That does not look like a bug to me:
bandit requires Python '>=3.7' but the running Python is 3.6.5
That's just bandit saying they dropped older Python support. Projects do this all the time. You might quibble that they should have only done this in a minor or major version bump and not in a patch version bump, but the ship has sailed.
If you look at latest bandit, this is still the case; so your proposed Pants fix won't work: https://pypi.org/project/bandit/1.7.4/#data 
bandit>=1.7.2
will simply never work for Python 3.6.
l
Hi John, I understand dropping support for older pythons. But that is not the case here When I read bandit==1.7.2 's pypi page. It does not have any "python requires" on the left side. So when I do 
pip3.6 install bandit
it is fetching bandit 1.7.2 If bandit 1.7.2 does not support py3.6 It should have fetched bandit=1.7.1 I expect pants to take bandit 1.7.1 if I have interpreter constraints as py3.6 When I tried it without pants, I saw the issue with even pip - pip3.6 should have taken bandit 1.7.1 Similar issues don't occur with other libraries. If I do 
pip3.6 install flask
it installs the last known version of flask that works with py3.6
e
@lively-dusk-46231 I think the issue here is you have such an old version of Pip that it does not understand 
Requires-Python
metadata. Here's where PBR did the right things in 
setup.cfg
and 
setup.py
to drop support: https://github.com/PyCQA/bandit/pull/777 And here's the correct result:
Copy code
unzip -qc bandit-1.7.2-py3-none-any.whl bandit-1.7.2.dist-info/METADATA | grep Requires-Python
Requires-Python: >=3.7
Aha, the issue is too new a Pip: https://github.com/pypa/pip/issues/11142
l
Hmm - you're right. When using 
pip
directly - I can fix the issue by using ~21 Any way for me to check the pip version that pants is using ? I think it is creating a pex file based on my interpreter constraints ?
e
It uses a vendored version of 20.3.4.
l
😕 I see - looks like 20.3.4 is known to have this bug: https://github.com/pypa/pip/issues/9925#issuecomment-830512306
actually - I'm not sure. Looks like this was just an optimization not a bug ... Hmm. I can't figure out what is going wrong
e
The short story is this does not seem like a bandit bug.
👍 1
l
Makes sense - not a bandit bug. Thanks for the to and fro to help me debug this🙂 But I feel like the pants or pex or pip behavior somewhere is not correct. Will redo my initial test and report the issue
This is reproducible:
Copy code
$ cat pants.toml
[GLOBAL]
pants_version = "2.13.0"
backend_packages = [
  "pants.backend.python",
  "pants.backend.python.lint.bandit",
]

[python]
interpreter_constraints = ['CPython>=3.6,<4']

[bandit]
lockfile = 'bandit.lock'

$ cat BUILD
python_sources(name="root")
Now when I do:
Copy code
./pants generate-lockfiles
It creates a lock file with 
bandit==1.7.2
Now when I run:
Copy code
./pants lint ::
It raises:
Copy code
Dependency on bandit not satisfied, 1 incompatible candidate found:
1.) bandit 1.7.2 (via: bandit<1.8,>=1.7.0) requires Python >=3.7
This is not the behavior I would have expected. Shouldn't "something" have chosen bandit==1.7.1 in the lockfile to satisfy my interpreter constraints ?
e
Yeah, that's a bug somewhere in Pants / Pex / Pip but not in bandit as determined above. I can take a look here shortly.
l
Thanks !
Just a FYI - using 
[bandit].version = 'bandit<1.8,>=1.7.0,!=1.7.2
- made it work (no clue why anymore).
w
Would any of this ticket you created be relevant in Pants? https://github.com/PyCQA/bandit/issues/960
🤷‍♂️ 1
b
I can't remember if the ICs make it into the lockfile. If you look at the lockfile, does it list the ICs it was generated with?
l
Yes, the interpreter constraints are mentioned:
Copy code
$ head -20 bandit.lock
// This lockfile was autogenerated by Pants. To regenerate, run:
//
//    ./pants generate-lockfiles --resolve=bandit
//
// --- BEGIN PANTS LOCKFILE METADATA: DO NOT EDIT OR REMOVE ---
// {
//   "version": 2,
//   "valid_for_interpreter_constraints": [
//     "CPython<4,>=3.6" <------------------------------------------ HERE?
//   ],
//   "generated_with_requirements": [
//     "GitPython==3.1.18",
//     "bandit<1.8,>=1.7.0",
//     "setuptools"
//   ]
// }
// --- END PANTS LOCKFILE METADATA ---
b
What happens if you pin 3.7 in your pants.toml and relock?
l
Using 
[python].interpreter_constraints = ['CPython>=3.7,<=3.8']
- makes it work (uses bandit 1.7.4)
Using 
[python].interpreter_constraints = ['CPython>=3.6,<=3.7']
- FAILS (uses bandit 1.7.2 - which is meant for py37+)
w
What does 
python3.6 -m pip install bandit==1.7.2
do?
l
I have 2 pythons installed on my system that are v3.6 Using anaconda:
Copy code
$ ~/python36/bin/python -V
Python 3.6.5 :: Anaconda, Inc.
$ ~/python36/bin/python -m pip -V
pip 21.0.1
$ ~/python36/bin/python -m pip install bandit==1.7.2
ERROR: Package 'bandit' requires a different Python: 3.6.5 not in '>=3.7'
$ ~/python36/bin/python -m pip install 'bandit<=1.7.2'
Successfully installed bandit-1.7.1
System python in Ubuntu 18.04 (WSL2 in Windows 10):
Copy code
$ /usr/bin/python3 -V
Python 3.6.9
$ /usr/bin/python3 -m pip -V
pip 9.0.1
$ /usr/bin/python3 -m pip install bandit==1.7.2
bandit requires Python '>=3.7' but the running Python is 3.6.9
$ /usr/bin/python3 -m pip install 'bandit<=1.7.2'
bandit requires Python '>=3.7' but the running Python is 3.6.9
e
I've played around with this and it seems to be this all boils down to the Requires-Python metadata being present for 1.7.2 but not published on PyPI. Neither 1.7.1 nor 1.7.3 have this problem. FWICT there is a known issue that Warehouse (the PyPI implementation), takes metadata from the 1st published dist for a given release version. If twine (used to upload dists) uploads the sdist 1st this can foil things since sdists rarely have full valid metadata in their PKG-INFO. This is unlike wheels which do have full valid metadata in their METADATA dist-info file.
It appears to be the case that once set for a release, metadata is immutable. I've tried a few things with a small test project of mine.
I need to dig more here into the twine / upload order issue thing and to see whether or not PyPI accepts requests to purge / re-create metadata fron existing uploaded files, but @lively-dusk-46231 your `!=1.72`clause is the right way to fix this. As to whether Pants should include that in its stock requirement string - I'm not sure. I can't see it hurting, but this only affects folks who use Python 3.5 and Python 3.6 fwict since 1.7.1 has 
Requires-Python >=3.5
.
l
Yikes - scary I'll go ahead with my workaround for now I'm going to create a ticket in the pants github so we can document this issue in case other people face it
e
Thanks - that would be good.
l
Feel free to add any notes from your side or close as you think best: https://github.com/pantsbuild/pants/issues/17329
e
Thank you. Yeah, it turns out the scary is in fact scary. Added data to the ticket, but for 1.7.2 and 1.7.2 only, the sdist was uploaded 1st.
7 Views
Open in Slack
PreviousNext
Powered by