In my project, we need to support python 3.6+ So, we use older dependencies for 3.6 And newer dependencies for 3.7+ Is there an option to do that with the lockfile approach in pants ? When I tried to set my interpreter constraints to 3.6+ bandit caused issues. And on setting bandit version to <1.8 and creating my own lockfile, bandit.lock has bandit==1.7.x So that lockfile could not be used with py3.6 When I set bandit version <1.6 it started working fine. But I would prefer using the newer bandit for latest python versions (Bandit isn't a big concern for me. But many of my runtime dependencies like pandas, network, flask. Etc need to be latest versions to avoid Vulnerabilities from the older versions)

Have you been able to play around with all the various

interpreter_constraints

?

pants.toml

https://www.pantsbuild.org/docs/reference-python#interpreter_constraints

python_source

https://www.pantsbuild.org/docs/reference-python_source#codeinterpreter_constraintscode

subsystem

(e.g. Black) https://www.pantsbuild.org/docs/reference-black#interpreter_constraints And there are probably more.... Or maybe a lockfile for a given tool? https://www.pantsbuild.org/docs/python-third-party-dependencies#tool-lockfiles Or if you have conflicting deps, maybe multiple lockfiles for the project? https://www.pantsbuild.org/docs/python-third-party-dependencies#multiple-lockfiles

I am already using a

[bandit].lockfile

for the bandit tool. But

[bandit].interpreter_constraints

does not exist. So, I have set

[python].interpreter_constraints

Also - I guess I need 2 lockfiles for bandit: • One for python 3.6 which uses bandit==1.6.x • Second for bandit 3.7+ which uses bandit=1.7.x I couldn't find a way to have 2 separate lockfiles for

bandit

You should be able to hack this with 1 lockfile by getting creative with the options here: https://www.pantsbuild.org/docs/reference-bandit#advanced-options Try:

[bandit]
version = "bandit==1.6.*; python_version < '3.7'"
extra_requirements = [
  "bandit==1.7.*; python_version >= '3.7'",
]

ah - interesting.

I was lazy - from my phone - so have not tried.

No worries - thanks for the idea, will try it out

Yeah - that doesn't work. So I think with the current state of the art in Pants, you simply have to accept and old version of bandit even for your Python >=3.7 projects. Supporting multiple versions of a tool would be a new feature.

I see. So, if I was to summarize: • Support bandit for both py3.6 and py3.7+ ◦ Supporting multiple lockfiles for bandit (which is not available now) ◦ Or a single lockfile for multiple pythons (which is not available now) • Run bandit for a specific version of python/bandit and just run that ◦ Assume the tool has a version which is compatible with

[python].interpreter_constraints

and use that (this is the main option right now) (normally the lowest version) ◦ Add a

[bandit].interpreter_constraints

and let users decide which interpreter to use so they can decide oldest/newest available version manually (which is not available now) ◦ Create separate pants.toml file for this specifically (feels like a workaround ?) I feel like the easiest option that pants could provide is to have a

[bandit].interpreter_constraints

So I can atleast constraint that based on py3.6 or py3.9 and then force all my developers + CI to just use that version for the linting I notice that [isort].interpreter_constraints and [black].interpreter_constraints , audoflake8, already exists but equivalent for other tools like flake8, bandit does not. Was that a conscious decision ?

Or a single lockfile for multiple pythons (which is not available now)

That part is incorrect. We support this and dogfood it in the lock files we ship with Pants which support multiple interpreters and platforms. You just happen to hit a case (bandit + 3.6) with buggy metadata on PyPI.

Oh! But then this should work no ? (note that I used explicit versions)

[bandit]
version = "bandit==1.7.1; python_version < '3.7'"
extra_requirements = [
  "bandit==1.7.4; python_version >= '3.7'",
]

No. That's 1 lockfile with divergent requirements. That, which I erroneously suggested, will never work.

I see, that is what I meant with:

Or a single lockfile for multiple pythons (which is not available now)

Maybe I didn't write it clearly. I meant to say a single lockfile which has multiple versions of bandit for different versions of python

How would that help? You'd just be fixing the ICs for bandit to a new value. FWICT you need to fix it to two different values in different contexts. Do I have that right?

I see.

IIUC yours is the 1st case that hits an issue here so it seems to be a rare corner. Not that more bug reports may not be forthcoming from others, but the basics of "I need to use unsupported Python" plus "I also want to use the latest bandit" is a tough ~conflicting set of requirements to have in a single repo.

I guess the way I run my CI right now is: • py3.6 -> installs latest flake8, bandit, isort, black for py36 and do lint/fmt. Then install latest pytest and run to ensure py3.6 works right • py3.7 -> installs latest flake8, bandit, isort, black for py37 and do lint/fmt. Then install latest pytest and run to ensure py3.7 works right • py3.8 -> installs latest flake8, bandit, isort, black for py38 and do lint/fmt. Then install latest pytest and run to ensure py3.8 works right • py3.9 -> installs latest flake8, bandit, isort, black for py39 and do lint/fmt. Then install latest pytest and run to ensure py3.9 works right I'm not able to visualize how I could do this with pants

For example, I maintain Pex, which still supports Python 2.7, It's ICs are ">=2.7,!=3.0.,!=3.1.,!=3.2.,!=3.3.,!=3.4.*,<3.12" and I just accept I need to use older libraries if I want to support that range of customers.

I see. I see. Making more sense

you just override the options that need to change

Ah - this is great. I was thinking I would need to repeat.

@lively-dusk-46231 do you use tox or nox to do this?

^ Right now I just have a simple Dockerfile which installs all the python versions and runs a

lint.sh

in a bash for loop 🙂 (with different

PATH

)

Aha, ok.

Yep, going through that right now - thanks for the heads up 👍 Appreciate all the inputs

FWIW, I would classify your current CI setup the tox style. I do ~this for Pex, installing newer tools for newer Pythons: https://github.com/pantsbuild/pex/blob/6a97efd24ec0b524d8a701d5491c8aa1791f34d4/tox.ini#L17-L71 That's two different versions of pytest depending of which python I'm testing on my machine or in CI and a whole heck of a lot of pythons taht are individually tested against: https://github.com/pantsbuild/pex/actions/runs/3301442033

Hmm, but I see your tests are also taking 1h+ (with multiple runners in parallel) Your total machine time is 30h+

Yup. There are tradeoffs currently.