https://pantsbuild.org/ logo
#general
Title
# general
l

lively-dusk-46231

10/22/2022, 7:17 PM
In my project, we need to support python 3.6+ So, we use older dependencies for 3.6 And newer dependencies for 3.7+ Is there an option to do that with the lockfile approach in pants ? When I tried to set my interpreter constraints to 3.6+ bandit caused issues. And on setting bandit version to <1.8 and creating my own lockfile, bandit.lock has bandit==1.7.x So that lockfile could not be used with py3.6 When I set bandit version <1.6 it started working fine. But I would prefer using the newer bandit for latest python versions (Bandit isn't a big concern for me. But many of my runtime dependencies like pandas, network, flask. Etc need to be latest versions to avoid Vulnerabilities from the older versions)
w

wide-midnight-78598

10/22/2022, 8:31 PM
Have you been able to play around with all the various
interpreter_constraints
?
pants.toml
https://www.pantsbuild.org/docs/reference-python#interpreter_constraints
python_source
https://www.pantsbuild.org/docs/reference-python_source#codeinterpreter_constraintscode
subsystem
(e.g. Black) https://www.pantsbuild.org/docs/reference-black#interpreter_constraints And there are probably more.... Or maybe a lockfile for a given tool? https://www.pantsbuild.org/docs/python-third-party-dependencies#tool-lockfiles Or if you have conflicting deps, maybe multiple lockfiles for the project? https://www.pantsbuild.org/docs/python-third-party-dependencies#multiple-lockfiles
l

lively-dusk-46231

10/23/2022, 3:37 AM
I am already using a
[bandit].lockfile
for the bandit tool. But
[bandit].interpreter_constraints
does not exist. So, I have set
[python].interpreter_constraints
Also - I guess I need 2 lockfiles for bandit: • One for python 3.6 which uses bandit==1.6.x • Second for bandit 3.7+ which uses bandit=1.7.x I couldn't find a way to have 2 separate lockfiles for
bandit
Looks like for the runtime libraries (i.e.
install_requires
) I can use resolves_to_interpreter_constraints So that every resolve has it's own set of interpreter constraints But nothing equivalent for tools like bandit
e

enough-analyst-54434

10/23/2022, 2:09 PM
You should be able to hack this with 1 lockfile by getting creative with the options here: https://www.pantsbuild.org/docs/reference-bandit#advanced-options Try:
Copy code
[bandit]
version = "bandit==1.6.*; python_version < '3.7'"
extra_requirements = [
  "bandit==1.7.*; python_version >= '3.7'",
]
l

lively-dusk-46231

10/23/2022, 2:27 PM
ah - interesting.
e

enough-analyst-54434

10/23/2022, 2:34 PM
I was lazy - from my phone - so have not tried.
l

lively-dusk-46231

10/23/2022, 2:36 PM
No worries - thanks for the idea, will try it out
e

enough-analyst-54434

10/23/2022, 4:31 PM
Yeah - that doesn't work. So I think with the current state of the art in Pants, you simply have to accept and old version of bandit even for your Python >=3.7 projects. Supporting multiple versions of a tool would be a new feature.
If just testing with a newer version of bandit in CI is acceptable you could always set up your CI to run 2 different
./pants
commands. One could use the default
pants.toml
and another could use
PANTS_CONFIG_FILES=pants.37plus.toml ./pants ...
or vice versa. That general idea of modifying Pants under CI is sketched out here: https://www.pantsbuild.org/docs/using-pants-in-ci#configuring-pants-for-ci-pantscitoml-optional
l

lively-dusk-46231

10/23/2022, 5:40 PM
I see. So, if I was to summarize: • Support bandit for both py3.6 and py3.7+ ◦ Supporting multiple lockfiles for bandit (which is not available now) ◦ Or a single lockfile for multiple pythons (which is not available now) • Run bandit for a specific version of python/bandit and just run that ◦ Assume the tool has a version which is compatible with
[python].interpreter_constraints
and use that (this is the main option right now) (normally the lowest version) ◦ Add a
[bandit].interpreter_constraints
and let users decide which interpreter to use so they can decide oldest/newest available version manually (which is not available now) ◦ Create separate pants.toml file for this specifically (feels like a workaround ?) I feel like the easiest option that pants could provide is to have a
[bandit].interpreter_constraints
So I can atleast constraint that based on py3.6 or py3.9 and then force all my developers + CI to just use that version for the linting I notice that [isort].interpreter_constraints and [black].interpreter_constraints , audoflake8, already exists but equivalent for other tools like flake8, bandit does not. Was that a conscious decision ?
e

enough-analyst-54434

10/23/2022, 5:45 PM
Or a single lockfile for multiple pythons (which is not available now)
That part is incorrect. We support this and dogfood it in the lock files we ship with Pants which support multiple interpreters and platforms. You just happen to hit a case (bandit + 3.6) with buggy metadata on PyPI.
l

lively-dusk-46231

10/23/2022, 5:47 PM
Oh! But then this should work no ? (note that I used explicit versions)
Copy code
[bandit]
version = "bandit==1.7.1; python_version < '3.7'"
extra_requirements = [
  "bandit==1.7.4; python_version >= '3.7'",
]
e

enough-analyst-54434

10/23/2022, 5:48 PM
No. That's 1 lockfile with divergent requirements. That, which I erroneously suggested, will never work.
Pex supports 1 lockfile with multiple separate locks, but Pants does not use that feature.
l

lively-dusk-46231

10/23/2022, 5:50 PM
I see, that is what I meant with:
Or a single lockfile for multiple pythons (which is not available now)
Maybe I didn't write it clearly. I meant to say a single lockfile which has multiple versions of bandit for different versions of python
Any thoughts on having a
[bandit].interpreter_constraints
? Or rather just having a
interpreter_constraints
that every tool could override ?
e

enough-analyst-54434

10/23/2022, 5:53 PM
How would that help? You'd just be fixing the ICs for bandit to a new value. FWICT you need to fix it to two different values in different contexts. Do I have that right?
So, yeah, you'd need the ability to specify more than one tool lockfile, each one covering a disjoint subset of the ICs your repository code uses.
That's certainly doable, but it also sounds complex; so that would need to be weighed against workarounds like using seperate tomls to divide up the
./pants lint runs
.
l

lively-dusk-46231

10/23/2022, 5:56 PM
I see.
e

enough-analyst-54434

10/23/2022, 5:57 PM
IIUC yours is the 1st case that hits an issue here so it seems to be a rare corner. Not that more bug reports may not be forthcoming from others, but the basics of "I need to use unsupported Python" plus "I also want to use the latest bandit" is a tough ~conflicting set of requirements to have in a single repo.
l

lively-dusk-46231

10/23/2022, 6:00 PM
I guess the way I run my CI right now is: • py3.6 -> installs latest flake8, bandit, isort, black for py36 and do lint/fmt. Then install latest pytest and run to ensure py3.6 works right • py3.7 -> installs latest flake8, bandit, isort, black for py37 and do lint/fmt. Then install latest pytest and run to ensure py3.7 works right • py3.8 -> installs latest flake8, bandit, isort, black for py38 and do lint/fmt. Then install latest pytest and run to ensure py3.8 works right • py3.9 -> installs latest flake8, bandit, isort, black for py39 and do lint/fmt. Then install latest pytest and run to ensure py3.9 works right I'm not able to visualize how I could do this with pants
e

enough-analyst-54434

10/23/2022, 6:00 PM
For example, I maintain Pex, which still supports Python 2.7, It's ICs are ">=2.7,!=3.0.,!=3.1.,!=3.2.,!=3.3.,!=3.4.*,<3.12" and I just accept I need to use older libraries if I want to support that range of customers.
You'd do it the same way - have one
pants.36.toml
, etc for each python version, then run pants with
PANTS_CONFIG_FILES=pants.36.toml ./pants ...
for each toml.
In all those specialized tomls you just override the options that need to change for each Python version. You may be able to do better than this and get faster runs by using less tomls and exploiting Pants internal parallelism and caching, but you might also just start with simple translation of your current CI to Pants and then whittle away from there.
This is not a ideal, but its a very straightforward way to start and compare results before digging further.
l

lively-dusk-46231

10/23/2022, 6:05 PM
I see. I see. Making more sense
you just override the options that need to change
Ah - this is great. I was thinking I would need to repeat.
e

enough-analyst-54434

10/23/2022, 6:05 PM
@lively-dusk-46231 do you use tox or nox to do this?
l

lively-dusk-46231

10/23/2022, 6:06 PM
^ Right now I just have a simple Dockerfile which installs all the python versions and runs a
lint.sh
in a bash for loop 🙂 (with different
PATH
)
e

enough-analyst-54434

10/23/2022, 6:06 PM
Aha, ok.
Yeah, read https://www.pantsbuild.org/docs/using-pants-in-ci#configuring-pants-for-ci-pantscitoml-optional closely and maybe https://www.pantsbuild.org/docs/options - PANTS_CONFIG_FILES=x adds x to the list of config files Pants parses and left-most files are evaluated before right-most files.
l

lively-dusk-46231

10/23/2022, 6:08 PM
Yep, going through that right now - thanks for the heads up 👍 Appreciate all the inputs
e

enough-analyst-54434

10/23/2022, 6:12 PM
FWIW, I would classify your current CI setup the tox style. I do ~this for Pex, installing newer tools for newer Pythons: https://github.com/pantsbuild/pex/blob/6a97efd24ec0b524d8a701d5491c8aa1791f34d4/tox.ini#L17-L71 That's two different versions of pytest depending of which python I'm testing on my machine or in CI and a whole heck of a lot of pythons taht are individually tested against: https://github.com/pantsbuild/pex/actions/runs/3301442033
Pants could stand to support this better. Part of the reason Pex does not use Pants yet is, in fact, this.
l

lively-dusk-46231

10/23/2022, 6:17 PM
Hmm, but I see your tests are also taking 1h+ (with multiple runners in parallel) Your total machine time is 30h+
My guess is that the pants caching mechanism would reduce that drastically to only test/run for modified files
e

enough-analyst-54434

10/23/2022, 6:23 PM
Yup. There are tradeoffs currently.
3 Views