https://pantsbuild.org/ logo
#general
Title
# general
f

flat-zoo-31952

02/01/2022, 5:14 PM
How does Pants decide how many jobs to run for parallel invocations? I'm running
./pants lint ::
with the
flake8
and
pylint
backends enabled, and it seemed to have run flake8 massively parallel, and pylint in just one job. Is this an intentional decision due to bugs in pylint?
w

witty-crayon-22786

02/01/2022, 5:15 PM
which version of Pants?
f

flat-zoo-31952

02/01/2022, 5:15 PM
2.9.0
w

witty-crayon-22786

02/01/2022, 5:16 PM
2.9.0 doesn’t do much to control parallelism in those tools, but 2.10.x will: https://github.com/pantsbuild/pants/pull/14186 and https://github.com/pantsbuild/pants/pull/14184 are in there
f

flat-zoo-31952

02/01/2022, 5:18 PM
that's fine, I can skip the pylint backend and just pass the results of a
--changed-dependees=transitive
call to pylint
w

witty-crayon-22786

02/01/2022, 5:19 PM
in the meantime, you can explicitly pass concurrency settings to linters: see https://github.com/pantsbuild/pants/issues/13462#issuecomment-1007504648 …
pylint
in particular is insanely slow: like 10x of
mypy
even
b

bitter-ability-32190

02/01/2022, 5:40 PM
pylint
looks at source code contents to infer types, whereas
mypy
has strict rules and uses annotations
f

flat-zoo-31952

02/01/2022, 5:41 PM
Yeah... I'm not wild about pylint in general, but we use it and I'm not ready to fight that battle yet
βž• 1
b

bitter-ability-32190

02/01/2022, 5:41 PM
Same 😐
I have a long-standing wish for a modern linter to enter the Python world which leverages type annotations (maybe even using
mypy
under the hood) (
mypy
only really lints "types", so it's not sufficient for all my linting needs)
f

flat-zoo-31952

02/01/2022, 5:43 PM
should be faster anyway when I can run it on changes rather than on the whole repo
b

bitter-ability-32190

02/01/2022, 5:43 PM
(It's still slow as hell πŸ™‚ )
f

flat-zoo-31952

02/01/2022, 5:44 PM
I'm a bit more broad, I'm a static analysis skeptic in general πŸ˜…
❗ 1
😒 1
b

bitter-ability-32190

02/01/2022, 5:44 PM
FWIW I plan on finding the
pylint
rules that
mypy
has basically made impotent and turning them off. I can share findings here. I would hope tossing those out would make it much speedier 🀞
πŸ™ 1
πŸ™πŸ» 1
f

flat-zoo-31952

02/01/2022, 5:44 PM
I think it's all a big excuse for devs to avoid having to write more testable code
b

bitter-ability-32190

02/01/2022, 5:45 PM
That's just sucky devs. Don't hate the game, player.
f

flat-zoo-31952

02/01/2022, 5:53 PM
It's good devs too. A lot of very talented devs don't consider testing as part of their design because their skill makes them overconfident. And I think they've looked at enough overly mocked tests and assumed that you can't test anything important with unit tests. So you end up with a bunch of code that's impossible to test, and everyone's afraid to refactor, and assume that tacking on linters and typecheckers will fix the problem. And they do a little bit, but in many cases I'm not convinced the cost is worth it
Going back to pylint... it seems to run much faster on folders than it does on a list of files. I'll need to gather some more data on this, but that's unfortunate for a goal of having small cache keys
b

bitter-ability-32190

02/01/2022, 6:00 PM
Those aren't "good" devs in my book πŸ˜‰
βž• 1
And yes, back on topic, @witty-crayon-22786 put in 2 changes to make
pylint
runs a bit more stomachable. I'll find the PRs
f

flat-zoo-31952

02/01/2022, 6:26 PM
I'll give that a try when 2.10 comes out. For now I think I can do something like
Copy code
./pants --changed-since=$CMP_BRANCH --changed-dependees=transitive list \
  | xargs ./pants filter --target-type=python_source \
  | sed -r 's,(^//|:[^:]+$),,g' \
  | xargs pylint $PYLINT_OPTIONS
h

hundreds-father-404

02/01/2022, 6:48 PM
We're hoping for first release candidate to be this week
πŸ‘πŸ» 1
πŸ‘ 1
b

bitter-ability-32190

02/03/2022, 9:28 PM
OK These are seemingly all handled by mypy:
Copy code
# E1700 (yield-inside-async-function)
# E0102 (function-redefined) [no-redef]
# E0103 (not-in-loop)
# E0104 (return-outside-function)
# E0105 (yield-outside-function)
# E0107 (nonexistent-operator)
# E0108 (duplicate-argument-name)
# E0110 (abstract-class-instantiated)
# E0112 (too-many-star-expressions)
# E0113 (invalid-star-assignment-target)
# E0114 (star-needs-assignment-target)
# E0119 (misplaced-format-function) [attr-defined]
# W0111 (assign-to-new-keyword)
# W0124 (confusing-with-statement)
# W0143 (comparison-with-callable)
# W0199 (assert-on-tuple)
# E0202 (method-hidden)
# E0203 (access-member-before-definition)
# E0211 (no-method-argument)
# E0213 (no-self-argument)
# E0237 (assigning-non-slot)
# E0238 (invalid-slots)
# E0239 (inherit-non-class)
# E0240 (inconsistent-mro)
# E0241 (duplicate-bases)
# R0203 (no-staticmethod-decorator)
# R0206 (property-with-parameters)
# W0221 (arguments-differ)
# W0222 (signature-differs)
# E0702 (raising-bad-type)
# E0703 (bad-exception-context)
# E0710 (raising-non-exception)
# E0712 (catching-non-exception)
# W0711 (binary-op-exception)
# W0716 (wrong-exception-operation)
# E1507 (invalid-envvar-value)
# W1506 (bad-thread-instantiation)
# E1101 (no-member)

# E1102 (not-callable)
# E1111 (assignment-from-no-return)
# E1120 (no-value-for-parameter)
# E1121 (too-many-function-args)
# E1123 (unexpected-keyword-arg)
# E1124 (redundant-keyword-arg)
# E1125 (missing-kwoa)
# E1126 (invalid-sequence-index)
# E1127 (invalid-slice-index)
# E1128 (assignment-from-none)
# E1129 (not-context-manager)
# E1130 (invalid-unary-operand-type)
# E1131 (unsupported-binary-operation)
# E1132 (repeated-keyword)
# E1133 (not-an-iterable)
# E1134 (not-a-mapping)
# E1135 (unsupported-membership-test)
# E1136 (unsubscriptable-object)
# E1137 (unsupported-assignment-operation)
# E1138 (unsupported-delete-operation)
# E1139 (invalid-metaclass)
# E1140 (unhashable-dict-key)
# E1141 (dict-iter-missing-items)
# I1101 (c-extension-no-mem)
# W1113 (keyword-arg-before-vararg)
# W1114 (arguments-out-of-order)
# E0604 (invalid-all-object)
# E0611 (no-name-in-module)
# E0633 (unpacking-non-sequence)
# W0601 (global-variable-undefined)
# W0632 (unbalanced-tuple-unpacking)
πŸ™Œ 1
Plus these if your code is annotated, and/or you call the relevant code (so not error by-default but will likely error by mypy at the callsite when you try and use the code)
Copy code
# E0100 (init-is-generator)
# E0101 (return-in-init)
# E0301 (non-iterator-returned)
# W0236 (invalid-overridden-method)
# E1701 (not-async-context-manager) [attr-defined]
# E0302 (unexpected-special-method-signature)
h

hundreds-father-404

02/03/2022, 9:31 PM
cool! how did you figure that out?
b

bitter-ability-32190

02/03/2022, 9:33 PM
Just copied the example code from https://vald-phoenix.github.io/pylint-errors/ into a file and ran
mypy
on it. Rinse Repeat.
πŸ™Œ 1
(I did look at the code first and did some reasoning. I found at least one example bug)
h

hundreds-father-404

02/03/2022, 9:34 PM
neat, that would be super cool to write a blog showing this. not necessarily Pantsbuild blog, just in genera
b

bitter-ability-32190

02/03/2022, 9:39 PM
I guess I need a blog
Trimming the fat got us from a completely serial run of
pylint
from
5m18s
to
4m40s
. So not groundbreaking, but something
I'll nuke the formatting codes as well once we're on
black
6 Views