https://pantsbuild.org/ logo
#general
Title
# general
h

happy-kitchen-89482

01/19/2023, 8:20 PM
@rhythmic-glass-66959 I think Pants doesn't generate reports by default, so presumably you're setting cli args for bandit and pylint (and flake8 for that matter) to make the emit reports? Can you share your config?
r

rhythmic-glass-66959

01/19/2023, 8:24 PM
Sure:
Copy code
[bandit]
args = ["-f", "json", "-o", "reports/bandit-report.json"]

[flake8]
args = ["--output-file=reports/flake8-report.txt"]

[pylint]
args = ["--output-format=text:reports/pylint-report.txt"]
h

happy-kitchen-89482

01/19/2023, 8:27 PM
That seems right to me, and Pants creates the
reports
subdir in the sandbox and captures its output into dist.
So the problem is that we have two different partitions with the same description
Why is this being partitioned at all, I wonder
partitioning should be by interpreter constraints, but these partitions use the same constraints, it looks like
r

rhythmic-glass-66959

01/19/2023, 8:30 PM
Yep.
Copy code
[python]
interpreter_constraints = ["==3.9.13"]
s

sparse-lifeguard-95737

01/19/2023, 8:33 PM
@happy-kitchen-89482 this is happening because we have a max-batch-size on lint processes. so even though there’s only 1 partition, the files within that partition get divided up again
h

happy-kitchen-89482

01/19/2023, 8:33 PM
Oh, this may be because of batching
s

sparse-lifeguard-95737

01/19/2023, 8:33 PM
coke
h

happy-kitchen-89482

01/19/2023, 8:34 PM
Yeah, we should be generating a single report for all batches in a partition, if that is possible
s

sparse-lifeguard-95737

01/19/2023, 8:34 PM
in the
test
goal we avoid this situation by naming any generated report files after a target in the generating batch (taking as an invariant that each target will be in 1 batch only)
h

happy-kitchen-89482

01/19/2023, 8:34 PM
Or at least, that is what the user expects
that may not be possible
Yeah, so to mitigate we need to give more sensible names to the partitions I guess
To the batches, rather
or at least, number them, instead of adding that
_
suffix
s

sparse-lifeguard-95737

01/19/2023, 8:36 PM
if we can make the individual report files uniquely-named across batches, then (I believe) we could write them all into a directory named after the partition (minus any `_`s)
you’d still end up with a bunch of files though…
h

happy-kitchen-89482

01/19/2023, 8:36 PM
True true!
But some tools at least may have a way to merge them
like codecov does for coverage files
That is probably the way to go - unique filenames within the partition's dir
r

rhythmic-glass-66959

01/19/2023, 8:37 PM
Hum, I'm wondering, we recently added a git submodule, could this cause the issue?
h

happy-kitchen-89482

01/19/2023, 8:37 PM
I like that
s

sparse-lifeguard-95737

01/19/2023, 8:37 PM
@rhythmic-glass-66959 I don’t think so
h

happy-kitchen-89482

01/19/2023, 8:37 PM
That may have added enough new files that it caused you to spill over into a second batch
s

sparse-lifeguard-95737

01/19/2023, 8:37 PM
the partitioning logic for
lint
was refactored internally in 2.15.x
so if you’re upgrading from 2.14.x that would explain the behavior difference to me
r

rhythmic-glass-66959

01/19/2023, 8:37 PM
👍
h

happy-kitchen-89482

01/19/2023, 8:37 PM
You could work around this by increasing the default batch size I suppose 🙂
r

rhythmic-glass-66959

01/19/2023, 8:39 PM
Trying right now...
Setting the
[lint].batch_size = 4096
seems to fix the problem.
Are there any potential issues running such a large batch_size? I'm far from the default (128)...
h

happy-kitchen-89482

01/19/2023, 9:24 PM
I don't foresee problems, it's a performance tradeoff
👍 1
r

rhythmic-glass-66959

01/19/2023, 9:54 PM
Oh, I can see that. With default batch size, the lint goal takes 26s to complete. With 4096, it takes 2m18s 😞
h

happy-kitchen-89482

01/19/2023, 11:31 PM
Hmm, I wouldn't have expected that big a discrepancy, interesting
r

rhythmic-glass-66959

01/19/2023, 11:42 PM
I think I had something in cache the first time. It takes ~1m now for both scenarios (128 and 4096).
Yeah, definitely something with the cache, ~10s now for both! 🤯
h

happy-kitchen-89482

01/20/2023, 1:41 AM
Excellent!