FYI <@U06A03HV1> <@U01D2SF5DU0> I just saw <Issue ...
# developmentb
bitter-ability-32190
10/21/2021, 5:30 PMFYI @witty-crayon-22786 @chilly-magazine-21545 I just saw Issue 10864: Improve MyPy performance which talks about MyPy's persistent cache and challenges because of it.
I'm volunteering some personal time to implement something similar for
astroidpylintpants👀 1
w
witty-crayon-22786
10/21/2021, 5:43 PMnice!
witty-crayon-22786
10/21/2021, 5:44 PMwe’ve come a long way on deciding how much flexibility to allow for caches, so i expect that we can make something work there.
witty-crayon-22786
10/21/2021, 5:45 PMeven without its caches, mypy is a lot faster than pylint though, so it hasn’t been as much of a priority
b
bitter-ability-32190
10/21/2021, 5:46 PMI currently have a very ugly WIP PR which handles the transformation from Python object to JSON: https://github.com/PyCQA/astroid/pull/1194
No code for the actual persisting logic, but that's easy enough to write (I have it offline) and it'll likely be in a follow-up PR
bitter-ability-32190
10/21/2021, 5:47 PMI suppose that's what happens when you have Guido-level visibility/development on a project 😂
w
witty-crayon-22786
10/21/2021, 5:47 PM😃
witty-crayon-22786
10/21/2021, 5:48 PMthe big question around mypy (and probably pylint) is whether the cache needs to be per-repository, or global. @enough-analyst-54434’s comment at the end of the thread about enabling the cache being global would make things easier for us.
b
bitter-ability-32190
10/21/2021, 5:48 PMAlso FWIW
astroidmypybitter-ability-32190
10/21/2021, 5:50 PMIIUC the cache key is absolute paths, so "global"
w
witty-crayon-22786
10/21/2021, 5:55 PMyea. the issue (which I had mostly forgotten: quite a thread) is that Pants runs things in sandboxes
witty-crayon-22786
10/21/2021, 5:55 PMso the absolute path would end up being a sandbox path, and would then need fixing.
witty-crayon-22786
10/21/2021, 5:56 PMJohn’s comment about switching to digests avoids that issue. but unclear how much work it would be upstream.
b
bitter-ability-32190
10/21/2021, 5:56 PMI'd be surprised if it became that much work, honestly.... but I've been surprised before 🙂
bitter-ability-32190
10/21/2021, 5:57 PMDo you have a doc page on Pant's sandboxing? I'm familiar with Bazel's but not Pants'
w
witty-crayon-22786
10/21/2021, 5:58 PMthe actual implementation isn’t very well described, but the public API is: https://www.pantsbuild.org/docs/rules-api-process
witty-crayon-22786
10/21/2021, 5:59 PMinputs and outputs are specified using digests (with the same underlying datastructures as for Bazel, because Pants supports remote execution with the REAPI)
b
bitter-ability-32190
10/21/2021, 6:05 PMAre the paths inside the temp folder copied or linked? Could the links be resolved by the underlying app (`mypy`/`pylint`)?
w
witty-crayon-22786
10/21/2021, 6:05 PMcopied
b
bitter-ability-32190
10/21/2021, 6:13 PMAh, yes. Digest would be a hard requirement then
w
witty-crayon-22786
10/21/2021, 6:24 PMi expect that mypy’s sqlite schema could be adapted pretty easily (behind a flag even)… and yea, digest keyed would probably be a good first place to go for pylint
witty-crayon-22786
10/21/2021, 6:26 PMone challenge that i haven’t thought through is how dependencies are accounted for in that cache: it’s possible that mypy is relying on timestamps to bump cache entries when the dependencies have changed (despite their direct content being identical). and that complicates things cc @enough-analyst-54434
b
bitter-ability-32190
10/21/2021, 6:48 PMI believe that is true
bitter-ability-32190
10/21/2021, 6:49 PMFWIW I chose path+mtime (and I suspect mypy did as well) is because that's what Python uses for pyc files