#development
Title
b
bitter-ability-32190
04/19/2023, 6:50 PMLet's say someone added
RustPythonlocal_cacheCC @witty-crayon-22786
(and FYI @happy-kitchen-89482)
w
witty-crayon-22786
04/19/2023, 6:54 PMIf it is fast enough, which I suspect that it could be, then I don't think you have to worry about caching it at all.
b
bitter-ability-32190
04/19/2023, 6:54 PMOh right, I remember there being 3 options. I forgot that one
w
witty-crayon-22786
04/19/2023, 6:54 PMI also suspect that's the easiest distribution model
b
bitter-ability-32190
04/19/2023, 6:54 PMYeah building an exe seems time-cosnuming and space-wasteful
w
witty-crayon-22786
04/19/2023, 6:54 PMWell, it's 1 but without caching unless it looks necessary
w
wide-midnight-78598
04/19/2023, 7:03 PMHow hypothetical is this …… hmmmmmm…..?
b
bitter-ability-32190
04/19/2023, 7:03 PM👀
w
wide-midnight-78598
04/19/2023, 7:03 PMI was inspired by Charlie Marsh’s new company post - all Rust tooling all the time
b
bitter-ability-32190
04/19/2023, 7:04 PMHehe I'll be seeing him this weekend 🙂
w
wide-midnight-78598
04/19/2023, 7:05 PMAsk him if he’ll lend 5% of his brainpower to fixing the incredibly slow JS ecosystem. Prettier is great, but man - I could hand format faster sometimes
b
bitter-ability-32190
04/19/2023, 7:17 PMI'll see y'all after PyCOn
h
happy-kitchen-89482
04/19/2023, 9:43 PMKeep in mind that dep inference supports custom python code (e.g., for extracting django deps) so that would need to be considered.
I think there are other ways to speed up dep inference, such as batching, that might be better in this case
b
bitter-ability-32190
04/19/2023, 9:44 PMYeah that's why I CCd you
I think that can remain. But the generic inferer is now in rust outside of the process
There certainly are other ways to speed it up... Too
This also gets us using RustPython. Which we can use to speed up rule visiting 😊
➕ 1
CC @curved-television-6568 who had the same idea
h
happy-kitchen-89482
04/19/2023, 9:50 PMI don't think it's ideal to have two mechanisms for dep inference
Or dep extraction, to be precise
b
bitter-ability-32190
04/19/2023, 9:50 PMNo not ideal. But really really fast. 😈
w
witty-crayon-22786
04/19/2023, 9:50 PMThere are already multiple implementations of inference: it's generic in that regard.
b
bitter-ability-32190
04/19/2023, 9:50 PMTo be fair the existing customization points to the deo inference script could've just been unions to start with
h
happy-kitchen-89482
04/19/2023, 10:04 PMI think rewriting the custom stuff in rust is not that prohibitive either
w
witty-crayon-22786
04/19/2023, 10:06 PMOne thing to note though is that it would put a cap on supported python versions, unless you kept the old implementation as a fallback
h
happy-kitchen-89482
04/19/2023, 10:07 PMOh hmmm, true
I still think we want to invest in smarter batching and splitting
we happen to have a rust hack for dep inference, but that won't help with test running
b
bitter-ability-32190
04/19/2023, 10:11 PMCertainly!
And good point, the old stuff needs to support Python 2. But we can conditionally use that
Until we drop Py2 support
I'd also feel less bad if our plugins weren't baked into pants itself. Then a different binary for Python wouldn't be baked in if you didn't need it
Ok I'll see about taking this into PoC at the airport today
Copy code
08:34:56.08 [WARN] ParsedPythonDependencies(imports=ParsedPythonImports({'__future__.annotations': ParsedPythonImportInfo(lineno=4, weak=False), 'pants.util.frozendict.FrozenDict': ParsedPythonImportInfo(lineno=9, weak=False), 'dataclasses.dataclass': ParsedPythonImportInfo(lineno=6, weak=False), 'pants.engine.collection.DeduplicatedCollection': ParsedPythonImportInfo(lineno=8, weak=False)}), assets=ParsedPythonAssetPaths([]))