Has anyone put thought into using <multiple Python...
# development
e
Has anyone put thought into using multiple Python interpreter instances towards the end of having maybe 1 gil per core instead of 1 gil? I'm sure there are lots of devils in the details or maybe even obvious no-gos, but at a high level each rule execution (pure function) is already conceptually fully isolated and so the only requirement is that identity and equality can be made to work across the interpreters.
f
To take an alternate position, why use CPython at that point? I recall there are some alternate interpreters (PyPy?) without a GIL?
e
Yeah, perhaps that works too.
w
We definitely explored this at the beginning of the rule engine, but pickling costs were significant
...oh. but that's multiple processes, not multiple GILs.
Yea, haven't explored the changes in 3.9/3.10 to make that more realistic
...but even with multiple GILs in one process, you need to marshall objects, iirc
e
Right. We'd probably need `@rule`s to be cached like processes for that to work performantly, but then that would bring up all sorts of other issues including at least serialization format stability and probably some massive memory / disk blow up not to mention lack of support in remote CAS / action CAS.
w
yep. and that part we really did experiment with… it was a bottleneck, unfortunately. i came to a similar conclusion at the time: bundling a faster interpreter was possible and probably an easier win
…but that was also pre-rust: another path to faster Pants right now is identifying and porting high-CPU codepaths to Rust intrinsics
i’ve been looking at the dependency computation loop as the next likely candidate.