Has anyone put thought into using multiple Python interpreter instances towards the end of having maybe 1 gil per core instead of 1 gil? I'm sure there are lots of devils in the details or maybe even obvious no-gos, but at a high level each rule execution (pure function) is already conceptually fully isolated and so the only requirement is that identity and equality can be made to work across the interpreters.
11/09/2022, 1:04 AM
To take an alternate position, why use CPython at that point? I recall there are some alternate interpreters (PyPy?) without a GIL?
11/09/2022, 1:12 AM
Yeah, perhaps that works too.
11/09/2022, 3:29 AM
We definitely explored this at the beginning of the rule engine, but pickling costs were significant
...oh. but that's multiple processes, not multiple GILs.
Yea, haven't explored the changes in 3.9/3.10 to make that more realistic
...but even with multiple GILs in one process, you need to marshall objects, iirc
11/09/2022, 6:15 PM
Right. We'd probably need `@rule`s to be cached like processes for that to work performantly, but then that would bring up all sorts of other issues including at least serialization format stability and probably some massive memory / disk blow up not to mention lack of support in remote CAS / action CAS.
11/09/2022, 6:23 PM
yep. and that part we really did experiment with… it was a bottleneck, unfortunately. i came to a similar conclusion at the time: bundling a faster interpreter was possible and probably an easier win
…but that was also pre-rust: another path to faster Pants right now is identifying and porting high-CPU codepaths to Rust intrinsics