For the GIL bottleneck thing, what stops us from r...
# developmentf
flat-zoo-31952
05/17/2023, 3:04 PMFor the GIL bottleneck thing, what stops us from running multiple interpreters via the engine and collecting their results in the graph? I'm assuming someone has thought of this and there are good reasons we don't or can't do it, but I'm curious what they are
b
bitter-ability-32190
05/17/2023, 3:04 PMThat's an excellent question 🙂
bitter-ability-32190
05/17/2023, 3:05 PMThe answer is the Python data is tied to the interpreter and there isn't a standard way for interpreters to share data
bitter-ability-32190
05/17/2023, 3:05 PM...yet
bitter-ability-32190
05/17/2023, 3:06 PMOtherwise we could maintain an interpreter pool and have the data float around (pun intended)
e
enough-analyst-54434
05/17/2023, 3:10 PMThere is, but sub-interpreters have only just got enough love to make them usable IIUC. And that's in ~3.12 (or maybe 3.11) and not sure if the PyO3 stuff supports yet.
b
bitter-ability-32190
05/17/2023, 3:11 PMIIRC Sub-interpreters were always there, it was just in CPython. The recent changes opened it to the library.
So presumably PyO3 has/had support already? It's presumption though...
e
enough-analyst-54434
05/17/2023, 3:12 PMThey were there for a long time. But I recently read about how they were ~unusable until roughly just now.
enough-analyst-54434
05/17/2023, 3:14 PMI'm pretty sure I found this somewhere else, but: https://developers.slashdot.org/story/23/05/06/2356202/faster-leaner-python-312-released-today-with-improvements-to-speed-multiprocessing
b
bitter-ability-32190
05/17/2023, 3:15 PMYeah Stu and I got to attend Eric Snow's PyCon talk on Sub-interpreters.
He wants (and has working library/branches) data sharing
e
enough-analyst-54434
05/17/2023, 3:15 PMYeah. So maybe 3.13
b
bitter-ability-32190
05/17/2023, 3:16 PM🤞
bitter-ability-32190
05/17/2023, 3:16 PMI still think GIL-less Python might be better for us
☝️ 1
f
flat-zoo-31952
05/17/2023, 4:12 PMthere isn't a standard way for interpreters to share dataSerialization? but then maybe the hit from deserializing in the main interpreter would offset any gains from un-gil-ing the execution of python code
a
ancient-vegetable-10556
05/17/2023, 4:12 PMYup.
w
witty-crayon-22786
05/17/2023, 4:14 PM@flat-zoo-31952: fwiw, i’ve begun porting some of the more fundamental datastructures to Rust, with an aim to begin porting the meatier graph
@rules👍🏻 1
f
flat-zoo-31952
05/17/2023, 4:14 PManyways, this seems to come back to "python embeds rust" vs "rust embeds python" ... which is a bigger thing
flat-zoo-31952
05/17/2023, 4:15 PMI've been working on GC as well, but i'm learning Rust still and my evenings are often interrupted by a child that doesn't like staying asleep
💜 4
b
bitter-ability-32190
05/17/2023, 4:15 PM(You are in very good company with that, buddy)
f
flat-zoo-31952
05/17/2023, 4:16 PMis there a support group for parents that want to do open source but can't because of time? 😅
👀 4
🤠 3
w
witty-crayon-22786
05/25/2023, 4:54 PMnext up: https://github.com/pantsbuild/pants/pull/19143 … as noted there, subclassing makes use of pyo3 slightly more awkward, but i’ll likely still work through
Targetgraph.py