https://pantsbuild.org/ logo
f

flat-zoo-31952

05/17/2023, 3:04 PM
For the GIL bottleneck thing, what stops us from running multiple interpreters via the engine and collecting their results in the graph? I'm assuming someone has thought of this and there are good reasons we don't or can't do it, but I'm curious what they are
b

bitter-ability-32190

05/17/2023, 3:04 PM
That's an excellent question 🙂
The answer is the Python data is tied to the interpreter and there isn't a standard way for interpreters to share data
...yet
Otherwise we could maintain an interpreter pool and have the data float around (pun intended)
e

enough-analyst-54434

05/17/2023, 3:10 PM
There is, but sub-interpreters have only just got enough love to make them usable IIUC. And that's in ~3.12 (or maybe 3.11) and not sure if the PyO3 stuff supports yet.
b

bitter-ability-32190

05/17/2023, 3:11 PM
IIRC Sub-interpreters were always there, it was just in CPython. The recent changes opened it to the library. So presumably PyO3 has/had support already? It's presumption though...
e

enough-analyst-54434

05/17/2023, 3:12 PM
They were there for a long time. But I recently read about how they were ~unusable until roughly just now.
b

bitter-ability-32190

05/17/2023, 3:15 PM
Yeah Stu and I got to attend Eric Snow's PyCon talk on Sub-interpreters. He wants (and has working library/branches) data sharing
e

enough-analyst-54434

05/17/2023, 3:15 PM
Yeah. So maybe 3.13
b

bitter-ability-32190

05/17/2023, 3:16 PM
🤞
I still think GIL-less Python might be better for us
☝️ 1
f

flat-zoo-31952

05/17/2023, 4:12 PM
there isn't a standard way for interpreters to share data
Serialization? but then maybe the hit from deserializing in the main interpreter would offset any gains from un-gil-ing the execution of python code
a

ancient-vegetable-10556

05/17/2023, 4:12 PM
Yup.
w

witty-crayon-22786

05/17/2023, 4:14 PM
@flat-zoo-31952: fwiw, i’ve begun porting some of the more fundamental datastructures to Rust, with an aim to begin porting the meatier graph
@rules
to rust as well: https://github.com/pantsbuild/pants/pull/18989
👍🏻 1
f

flat-zoo-31952

05/17/2023, 4:14 PM
anyways, this seems to come back to "python embeds rust" vs "rust embeds python" ... which is a bigger thing
I've been working on GC as well, but i'm learning Rust still and my evenings are often interrupted by a child that doesn't like staying asleep
💜 4
b

bitter-ability-32190

05/17/2023, 4:15 PM
(You are in very good company with that, buddy)
f

flat-zoo-31952

05/17/2023, 4:16 PM
is there a support group for parents that want to do open source but can't because of time? 😅
👀 4
🤠 3
w

witty-crayon-22786

05/25/2023, 4:54 PM
next up: https://github.com/pantsbuild/pants/pull/19143 … as noted there, subclassing makes use of pyo3 slightly more awkward, but i’ll likely still work through
Target
before diving into
graph.py