I think a weakref dict should work, but I haven’t completely thought it through