Oh wow, using the mutex trick that @witty-crayon-22786 mentioned fixed all of the issues that I observed