would be ideal at a minimum to optimally format and dedupe there