possible that you could just flag to exclude the dupes