As long as we do not descend into directories that match the spec_exclude, but additionally happen to match files that are in this set, I think this is feasible