It seems like this isn't possible on the scalac side of things, but I would be happy with letting warnings be non-fatal so that pants can inspect the warnings and make them fatal based on configuration