Edit the 1st file like so:

$ git diff
diff --git a/src/python/pants/engine/collection.py b/src/python/pants/engine/collection.py
index c340fa2e6..9f3bb81c2 100644
--- a/src/python/pants/engine/collection.py
+++ b/src/python/pants/engine/collection.py
@@ -52,7 +52,8 @@ class Collection(Tuple[T, ...]):
     # Unlike in Python 2 we must explicitly implement __hash__ since we explicitly implement __eq__
     # per the Python 3 data model.
     # See: <https://docs.python.org/3/reference/datamodel.html#object.__hash__>
-    __hash__ = Tuple.__hash__
+    def __hash__(self) -> int:
+        return super().__hash__()

     def __repr__(self) -> str:
         return f"{self.__class__.__name__}({list(self)})"

And then horror:

$ pants check src/python/pants/engine::
11:46:15.08 [ERROR] 1 Exception encountered:

Engine traceback:
  in select
    ..
  in pants.core.util_rules.environments.determine_local_environment
    ..
  in pants.core.util_rules.environments.determine_all_environments
    ..

Traceback (most recent call last):
  File "/home/jsirois/dev/pantsbuild/jsirois-pants/src/python/pants/util/frozendict.py", line 91, in _calculate_hash
    return hash(tuple(self._data.items()))
  File "/home/jsirois/dev/pantsbuild/jsirois-pants/src/python/pants/engine/collection.py", line 56, in __hash__
    super().__hash__()
  File "<string>", line 3, in __hash__
TypeError: unhashable type: 'dict'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/jsirois/dev/pantsbuild/jsirois-pants/src/python/pants/engine/internals/selectors.py", line 623, in native_engine_generator_send
    res = rule.send(arg) if err is None else rule.throw(throw or err)
  File "/home/jsirois/dev/pantsbuild/jsirois-pants/src/python/pants/core/util_rules/environments.py", line 592, in determine_all_environments
    Get(EnvironmentTarget, EnvironmentName(name)) for name in environments_subsystem.names
  File "/home/jsirois/dev/pantsbuild/jsirois-pants/src/python/pants/engine/internals/selectors.py", line 358, in MultiGet
    return await _MultiGet(tuple(__arg0))
  File "/home/jsirois/dev/pantsbuild/jsirois-pants/src/python/pants/engine/internals/selectors.py", line 165, in __await__
    result = yield self.gets
  File "/home/jsirois/dev/pantsbuild/jsirois-pants/src/python/pants/engine/internals/selectors.py", line 623, in native_engine_generator_send
    res = rule.send(arg) if err is None else rule.throw(throw or err)
  File "/home/jsirois/dev/pantsbuild/jsirois-pants/src/python/pants/core/util_rules/environments.py", line 827, in get_target_for_environment_name
    WrappedTargetRequest(address, description_of_origin=_description_of_origin),
  File "/home/jsirois/dev/pantsbuild/jsirois-pants/src/python/pants/engine/internals/selectors.py", line 118, in __await__
    result = yield self
  File "/home/jsirois/dev/pantsbuild/jsirois-pants/src/python/pants/engine/internals/selectors.py", line 623, in native_engine_generator_send
    res = rule.send(arg) if err is None else rule.throw(throw or err)
  File "/home/jsirois/dev/pantsbuild/jsirois-pants/src/python/pants/engine/internals/graph.py", line 401, in resolve_target_for_bootstrapping
    description_of_origin=request.description_of_origin,
  File "/home/jsirois/dev/pantsbuild/jsirois-pants/src/python/pants/engine/internals/graph.py", line 180, in _determine_target_adaptor_and_type
    TargetAdaptorRequest(address, description_of_origin=description_of_origin),
  File "/home/jsirois/dev/pantsbuild/jsirois-pants/src/python/pants/engine/internals/selectors.py", line 118, in __await__
    result = yield self
  File "/home/jsirois/dev/pantsbuild/jsirois-pants/src/python/pants/engine/internals/selectors.py", line 623, in native_engine_generator_send
    res = rule.send(arg) if err is None else rule.throw(throw or err)
  File "/home/jsirois/dev/pantsbuild/jsirois-pants/src/python/pants/engine/internals/build_files.py", line 395, in find_target_adaptor
    address_family = await Get(AddressFamily, AddressFamilyDir(address.spec_path))
  File "/home/jsirois/dev/pantsbuild/jsirois-pants/src/python/pants/engine/internals/selectors.py", line 118, in __await__
    result = yield self
  File "/home/jsirois/dev/pantsbuild/jsirois-pants/src/python/pants/engine/internals/selectors.py", line 623, in native_engine_generator_send
    res = rule.send(arg) if err is None else rule.throw(throw or err)
  File "/home/jsirois/dev/pantsbuild/jsirois-pants/src/python/pants/engine/internals/build_files.py", line 395, in find_target_adaptor
    address_family = await Get(AddressFamily, AddressFamilyDir(address.spec_path))
  File "/home/jsirois/dev/pantsbuild/jsirois-pants/src/python/pants/engine/internals/selectors.py", line 118, in __await__
    result = yield self
  File "/home/jsirois/dev/pantsbuild/jsirois-pants/src/python/pants/engine/internals/selectors.py", line 623, in native_engine_generator_send
    res = rule.send(arg) if err is None else rule.throw(throw or err)
  File "/home/jsirois/dev/pantsbuild/jsirois-pants/src/python/pants/engine/internals/build_files.py", line 260, in parse_address_family
    Get(SyntheticAddressMaps, SyntheticAddressMapsRequest(directory.path)),
  File "/home/jsirois/dev/pantsbuild/jsirois-pants/src/python/pants/engine/internals/selectors.py", line 509, in MultiGet
    return await _MultiGet((__arg0, __arg1))
  File "/home/jsirois/dev/pantsbuild/jsirois-pants/src/python/pants/engine/internals/selectors.py", line 165, in __await__
    result = yield self.gets
  File "/home/jsirois/dev/pantsbuild/jsirois-pants/src/python/pants/engine/internals/selectors.py", line 623, in native_engine_generator_send
    res = rule.send(arg) if err is None else rule.throw(throw or err)
  File "/home/jsirois/dev/pantsbuild/jsirois-pants/src/python/pants/engine/internals/synthetic_targets.py", line 314, in all_synthetic_targets
    spec_paths=set(itertools.chain.from_iterable(all_spec_paths)),
  File "/home/jsirois/dev/pantsbuild/jsirois-pants/src/python/pants/engine/internals/synthetic_targets.py", line 255, in create
    sorted(address_maps, key=address_map_key), key=address_map_key
  File "/home/jsirois/dev/pantsbuild/jsirois-pants/src/python/pants/util/frozendict.py", line 49, in __init__
    self._hash = self._calculate_hash()
  File "/home/jsirois/dev/pantsbuild/jsirois-pants/src/python/pants/util/frozendict.py", line 104, in _calculate_hash
    """
TypeError: Even though you are using a `FrozenDict`, the underlying values are not hashable. Please use hashable (and preferably immutable) types for the underlying values, e.g. use tuples instead of lists and use FrozenOrderedSet instead of set().
...

Soo it seems: 1. Somehow CI has been skipping this for a good long while. 2. The

__hash__

override we though we had - with a big comment as to why it was important - we apparently didn't have?

Yeah, this is a bit of a rabbit hole. If you naively freeze up the backtrace objects you eventually hit

sources=[...]

BUILD file arguments and that's a whole can of worms mapping lists to tuples and dicts to FrozenDicts recursively. It may just be that a less naive hash would work over some proxy. I'll file an issue for someone more tuned-in here to take a look at.

Ok, the MyPy check failing locally but not in CI appears to be a Heisenbug. I no longer see MyPy failures today. So ... no clue, but bug for another day I guess. The bug it revealed though is real. We currently hash every pants.engine.collection.Collection to the same value and ditto pants.engine.target.CoarsenedTarget. Fixing that leads to more bad as described above - a bunch of unhashable collected things - at least in CoarsenedTarget: https://github.com/pantsbuild/pants/issues/18250 I'll turn this over to others.