https://pantsbuild.org/ logo
e

enough-analyst-54434

02/12/2023, 7:46 PM
Pants main, 1st reaction a bit of horror. Has anyone seen this?:
Copy code
$ pants check src/python/pants/engine::
11:45:13.52 [ERROR] Completed: Typecheck using MyPy - mypy - mypy failed (exit code 1).
src/python/pants/engine/collection.py: note: In class "Collection":
src/python/pants/engine/collection.py:55:16: error: Incompatible types in assignment (expression has type "Callable[[], int]", base class "object" defined the type as "Callable[[object], int]")  [assignment]
        __hash__ = Tuple.__hash__
                   ^
src/python/pants/engine/target.py: note: In class "CoarsenedTargets":
src/python/pants/engine/target.py:917:16: error: Incompatible types in assignment (expression has type "Callable[[], int]", base class "object" defined the type as "Callable[[object], int]")  [assignment]
        __hash__ = Tuple.__hash__
                   ^
Found 2 errors in 2 files (checked 69 source files)



āœ• mypy failed.
Edit the 1st file like so:
Copy code
$ git diff
diff --git a/src/python/pants/engine/collection.py b/src/python/pants/engine/collection.py
index c340fa2e6..9f3bb81c2 100644
--- a/src/python/pants/engine/collection.py
+++ b/src/python/pants/engine/collection.py
@@ -52,7 +52,8 @@ class Collection(Tuple[T, ...]):
     # Unlike in Python 2 we must explicitly implement __hash__ since we explicitly implement __eq__
     # per the Python 3 data model.
     # See: <https://docs.python.org/3/reference/datamodel.html#object.__hash__>
-    __hash__ = Tuple.__hash__
+    def __hash__(self) -> int:
+        return super().__hash__()

     def __repr__(self) -> str:
         return f"{self.__class__.__name__}({list(self)})"
And then horror:
Copy code
$ pants check src/python/pants/engine::
11:46:15.08 [ERROR] 1 Exception encountered:

Engine traceback:
  in select
    ..
  in pants.core.util_rules.environments.determine_local_environment
    ..
  in pants.core.util_rules.environments.determine_all_environments
    ..

Traceback (most recent call last):
  File "/home/jsirois/dev/pantsbuild/jsirois-pants/src/python/pants/util/frozendict.py", line 91, in _calculate_hash
    return hash(tuple(self._data.items()))
  File "/home/jsirois/dev/pantsbuild/jsirois-pants/src/python/pants/engine/collection.py", line 56, in __hash__
    super().__hash__()
  File "<string>", line 3, in __hash__
TypeError: unhashable type: 'dict'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/jsirois/dev/pantsbuild/jsirois-pants/src/python/pants/engine/internals/selectors.py", line 623, in native_engine_generator_send
    res = rule.send(arg) if err is None else rule.throw(throw or err)
  File "/home/jsirois/dev/pantsbuild/jsirois-pants/src/python/pants/core/util_rules/environments.py", line 592, in determine_all_environments
    Get(EnvironmentTarget, EnvironmentName(name)) for name in environments_subsystem.names
  File "/home/jsirois/dev/pantsbuild/jsirois-pants/src/python/pants/engine/internals/selectors.py", line 358, in MultiGet
    return await _MultiGet(tuple(__arg0))
  File "/home/jsirois/dev/pantsbuild/jsirois-pants/src/python/pants/engine/internals/selectors.py", line 165, in __await__
    result = yield self.gets
  File "/home/jsirois/dev/pantsbuild/jsirois-pants/src/python/pants/engine/internals/selectors.py", line 623, in native_engine_generator_send
    res = rule.send(arg) if err is None else rule.throw(throw or err)
  File "/home/jsirois/dev/pantsbuild/jsirois-pants/src/python/pants/core/util_rules/environments.py", line 827, in get_target_for_environment_name
    WrappedTargetRequest(address, description_of_origin=_description_of_origin),
  File "/home/jsirois/dev/pantsbuild/jsirois-pants/src/python/pants/engine/internals/selectors.py", line 118, in __await__
    result = yield self
  File "/home/jsirois/dev/pantsbuild/jsirois-pants/src/python/pants/engine/internals/selectors.py", line 623, in native_engine_generator_send
    res = rule.send(arg) if err is None else rule.throw(throw or err)
  File "/home/jsirois/dev/pantsbuild/jsirois-pants/src/python/pants/engine/internals/graph.py", line 401, in resolve_target_for_bootstrapping
    description_of_origin=request.description_of_origin,
  File "/home/jsirois/dev/pantsbuild/jsirois-pants/src/python/pants/engine/internals/graph.py", line 180, in _determine_target_adaptor_and_type
    TargetAdaptorRequest(address, description_of_origin=description_of_origin),
  File "/home/jsirois/dev/pantsbuild/jsirois-pants/src/python/pants/engine/internals/selectors.py", line 118, in __await__
    result = yield self
  File "/home/jsirois/dev/pantsbuild/jsirois-pants/src/python/pants/engine/internals/selectors.py", line 623, in native_engine_generator_send
    res = rule.send(arg) if err is None else rule.throw(throw or err)
  File "/home/jsirois/dev/pantsbuild/jsirois-pants/src/python/pants/engine/internals/build_files.py", line 395, in find_target_adaptor
    address_family = await Get(AddressFamily, AddressFamilyDir(address.spec_path))
  File "/home/jsirois/dev/pantsbuild/jsirois-pants/src/python/pants/engine/internals/selectors.py", line 118, in __await__
    result = yield self
  File "/home/jsirois/dev/pantsbuild/jsirois-pants/src/python/pants/engine/internals/selectors.py", line 623, in native_engine_generator_send
    res = rule.send(arg) if err is None else rule.throw(throw or err)
  File "/home/jsirois/dev/pantsbuild/jsirois-pants/src/python/pants/engine/internals/build_files.py", line 395, in find_target_adaptor
    address_family = await Get(AddressFamily, AddressFamilyDir(address.spec_path))
  File "/home/jsirois/dev/pantsbuild/jsirois-pants/src/python/pants/engine/internals/selectors.py", line 118, in __await__
    result = yield self
  File "/home/jsirois/dev/pantsbuild/jsirois-pants/src/python/pants/engine/internals/selectors.py", line 623, in native_engine_generator_send
    res = rule.send(arg) if err is None else rule.throw(throw or err)
  File "/home/jsirois/dev/pantsbuild/jsirois-pants/src/python/pants/engine/internals/build_files.py", line 260, in parse_address_family
    Get(SyntheticAddressMaps, SyntheticAddressMapsRequest(directory.path)),
  File "/home/jsirois/dev/pantsbuild/jsirois-pants/src/python/pants/engine/internals/selectors.py", line 509, in MultiGet
    return await _MultiGet((__arg0, __arg1))
  File "/home/jsirois/dev/pantsbuild/jsirois-pants/src/python/pants/engine/internals/selectors.py", line 165, in __await__
    result = yield self.gets
  File "/home/jsirois/dev/pantsbuild/jsirois-pants/src/python/pants/engine/internals/selectors.py", line 623, in native_engine_generator_send
    res = rule.send(arg) if err is None else rule.throw(throw or err)
  File "/home/jsirois/dev/pantsbuild/jsirois-pants/src/python/pants/engine/internals/synthetic_targets.py", line 314, in all_synthetic_targets
    spec_paths=set(itertools.chain.from_iterable(all_spec_paths)),
  File "/home/jsirois/dev/pantsbuild/jsirois-pants/src/python/pants/engine/internals/synthetic_targets.py", line 255, in create
    sorted(address_maps, key=address_map_key), key=address_map_key
  File "/home/jsirois/dev/pantsbuild/jsirois-pants/src/python/pants/util/frozendict.py", line 49, in __init__
    self._hash = self._calculate_hash()
  File "/home/jsirois/dev/pantsbuild/jsirois-pants/src/python/pants/util/frozendict.py", line 104, in _calculate_hash
    """
TypeError: Even though you are using a `FrozenDict`, the underlying values are not hashable. Please use hashable (and preferably immutable) types for the underlying values, e.g. use tuples instead of lists and use FrozenOrderedSet instead of set().
...
Soo it seems: 1. Somehow CI has been skipping this for a good long while. 2. The
__hash__
override we though we had - with a big comment as to why it was important - we apparently didn't have?
w

worried-painter-31382

02/12/2023, 7:53 PM
Just fyi for the
__hash__
the diff is missing a return statement šŸ˜¬
e

enough-analyst-54434

02/12/2023, 8:02 PM
Yeah - added it, but - of course - same issue. The return was/would've never got hit in the backtrace.
šŸ‘ 1
Yeah, this is a bit of a rabbit hole. If you naively freeze up the backtrace objects you eventually hit
sources=[...]
BUILD file arguments and that's a whole can of worms mapping lists to tuples and dicts to FrozenDicts recursively. It may just be that a less naive hash would work over some proxy. I'll file an issue for someone more tuned-in here to take a look at.
Ok, the MyPy check failing locally but not in CI appears to be a Heisenbug. I no longer see MyPy failures today. So ... no clue, but bug for another day I guess. The bug it revealed though is real. We currently hash every pants.engine.collection.Collection to the same value and ditto pants.engine.target.CoarsenedTarget. Fixing that leads to more bad as described above - a bunch of unhashable collected things - at least in CoarsenedTarget: https://github.com/pantsbuild/pants/issues/18250 I'll turn this over to others.
h

happy-kitchen-89482

02/14/2023, 10:24 PM
Yikes!
5 Views