https://pantsbuild.org/ logo
Join Slack
Powered by
<@U06A03HV1>: I am trying to get `test_compile_mul...
# development
f
@witty-crayon-22786: I am trying to get 
test_compile_multiple_source_files
in 
src/python/pants/backend/java/compile/javac_test.py
working with same package dependency inference. That test creates two files in the same package but they do not refer to each other’s types, so they should end up in separate `CoarsenedTarget`’s. However, it appears the shared dependency of the 
coursier_lockfile
is being found to link them together, and so they end up in a single CoarsenedTarget. Is this expected?
I may be missing something here. File-level subtargets are turned off unless they are coming from somewhere else.
debugging prints:
Copy code
dependency_mapping=_DependencyMapping(mapping=FrozenDict({Address(//ExampleLib.java:lib): (Address(//:lockfile), Address(//ExampleLib.java:lib), Address(//OtherLib.java:lib)), Address(//OtherLib.java:lib): (Address(//:lockfile), Address(//ExampleLib.java:lib), Address(//OtherLib.java:lib)), Address(//:lockfile): ()}), visited=FrozenOrderedSet([<class 'pants.jvm.target_types.JvmDependencyLockfile'>(address=//:lockfile, alias='coursier_lockfile', residence_dir='', description=None, source=coursier_resolve.lockfile, tags=None), <class 'pants.backend.java.target_types.JavaSourceTarget'>(address=//ExampleLib.java:lib, alias='java_source', residence_dir='', dependencies=(':lockfile',), description=None, source=ExampleLib.java, tags=None), <class 'pants.backend.java.target_types.JavaSourceTarget'>(address=//OtherLib.java:lib, alias='java_source', residence_dir='', dependencies=(':lockfile',), description=None, source=OtherLib.java, tags=None)]), roots_as_targets=UnexpandedTargets([<class 'pants.backend.java.target_types.JavaSourceTarget'>(address=//ExampleLib.java:lib, alias='java_source', residence_dir='', dependencies=(':lockfile',), description=None, source=ExampleLib.java, tags=None), <class 'pants.backend.java.target_types.JavaSourceTarget'>(address=//OtherLib.java:lib, alias='java_source', residence_dir='', dependencies=(':lockfile',), description=None, source=OtherLib.java, tags=None)]))
addresses_to_targets={Address(//:lockfile): <class 'pants.jvm.target_types.JvmDependencyLockfile'>(address=//:lockfile, alias='coursier_lockfile', residence_dir='', description=None, source=coursier_resolve.lockfile, tags=None),
 Address(//ExampleLib.java:lib): <class 'pants.backend.java.target_types.JavaSourceTarget'>(address=//ExampleLib.java:lib, alias='java_source', residence_dir='', dependencies=(':lockfile',), description=None, source=ExampleLib.java, tags=None),
 Address(//OtherLib.java:lib): <class 'pants.backend.java.target_types.JavaSourceTarget'>(address=//OtherLib.java:lib, alias='java_source', residence_dir='', dependencies=(':lockfile',), description=None, source=OtherLib.java, tags=None)}
components=[[Address(//:lockfile)],
 [Address(//OtherLib.java:lib), Address(//ExampleLib.java:lib)]]
root_addresses_set={Address(//ExampleLib.java:lib), Address(//OtherLib.java:lib)}
component=[Address(//:lockfile)]
component=[Address(//OtherLib.java:lib), Address(//ExampleLib.java:lib)]
w
i don’t see 
CoarsenedTargets
in there… what makes you think they are being coarsened together?
f
Copy code
component=[Address(//:lockfile)]
component=[Address(//OtherLib.java:lib), Address(//ExampleLib.java:lib)]
the print is in the loop that constructs the CoarsenedTargets
Copy code
diff --git a/src/python/pants/engine/internals/graph.py b/src/python/pants/engine/internals/graph.py
index a9f94adca..a31c8cfd9 100644
--- a/src/python/pants/engine/internals/graph.py
+++ b/src/python/pants/engine/internals/graph.py
@@ -413,6 +413,7 @@ async def transitive_targets(request: TransitiveTargetsRequest) -> TransitiveTar
 
 @rule
 async def coarsened_targets(addresses: Addresses) -> CoarsenedTargets:
+    from pprint import pformat
     dependency_mapping = await Get(
         _DependencyMapping,
         _DependencyMappingRequest(
@@ -424,9 +425,11 @@ async def coarsened_targets(addresses: Addresses) -> CoarsenedTargets:
             expanded_targets=False,
         ),
     )
+    print(f"dependency_mapping={pformat(dependency_mapping)}")
     addresses_to_targets = {
         t.address: t for t in [*dependency_mapping.visited, *dependency_mapping.roots_as_targets]
     }
+    print(f"addresses_to_targets={pformat(addresses_to_targets)}")
 
     # Because this is Tarjan's SCC (TODO: update signature to guarantee), components are returned
     # in reverse topological order. We can thus assume when building the structure shared
@@ -435,11 +438,14 @@ async def coarsened_targets(addresses: Addresses) -> CoarsenedTargets:
     components = native_engine.strongly_connected_components(
         list(dependency_mapping.mapping.items())
     )
+    print(f"components={pformat(components)}")
 
     coarsened_targets: dict[Address, CoarsenedTarget] = {}
     root_coarsened_targets = []
     root_addresses_set = set(addresses)
+    print(f"root_addresses_set={pformat(root_addresses_set)}")
     for component in components:
+        print(f"component={component}")
         component = sorted(component)
         component_set = set(component)
w
in a meeting, but will look more closely afterward
f
side note: seems weird that an address can be a dep of itself in `depending_mapping`: 
Address(//ExampleLib.java:lib): (Address(//:lockfile), Address(//ExampleLib.java:lib), Address(//OtherLib.java:lib))
w
as mentioned above:
Copy code
component=[Address(//:lockfile)]
component=[Address(//OtherLib.java:lib), Address(//ExampleLib.java:lib)]
…looks like two targets
f
those are the components coming from the Rust impl of the algorithm
the result is:
Copy code
coarsened_target[0]:
-  java_source(address="//ExampleLib.java:lib", dependencies=(':lockfile',), description=None, source=ExampleLib.java, tags=None)
-  java_source(address="//OtherLib.java:lib", dependencies=(':lockfile',), description=None, source=OtherLib.java, tags=None)
w
right, so those are two different targets.
i can’t explain why 
Address(//OtherLib.java:lib), Address(//ExampleLib.java:lib)
are in the same cycle: you’d have to look at the inferred deps
f
that’s the same CoarsenedTarget (I’m just printing out the members separately).
okay will dig further
w
yes. if they are in the same CoarsenedTarget, it’s because they form a cycle.
f
and yet dep inference doesn’t infer or inject any deps:
Copy code
resolve_dependencies(java_source(address="//OtherLib.java:lib", dependencies=(':lockfile',), description=None, source=OtherLib.java, tags=None))
  explicitly_provided=ExplicitlyProvidedDependencies(address=Address(//OtherLib.java:lib), includes=FrozenOrderedSet([Address(//:lockfile)]), ignores=FrozenOrderedSet())
  injected=()
  inferred=()
  generated_addresses=()
  special_cased=()
resolve_dependencies(java_source(address="//ExampleLib.java:lib", dependencies=(':lockfile',), description=None, source=ExampleLib.java, tags=None))
  explicitly_provided=ExplicitlyProvidedDependencies(address=Address(//ExampleLib.java:lib), includes=FrozenOrderedSet([Address(//:lockfile)]), ignores=FrozenOrderedSet())
  injected=()
  inferred=()
  generated_addresses=()
  special_cased=()
resolve_dependencies(coursier_lockfile(address="//:lockfile", description=None, source=coursier_resolve.lockfile, tags=None))
  explicitly_provided=ExplicitlyProvidedDependencies(address=Address(//:lockfile), includes=FrozenOrderedSet(), ignores=FrozenOrderedSet())
  injected=()
  inferred=()
  generated_addresses=()
  special_cased=()
(as expected)
w
the dependency_mapping from your first example is:
Copy code
mapping=FrozenDict({
        Address(//ExampleLib.java:lib): (Address(//:lockfile), Address(//ExampleLib.java:lib), Address(//OtherLib.java:lib)),
        Address(//OtherLib.java:lib): (Address(//:lockfile), Address(//ExampleLib.java:lib), Address(//OtherLib.java:lib)),
        Address(//:lockfile): ()}
    )
so both of those files do depend on one another
the lockfile doesn’t have anything to do with it though: it doesn’t have any dependencies of its own, so it can’t participate in any cycles.
f
but the logging from resolve_dependencies does not show a source of the dep
can deps be introduced elsewhere?
CoarsenedTargets is using _DependencyMappingRequest, will go look there
w
it’s just using resolve_dependencies under the hood.
you’re sure that the return value of 
resolve_dependencies
doesn’t include them?
you dumped the individual components, but maybe you missed one?
f
dumping the result does show the dep, wtf
grr my print statement for inferred had the wrong variable in it
found it
searching for candidate types for org.pantsbuild.example.lib.String ends up reducing it to the package 
org.pantsbuild.example.lib
as the algorithm strips off the parts of the FQT
so need to looks for an exact match only instead of using FirstPartyJavaPackageMapping.addresses_for_symbol
thanks for playing along with my debugging journey <sigh>
❤️ 2
Open in Slack
PreviousNext
Powered by