https://pantsbuild.org/ logo
Powered by
f

fast-nail-55400

10/20/2021, 11:10 PM
@witty-crayon-22786: I am trying to get 
test_compile_multiple_source_files
in 
src/python/pants/backend/java/compile/javac_test.py
working with same package dependency inference. That test creates two files in the same package but they do not refer to each other’s types, so they should end up in separate `CoarsenedTarget`’s. However, it appears the shared dependency of the 
coursier_lockfile
is being found to link them together, and so they end up in a single CoarsenedTarget. Is this expected?
I may be missing something here. File-level subtargets are turned off unless they are coming from somewhere else.
debugging prints:
Copy code
dependency_mapping=_DependencyMapping(mapping=FrozenDict({Address(//ExampleLib.java:lib): (Address(//:lockfile), Address(//ExampleLib.java:lib), Address(//OtherLib.java:lib)), Address(//OtherLib.java:lib): (Address(//:lockfile), Address(//ExampleLib.java:lib), Address(//OtherLib.java:lib)), Address(//:lockfile): ()}), visited=FrozenOrderedSet([<class 'pants.jvm.target_types.JvmDependencyLockfile'>(address=//:lockfile, alias='coursier_lockfile', residence_dir='', description=None, source=coursier_resolve.lockfile, tags=None), <class 'pants.backend.java.target_types.JavaSourceTarget'>(address=//ExampleLib.java:lib, alias='java_source', residence_dir='', dependencies=(':lockfile',), description=None, source=ExampleLib.java, tags=None), <class 'pants.backend.java.target_types.JavaSourceTarget'>(address=//OtherLib.java:lib, alias='java_source', residence_dir='', dependencies=(':lockfile',), description=None, source=OtherLib.java, tags=None)]), roots_as_targets=UnexpandedTargets([<class 'pants.backend.java.target_types.JavaSourceTarget'>(address=//ExampleLib.java:lib, alias='java_source', residence_dir='', dependencies=(':lockfile',), description=None, source=ExampleLib.java, tags=None), <class 'pants.backend.java.target_types.JavaSourceTarget'>(address=//OtherLib.java:lib, alias='java_source', residence_dir='', dependencies=(':lockfile',), description=None, source=OtherLib.java, tags=None)]))
addresses_to_targets={Address(//:lockfile): <class 'pants.jvm.target_types.JvmDependencyLockfile'>(address=//:lockfile, alias='coursier_lockfile', residence_dir='', description=None, source=coursier_resolve.lockfile, tags=None),
 Address(//ExampleLib.java:lib): <class 'pants.backend.java.target_types.JavaSourceTarget'>(address=//ExampleLib.java:lib, alias='java_source', residence_dir='', dependencies=(':lockfile',), description=None, source=ExampleLib.java, tags=None),
 Address(//OtherLib.java:lib): <class 'pants.backend.java.target_types.JavaSourceTarget'>(address=//OtherLib.java:lib, alias='java_source', residence_dir='', dependencies=(':lockfile',), description=None, source=OtherLib.java, tags=None)}
components=[[Address(//:lockfile)],
 [Address(//OtherLib.java:lib), Address(//ExampleLib.java:lib)]]
root_addresses_set={Address(//ExampleLib.java:lib), Address(//OtherLib.java:lib)}
component=[Address(//:lockfile)]
component=[Address(//OtherLib.java:lib), Address(//ExampleLib.java:lib)]
w

witty-crayon-22786

10/20/2021, 11:15 PM
i don’t see 
CoarsenedTargets
in there… what makes you think they are being coarsened together?
f

fast-nail-55400

10/20/2021, 11:16 PM
Copy code
component=[Address(//:lockfile)]
component=[Address(//OtherLib.java:lib), Address(//ExampleLib.java:lib)]
the print is in the loop that constructs the CoarsenedTargets
Copy code
diff --git a/src/python/pants/engine/internals/graph.py b/src/python/pants/engine/internals/graph.py
index a9f94adca..a31c8cfd9 100644
--- a/src/python/pants/engine/internals/graph.py
+++ b/src/python/pants/engine/internals/graph.py
@@ -413,6 +413,7 @@ async def transitive_targets(request: TransitiveTargetsRequest) -> TransitiveTar
 
 @rule
 async def coarsened_targets(addresses: Addresses) -> CoarsenedTargets:
+    from pprint import pformat
     dependency_mapping = await Get(
         _DependencyMapping,
         _DependencyMappingRequest(
@@ -424,9 +425,11 @@ async def coarsened_targets(addresses: Addresses) -> CoarsenedTargets:
             expanded_targets=False,
         ),
     )
+    print(f"dependency_mapping={pformat(dependency_mapping)}")
     addresses_to_targets = {
         t.address: t for t in [*dependency_mapping.visited, *dependency_mapping.roots_as_targets]
     }
+    print(f"addresses_to_targets={pformat(addresses_to_targets)}")
 
     # Because this is Tarjan's SCC (TODO: update signature to guarantee), components are returned
     # in reverse topological order. We can thus assume when building the structure shared
@@ -435,11 +438,14 @@ async def coarsened_targets(addresses: Addresses) -> CoarsenedTargets:
     components = native_engine.strongly_connected_components(
         list(dependency_mapping.mapping.items())
     )
+    print(f"components={pformat(components)}")
 
     coarsened_targets: dict[Address, CoarsenedTarget] = {}
     root_coarsened_targets = []
     root_addresses_set = set(addresses)
+    print(f"root_addresses_set={pformat(root_addresses_set)}")
     for component in components:
+        print(f"component={component}")
         component = sorted(component)
         component_set = set(component)
w

witty-crayon-22786

10/20/2021, 11:17 PM
in a meeting, but will look more closely afterward
f

fast-nail-55400

10/20/2021, 11:18 PM
side note: seems weird that an address can be a dep of itself in `depending_mapping`: 
Address(//ExampleLib.java:lib): (Address(//:lockfile), Address(//ExampleLib.java:lib), Address(//OtherLib.java:lib))
w

witty-crayon-22786

10/20/2021, 11:22 PM
as mentioned above:
Copy code
component=[Address(//:lockfile)]
component=[Address(//OtherLib.java:lib), Address(//ExampleLib.java:lib)]
…looks like two targets
f

fast-nail-55400

10/20/2021, 11:23 PM
those are the components coming from the Rust impl of the algorithm
the result is:
Copy code
coarsened_target[0]:
-  java_source(address="//ExampleLib.java:lib", dependencies=(':lockfile',), description=None, source=ExampleLib.java, tags=None)
-  java_source(address="//OtherLib.java:lib", dependencies=(':lockfile',), description=None, source=OtherLib.java, tags=None)
w

witty-crayon-22786

10/20/2021, 11:24 PM
right, so those are two different targets.
i can’t explain why 
Address(//OtherLib.java:lib), Address(//ExampleLib.java:lib)
are in the same cycle: you’d have to look at the inferred deps
f

fast-nail-55400

10/20/2021, 11:27 PM
that’s the same CoarsenedTarget (I’m just printing out the members separately).
okay will dig further
w

witty-crayon-22786

10/20/2021, 11:27 PM
yes. if they are in the same CoarsenedTarget, it’s because they form a cycle.
f

fast-nail-55400

10/20/2021, 11:40 PM
and yet dep inference doesn’t infer or inject any deps:
Copy code
resolve_dependencies(java_source(address="//OtherLib.java:lib", dependencies=(':lockfile',), description=None, source=OtherLib.java, tags=None))
  explicitly_provided=ExplicitlyProvidedDependencies(address=Address(//OtherLib.java:lib), includes=FrozenOrderedSet([Address(//:lockfile)]), ignores=FrozenOrderedSet())
  injected=()
  inferred=()
  generated_addresses=()
  special_cased=()
resolve_dependencies(java_source(address="//ExampleLib.java:lib", dependencies=(':lockfile',), description=None, source=ExampleLib.java, tags=None))
  explicitly_provided=ExplicitlyProvidedDependencies(address=Address(//ExampleLib.java:lib), includes=FrozenOrderedSet([Address(//:lockfile)]), ignores=FrozenOrderedSet())
  injected=()
  inferred=()
  generated_addresses=()
  special_cased=()
resolve_dependencies(coursier_lockfile(address="//:lockfile", description=None, source=coursier_resolve.lockfile, tags=None))
  explicitly_provided=ExplicitlyProvidedDependencies(address=Address(//:lockfile), includes=FrozenOrderedSet(), ignores=FrozenOrderedSet())
  injected=()
  inferred=()
  generated_addresses=()
  special_cased=()
(as expected)
w

witty-crayon-22786

10/20/2021, 11:43 PM
the dependency_mapping from your first example is:
Copy code
mapping=FrozenDict({
        Address(//ExampleLib.java:lib): (Address(//:lockfile), Address(//ExampleLib.java:lib), Address(//OtherLib.java:lib)),
        Address(//OtherLib.java:lib): (Address(//:lockfile), Address(//ExampleLib.java:lib), Address(//OtherLib.java:lib)),
        Address(//:lockfile): ()}
    )
so both of those files do depend on one another
the lockfile doesn’t have anything to do with it though: it doesn’t have any dependencies of its own, so it can’t participate in any cycles.
f

fast-nail-55400

10/20/2021, 11:48 PM
but the logging from resolve_dependencies does not show a source of the dep
can deps be introduced elsewhere?
CoarsenedTargets is using _DependencyMappingRequest, will go look there
w

witty-crayon-22786

10/20/2021, 11:51 PM
it’s just using resolve_dependencies under the hood.
you’re sure that the return value of 
resolve_dependencies
doesn’t include them?
you dumped the individual components, but maybe you missed one?
f

fast-nail-55400

10/20/2021, 11:54 PM
dumping the result does show the dep, wtf
grr my print statement for inferred had the wrong variable in it
found it
searching for candidate types for org.pantsbuild.example.lib.String ends up reducing it to the package 
org.pantsbuild.example.lib
as the algorithm strips off the parts of the FQT
so need to looks for an exact match only instead of using FirstPartyJavaPackageMapping.addresses_for_symbol
thanks for playing along with my debugging journey <sigh>
❤️ 2
Powered by