#20242 Exporting project's dependency graph New discussion created by AlexTereshenkov Introduction Pants already provides a rich set of tools to query the dependency graph to find out information about dependencies and dependents of individual build targets (be it a file, a package, or a glob pattern): •

dependencies

goal lets you list the dependencies:

$ pants dependencies cheeseshop/cli/cli.py                 
cheeseshop/cli/cli.py
cheeseshop/cli/utils/utils.py
cheeseshop/repository/package.py
cheeseshop/repository/properties.py
cheeseshop/repository/query.py
cheeseshop/repository/repository.py
cheeseshop/version.py
cheeseshop:project-version
requirements#click
requirements#loguru

•

dependents

(formerly known as

dependees

) goal lets you list the dependents (also known as reverse dependencies):

$ pants dependents cheeseshop/repository/parsing/casts.py  
cheeseshop/repository/package.py
cheeseshop/repository/parsing:parsing
tests/repository/parsing/test_casts.py:tests

It is possible to list dependencies for multiple files, one after another, e.g.

$ pants dependencies cheeseshop/repository/*.py            
cheeseshop/configs.py
cheeseshop/repository/package.py
cheeseshop/repository/parsing/casts.py
cheeseshop/repository/parsing/exceptions.py
cheeseshop/repository/properties.py
requirements#loguru
requirements#packaging
requirements#requests
requirements#typing-extensions

Motivation Having the dependencies listed for multiple targets such as individual source files, you don't know what modules out of those files in the

cheeseshop/repository

package depends on what. Running Pants goal on each individual file is very inefficient: each invocation of Pants has an overhead, so it's more preferrable to get all the work done within a single Pants call. It is also possible that a command would be run in a environment without

pantsd

process already running and/or any cache available. So even though this works, it will prove to be unreasonably slow for even a medium sized codebase:

$ for filename in $(ls cheeseshop/repository/*.py); do 
  echo "--${filename}--"; pants dependencies ${filename}
done

--cheeseshop/repository/package.py--
cheeseshop/repository/parsing/casts.py
cheeseshop/repository/properties.py
requirements#packaging
requirements#typing-extensions

--cheeseshop/repository/query.py--
cheeseshop/repository/package.py
requirements#packaging

--cheeseshop/repository/repository.py--
cheeseshop/configs.py
cheeseshop/repository/package.py
cheeseshop/repository/parsing/exceptions.py
requirements#loguru
requirements#requests

It is therefore more helpful to list dependencies for multiple files individually to be able to distinguish them, using a new goal when construction of the graph happens only once:

$ pants <goal> <options> cheeseshop/repository/*.py
{
    "cheeseshop/repository/__init__.py": [],
    "cheeseshop/repository/package.py": [
        "cheeseshop/repository/parsing/casts.py",
        "cheeseshop/repository/properties.py",
        "requirements#packaging",
        "requirements#typing-extensions"
    ],
    "cheeseshop/repository/properties.py": [],
    "cheeseshop/repository/query.py": [
        "cheeseshop/repository/package.py",
        "requirements#packaging"
    ],
    "cheeseshop/repository/repository.py": [
        "cheeseshop/configs.py",
        "cheeseshop/repository/package.py",
        "cheeseshop/repository/parsing/exceptions.py",
        "requirements#loguru",
        "requirements#requests"
    ],
    "cheeseshop/repository/types.py": []
}

The information produced by this new goal would return adjacency representation of the dependency graph as a dictionary of lists. The output is JSON compatible which makes it trivial to filter and query the graph using standard tooling such as

jq

and standard library of most programming languages. More importantly, this data structure may be used to construct graphs using 3rd party tooling such as

networkx

to be able to query and manipulate it, see networkx.convert.from_dict_of_lists:

$ pants <goal> <options> cheeseshop/repository/*.py > depgraph.json
$ python3
>>> import json
>>> import networkx
>>> with open("depgraph.json") as fh:
...     g = networkx.from_dict_of_lists(json.load(fh), create_using=networkx.DiGraph)
>>> networkx.shortest_path(g, "cheeseshop/repository/query.py", "cheeseshop/repository/properties.py")
['cheeseshop/repository/query.py', 'cheeseshop/repository/package.py', 'cheeseshop/repository/properties.py']

Having the dependency graph exported makes it possible to cheaply answer a variety of useful questions such as: • are there any build targets that no one depends on? • what is the longest path in the graph? • what source module leads to most tests? • what test module has most dependencies? Having the graph exported also opens up the opportunity to visualize the whole graph or its parts using visualization libraries such as `graphviz`:

$ python3 -m venv .venv && source .venv/bin/activate
$ pip install networkx pydot

import json
import networkx
from networkx.drawing.nx_pydot import write_dot

with open("depgraph.json") as fh:
  g = networkx.from_dict_of_lists(json.load(fh), create_using=networkx.DiGraph)
  write_dot(g, "graph.dot")

$ dot -Tpng graph.dot > graph.png

graph Having the graph exported into a JSON data structure is enough to be able to perform any query/manipulation with the graph, but for practical reasons, it may be helpful to provide additional functionality available out-of-the-box to avoid forcing users to write additional programs. This could mean: • listing the dependents (reverse dependencies) • listing the dependencies/dependents transitively Implementation Practically, fetching dependencies (direct or transitive) is trivial:

direct_deps_request_result = await Get(Targets, DependenciesRequest(target[Dependencies]))
deps = [str(d.address) for d in FrozenOrderedSet(direct_deps_request_result)]
...
transitive_deps_request_result = await Get(TransitiveTargets, TransitiveTargetsRequest([target.address]))
dependencies = transitive_deps_request_result.dependencies

and so is fetching dependents:

dependees = await Get(
	Dependents,
	DependentsRequest(
		(target.address,),
		transitive=True,
		include_roots=False,
	),
)

Fetching dependencies for multiple targets is likely to happen in a

MultiGet

call to a rule, filling a mapping of build targets and their dependencies which will be the output of the new goal. With the naming of the goal and the options being subject to change, this is how the user interface may look like: ``` # fetching direct dependencies of two files $ pants dep-graph --dependencies cheeseshop/repository/query.py cheeseshop/repository/package.py { "cheeseshop/repository/package.py": [ "cheeseshop/repository/parsing/casts.py", "cheeseshop/repository/properties.p… pantsbuild/pants