https://pantsbuild.org/ logo
Join Slack
Powered by
Is there a built-in way to detect and warn about i...
# general
h
Is there a built-in way to detect and warn about internal dependencies being deleted and not removed from their respective dependent’s imports? Or would the closest to this be getting a list of deleted targets and comparing them against dependencies of other targets?
f
I am sorry if I am getting it wrong, but I think you are looking for https://www.pantsbuild.org/2.21/reference/subsystems/python-infer#unowned_dependency_behavior. If you have 
from foo import bar
and 
foo
Python requirement target was deleted, when constructing dependency graph (e.g. running 
pants dependencies ::
) you'll get Pants failing if you've set that property to error. Just in case if you are looking for the opposite, say you have a Python requirement and you are no longer sure whether it's actually needed by anyone (e.g. you had at some point a module needing it, but maybe after some refactoring this is no longer needed?), you would need to inspect every relevant build target to find if they don't have dependents (i.e. no one is using them) and they can perhaps be deleted.
since 2.20, you can export the dep graph as an adjacency list which would make querying for dependencies/dependents a lot easier https://www.pantsbuild.org/2.20/reference/goals/dependents#format
h
I have https://www.pantsbuild.org/2.21/reference/subsystems/python-infer#unowned_dependency_behavior enabled but what I was thinking was detection specific to internal dependencies. Since in its current state, if 
foo
internal dependency has been deleted, Pants becomes unaware and unable to infer an import like 
from foo import bar
as an internal dependency. Perhaps if it could utilise mapping source roots to namespaces it could provide a more informative “import of first-party code failed / target not found” error rather than a more general (albeit still useful) “cannot infer owner” error.
I’ll take a look at that dep graph feature as it may be simpler to write a small check for this myself. 🙂
My suggestion above does rely on being able to resolve imports successfully without without the target code so it may not be easy, however. Maybe this is a case for a simple diff of targets instead based on changed-since.
f
right so by "internal dependency" you mean first party code that was refactored? I.e. the 
foo
module is gone and now 
from foo import bar
will break.
Copy code
diff --git a/cheeseshop/cli/cli.py b/cheeseshop/cli/cli.py
index e51f256..bad528a 100644
--- a/cheeseshop/cli/cli.py
+++ b/cheeseshop/cli/cli.py
@@ -3,7 +3,7 @@ import sys
 import click
 from loguru import logger
 
-from cheeseshop.cli.utils.utils import choices
+from cheeseshop.cli.utils.utils_foo import choices
 from cheeseshop.repository.package import PackageType
 from cheeseshop.repository.properties import Architecture, Platform
 from cheeseshop.repository.query import filter_releases, get_versions, is_stable_version
this may be slow depending on the size of the codebase, but this gives you list of unowned modules:
Copy code
❯ pants --backend-packages="+['pants.backend.experimental.python']" python-dump-source-analysis --python-dump-source-analysis-analysis-flavor=raw_dependency_inference cheeseshop/cli/cli.py | jq
[
  {
    "fs": {
      "address": "cheeseshop/cli/cli.py",
      "source": "cli.py",
      "dependencies": [
        "cheeseshop:project-version"
      ],
      "resolve": null,
      "interpreter_constraints": null
    },
    "identified": {
      "imports": {
        "click": {
          "lineno": 3,
          "weak": false
        },
        "cheeseshop.repository.query.is_stable_version": {
          "lineno": 9,
          "weak": false
        },
        "cheeseshop.repository.repository.Repository": {
          "lineno": 10,
          "weak": false
        },
        "cheeseshop.version.VERSION": {
          "lineno": 11,
          "weak": false
        },
        "cheeseshop.cli.utils.utils_foo.choices": {
          "lineno": 6,
          "weak": false
        },
        "cheeseshop.repository.package.PackageType": {
          "lineno": 7,
          "weak": false
        },
        "cheeseshop.repository.query.filter_releases": {
          "lineno": 9,
          "weak": false
        },
        "loguru.logger": {
          "lineno": 4,
          "weak": false
        },
        "sys": {
          "lineno": 1,
          "weak": false
        },
        "cheeseshop.repository.properties.Architecture": {
          "lineno": 8,
          "weak": false
        },
        "cheeseshop.repository.properties.Platform": {
          "lineno": 8,
          "weak": false
        },
        "cheeseshop.repository.query.get_versions": {
          "lineno": 9,
          "weak": false
        }
      },
      "assets": []
    },
    "resolved": {
      "resolve_results": {
        "click": {
          "status": "ImportOwnerStatus.unambiguous",
          "address": [
            "requirements#click"
          ]
        },
        "cheeseshop.repository.query.is_stable_version": {
          "status": "ImportOwnerStatus.unambiguous",
          "address": [
            "cheeseshop/repository/query.py"
          ]
        },
        "cheeseshop.repository.repository.Repository": {
          "status": "ImportOwnerStatus.unambiguous",
          "address": [
            "cheeseshop/repository/repository.py"
          ]
        },
        "cheeseshop.version.VERSION": {
          "status": "ImportOwnerStatus.unambiguous",
          "address": [
            "cheeseshop/version.py"
          ]
        },
        "cheeseshop.cli.utils.utils_foo.choices": {
          "status": "ImportOwnerStatus.unowned",
          "address": []
        },
        "cheeseshop.repository.package.PackageType": {
          "status": "ImportOwnerStatus.unambiguous",
          "address": [
            "cheeseshop/repository/package.py"
          ]
        },
        "cheeseshop.repository.query.filter_releases": {
          "status": "ImportOwnerStatus.unambiguous",
          "address": [
            "cheeseshop/repository/query.py"
          ]
        },
        "loguru.logger": {
          "status": "ImportOwnerStatus.unambiguous",
          "address": [
            "requirements#loguru"
          ]
        },
        "sys": {
          "status": "ImportOwnerStatus.unownable",
          "address": []
        },
        "cheeseshop.repository.properties.Architecture": {
          "status": "ImportOwnerStatus.unambiguous",
          "address": [
            "cheeseshop/repository/properties.py"
          ]
        },
        "cheeseshop.repository.properties.Platform": {
          "status": "ImportOwnerStatus.unambiguous",
          "address": [
            "cheeseshop/repository/properties.py"
          ]
        },
        "cheeseshop.repository.query.get_versions": {
          "status": "ImportOwnerStatus.unambiguous",
          "address": [
            "cheeseshop/repository/query.py"
          ]
        }
      },
      "assets": {},
      "explicit": {
        "address": "cheeseshop/cli/cli.py",
        "includes": [
          "cheeseshop:project-version"
        ],
        "ignores": []
      }
    },
    "possible_owners": {
      "value": {}
    }
  }
]
see
Copy code
"cheeseshop.cli.utils.utils_foo.choices": {
          "status": "ImportOwnerStatus.unowned",
          "address": []
        },
if you do something fancy, you could override the error message, but this would require having a plugin:
Copy code
pants dependencies root/sources/project/mod.py                        
11:52:23.17 [INFO] Initializing scheduler...
11:52:26.47 [INFO] Scheduler initialized.
11:52:30.23 [ERROR] 1 Exception encountered:

Engine traceback:
  in `dependencies` goal

UnownedDependencyError: Pants cannot infer owners for the following imports in the target root/sources/project/mod.py:

  * foobar (line: 62)

If you do not expect an import to be inferrable, add `# pants: no-infer-dep` to the import line. Otherwise, see <https://www.pantsbuild.org/v2.19/docs/troubleshooting#import-errors-and-missing-dependencies> for common problems.

-----------------------------------------------------------------------------
To learn how to fix import ownership rules errors, visit:
<internal wiki address>
-----------------------------------------------------------------------------
Copy code
from internal_plugins.epilogues.messages import construct_epilogue_body, UnownedDependencyErrorEpilogue
from pants.backend.python.dependency_inference import rules as dep_infer_rules
from pants.engine.rules import collect_rules


class UnownedDependencyError(Exception):
    def __init__(self, message):
        super(UnownedDependencyError, self).__init__(  
            message + "\n\n" + construct_epilogue_body(UnownedDependencyErrorEpilogue)
        )


def rules(): 
    dep_infer_rules.UnownedDependencyError = UnownedDependencyError
    return collect_rules()
h
Nice, very cool! I hadn’t used 
python-dump-source-analysis
before. I could extend that plugin to handle first-party targets via source roots of known projects list. Looks like what I’m wanting. Thanks, Alexey - hope you’re doing well 🙏
❤️ 1
Open in Slack
PreviousNext
Powered by