https://pantsbuild.org/ logo
Powered by
r

rich-london-74860

03/27/2023, 3:16 PM
Hello everyone 👋 I’m trying to add a plug-in to our pants repo that will: 1. Find all python files that have changed - this is very easy 🙌 with 
--changed-since
2. Find the 
python_distribution
those python files belong to (if any) 3. Verify that version used in the 
python_distribution
has changed - since we use 
bumpversion
this is very easy to do by checking if the 
setup.cfg
file has changed, which we can assume lives in the same directory as the 
BUILD
file where a 
python_distribution
is defined The purpose of this plug-in is to prevent code from getting merged that changes distribution, without also changing and releasing a new version. I should note that we are using artifactory to serve a private pypi. As I mention, 1 and 3 are easy, but 2 has a gotcha. By itself 
--changed-since
will only list the specific targets that have changed, In the primary use case we are considering (a developer changed a few python files without changing anything else) this will only include python source targets. Adding 
--changed-dependees=transitive
will recursively include every dependency. This does include the python distribution target that the changed python source files live in, but for python distributions that are dependencies of other python distributions, this will include those other python distribution targets as well. This is correct and desirable behavior in some cases. For example, for testing, if a deep dependency has changed, all of it’s dependees should be tested. However, this is not desirable for deploying python packages. When a deep dependency has changed, it is not necessary to deploy it’s dependees because those dependees do not package and include that dependency. For an app that uses the deep dependency and one or some of its dependees, it only needs to install a new version of that deep dependency. For example, suppose in a pants repo, we have python distribution A and B where B depends on A. A includes file 
a.py
and B includes file 
b.py
. If file 
a.py
changed, then we want to make sure that distribution A has also changed. 
--changed-since
will get us 
a.py
only and adding 
--changed-dependees=transitive
gets us A and B. Outside of the pants repo, suppose there is an app C that installs A and B. Even though distribution B is a dependee of distribution A, app C only needs to install a new version of A. There are a few fixes and workarounds that I have considered: 1. Forget about doing all of this and instead prevent overwrites on our artifactory pypi host. However, there are other use cases where we do want to overwrite existing versions and finding this problem after merging is annoying. 2. Do not express dependencies between distributions in 
pants
at all. In this example, this would mean B has a dependency on requirement A instead of the python sources directly. However, this will mean a different 
resolve
for every python distribution and it seems to defeat the point of having everything in 
pants
to take advantage of those other use cases where we do want full recursive dependencies. 3. Use 2 
dependees
commands to get the next python distribution dependency, without getting deeper distribution dependencies 
./pants list --changed-since=other | xargs ./pants dependees | xargs ./pants dependees
Option 3 is the current plan, but this feels like a hack. Ideally, I would like to be able to traverse the DAG of dependencies inside the plugin code. Is this possible?
CC @polite-angle-82480 @adamant-coat-28831
Hello again 👋 I think I made this question a little too complicated and too confusing. Sorry about that. Let me try this again in a different way. If I want to traverse the graph of dependencies, is using 
./pants dependees
and 
./pants dependencies
the only way to do that? Is there anyway to get dependees and dependencies in python code, namely in a plugin? Thanks in advance.
b

bitter-ability-32190

03/29/2023, 1:51 PM
From the CLI, you might try 
pants peek ::
and then scrpt on top of the resulting JSON
From inside a plugin, you would use 
TransitiveTargetsRequest
I think
👀 1
b

busy-vase-39202

03/29/2023, 1:58 PM
And if the repo is public and you just want to visualize the graph, there is also graphmyrepo.com, which iirc is built on top of 
peel
capabilities.
c

curved-television-6568

03/29/2023, 2:24 PM
there’s also 
DependenciesRequest
Copy code
Get(Addresses, DependenciesRequest(tgt.get(Dependencies), include_special_cased_deps=True/False))
r

rich-london-74860

03/29/2023, 2:28 PM
Is there a documentation resource on these different request types?
b

bitter-ability-32190

03/29/2023, 2:29 PM
Not currently. Source code is your friend 😕 I think eventually there'll be enough of a push to maintain healthy docstrings and generate docs from them. But not today
r

rich-london-74860

03/29/2023, 2:31 PM
Ok, thanks a lot, this is exactly what I was looking for!
c

curved-television-6568

03/29/2023, 2:48 PM
not well documented, but there is information available on the CLI help:
Copy code
pants DependenciesRequest --help-advanced

`pants.engine.target.DependenciesRequest` api type
--------------------------------------------------

DependenciesRequest(field: 'Dependencies', include_special_cased_deps: 'bool' = False)

activated by       : pants.engine.target
dependencies       :
dependents         :
returned by 0 rules:
consumed by 0 rules:
used in 10 rules   : pants.backend.awslambda.python.target_types.infer_lambda_handler_dependency
                     pants.backend.project_info.dependencies.dependencies
                     pants.backend.project_info.dependents.map_addresses_to_dependents
                     pants.backend.project_info.paths.paths
                     pants.backend.project_info.peek.get_target_data
                     pants.backend.python.dependency_inference.rules.resolve_parsed_dependencies
                     pants.backend.python.goals.setup_py.get_requirements
                     pants.backend.python.target_types_rules.infer_pex_binary_entry_point_dependency
                     pants.backend.python.target_types_rules.infer_python_distribution_dependencies
                     pants.backend.shell.dependency_inference.infer_shell_dependencies
r

rich-london-74860

03/31/2023, 8:48 PM
Thanks @bitter-ability-32190 and @curved-television-6568 for leading me to 
DependenciesRequest
! I started taking a deeper look at this today, and this prompted 3 other issues for me. 1. This is 
DependenciesRequest
, but what I need is dependees, is it possible to go the other way? 2. Assuming it is possible to request addresses for dependees - How do I get 
Targets
from 
Addresses
? This 
Get
will returns 
Addresses
and I see that each 
Address
object has a 
target_name
field, but I would need the 
Target
object to get 2nd degree dependees. I figured this one out on my own using 
Get(Targets, DependenciesRequest(tgt.get(Dependencies)))
3. Assuming that it is not possible to request addresses for dependees, one alternative would get to collect all targets (or at least all python distribution targets), get the dependencies for every target, and then reverse the graph to get dependees. How do I get all targets? I tried to use 
AllTargets
and 
AllTargetsRequest
, but that gave me this error:
Copy code
17:47:58.77 [ERROR] Encountered 12 rule graph errors:
  No installed rules return the type BuildTargetIdentifier, and it was not provided by potential callers of @rule(pants.bsp.util_rules.targets:215:resolve_bsp_build_target_identifier(BuildTargetIdentifier, BSPBuildTargets) -> BSPBuildTargetInternal).
    If that type should be computed by a rule, ensure that that rule is installed.
    If it should be provided by a caller, ensure that it is included in any relevant Query or Get.
  No installed rules return the type BuildTargetIdentifier, and it was not provided by potential callers of @rule(pants.bsp.util_rules.targets:215:resolve_bsp_build_target_identifier(BuildTargetIdentifier, BSPBuildTargets) -> BSPBuildTargetInternal).
    If that type should be computed by a rule, ensure that that rule is installed.
    If it should be provided by a caller, ensure that it is included in any relevant Query or Get.
  No installed rules return the type BuildTargetIdentifier, and it was not provided by potential callers of @rule(pants.bsp.util_rules.targets:215:resolve_bsp_build_target_identifier(BuildTargetIdentifier, BSPBuildTargets) -> BSPBuildTargetInternal).
    If that type should be computed by a rule, ensure that that rule is installed.
    If it should be provided by a caller, ensure that it is included in any relevant Query or Get.
  No installed rules return the type BuildTargetIdentifier, and it was not provided by potential callers of @rule(pants.bsp.util_rules.targets:215:resolve_bsp_build_target_identifier(BuildTargetIdentifier, BSPBuildTargets) -> BSPBuildTargetInternal).
    If that type should be computed by a rule, ensure that that rule is installed.
    If it should be provided by a caller, ensure that it is included in any relevant Query or Get.
  No installed rules return the type _ParseOneBSPMappingRequest, and it was not provided by potential callers of @rule(pants.bsp.util_rules.targets:137:parse_one_bsp_mapping(_ParseOneBSPMappingRequest) -> BSPBuildTargetInternal).
    If that type should be computed by a rule, ensure that that rule is installed.
    If it should be provided by a caller, ensure that it is included in any relevant Query or Get.
  No installed rules return the type _ParseOneBSPMappingRequest, and it was not provided by potential callers of @rule(pants.bsp.util_rules.targets:137:parse_one_bsp_mapping(_ParseOneBSPMappingRequest) -> BSPBuildTargetInternal).
    If that type should be computed by a rule, ensure that that rule is installed.
    If it should be provided by a caller, ensure that it is included in any relevant Query or Get.
  No installed rules return the type _ParseOneBSPMappingRequest, and it was not provided by potential callers of @rule(pants.bsp.util_rules.targets:137:parse_one_bsp_mapping(_ParseOneBSPMappingRequest) -> BSPBuildTargetInternal).
    If that type should be computed by a rule, ensure that that rule is installed.
    If it should be provided by a caller, ensure that it is included in any relevant Query or Get.
  No installed rules return the type _ParseOneBSPMappingRequest, and it was not provided by potential callers of @rule(pants.bsp.util_rules.targets:137:parse_one_bsp_mapping(_ParseOneBSPMappingRequest) -> BSPBuildTargetInternal).
    If that type should be computed by a rule, ensure that that rule is installed.
    If it should be provided by a caller, ensure that it is included in any relevant Query or Get.
  No source of dependency BSPBuildTargets for @rule(pants.bsp.util_rules.targets:215:resolve_bsp_build_target_identifier(BuildTargetIdentifier, BSPBuildTargets) -> BSPBuildTargetInternal). All potential sources were eliminated: []
  No source of dependency BSPBuildTargets for @rule(pants.bsp.util_rules.targets:215:resolve_bsp_build_target_identifier(BuildTargetIdentifier, BSPBuildTargets) -> BSPBuildTargetInternal). All potential sources were eliminated: []
  No source of dependency BSPBuildTargets for @rule(pants.bsp.util_rules.targets:215:resolve_bsp_build_target_identifier(BuildTargetIdentifier, BSPBuildTargets) -> BSPBuildTargetInternal). All potential sources were eliminated: []
  No source of dependency BSPBuildTargets for @rule(pants.bsp.util_rules.targets:215:resolve_bsp_build_target_identifier(BuildTargetIdentifier, BSPBuildTargets) -> BSPBuildTargetInternal). All potential sources were eliminated: []
Okay, I think I’ve found what I need ☝️~, it’s this rule `map_addresses_to_dependents`~ here, but I do not know how to invoke this rule. How do I instantiate 
AllUnexpandedTargets
to invoke this rule? Nevermind, I figured out I can use this to get what I need
Copy code
dep_addresses = await Get(
                Dependents,
                DependentsRequest(
                    addresses=[target.address],
                    transitive=False,
                    include_roots=False,
                ),
            )
c

curved-television-6568

04/02/2023, 1:15 PM
Nice work navigating this @rich-london-74860 👍
2 Views
Powered by