rich-london-74860
03/27/2023, 3:16 PM--changed-since
2. Find the python_distribution
those python files belong to (if any)
3. Verify that version used in the python_distribution
has changed - since we use bumpversion
this is very easy to do by checking if the setup.cfg
file has changed, which we can assume lives in the same directory as the BUILD
file where a python_distribution
is defined
The purpose of this plug-in is to prevent code from getting merged that changes distribution, without also changing and releasing a new version. I should note that we are using artifactory to serve a private pypi.
As I mention, 1 and 3 are easy, but 2 has a gotcha. By itself --changed-since
will only list the specific targets that have changed, In the primary use case we are considering (a developer changed a few python files without changing anything else) this will only include python source targets. Adding --changed-dependees=transitive
will recursively include every dependency. This does include the python distribution target that the changed python source files live in, but for python distributions that are dependencies of other python distributions, this will include those other python distribution targets as well.
This is correct and desirable behavior in some cases. For example, for testing, if a deep dependency has changed, all of it’s dependees should be tested. However, this is not desirable for deploying python packages. When a deep dependency has changed, it is not necessary to deploy it’s dependees because those dependees do not package and include that dependency. For an app that uses the deep dependency and one or some of its dependees, it only needs to install a new version of that deep dependency.
For example, suppose in a pants repo, we have python distribution A and B where B depends on A. A includes file a.py
and B includes file b.py
. If file a.py
changed, then we want to make sure that distribution A has also changed. --changed-since
will get us a.py
only and adding --changed-dependees=transitive
gets us A and B. Outside of the pants repo, suppose there is an app C that installs A and B. Even though distribution B is a dependee of distribution A, app C only needs to install a new version of A.
There are a few fixes and workarounds that I have considered:
1. Forget about doing all of this and instead prevent overwrites on our artifactory pypi host. However, there are other use cases where we do want to overwrite existing versions and finding this problem after merging is annoying.
2. Do not express dependencies between distributions in pants
at all. In this example, this would mean B has a dependency on requirement A instead of the python sources directly. However, this will mean a different resolve
for every python distribution and it seems to defeat the point of having everything in pants
to take advantage of those other use cases where we do want full recursive dependencies.
3. Use 2 dependees
commands to get the next python distribution dependency, without getting deeper distribution dependencies ./pants list --changed-since=other | xargs ./pants dependees | xargs ./pants dependees
Option 3 is the current plan, but this feels like a hack. Ideally, I would like to be able to traverse the DAG of dependencies inside the plugin code. Is this possible?./pants dependees
and ./pants dependencies
the only way to do that?
Is there anyway to get dependees and dependencies in python code, namely in a plugin?
Thanks in advance.bitter-ability-32190
03/29/2023, 1:51 PMpants peek ::
and then scrpt on top of the resulting JSONTransitiveTargetsRequest
I thinkbusy-vase-39202
03/29/2023, 1:58 PMpeel
capabilities.curved-television-6568
03/29/2023, 2:24 PMDependenciesRequest
Get(Addresses, DependenciesRequest(tgt.get(Dependencies), include_special_cased_deps=True/False))
rich-london-74860
03/29/2023, 2:28 PMbitter-ability-32190
03/29/2023, 2:29 PMrich-london-74860
03/29/2023, 2:31 PMcurved-television-6568
03/29/2023, 2:48 PMpants DependenciesRequest --help-advanced
`pants.engine.target.DependenciesRequest` api type
--------------------------------------------------
DependenciesRequest(field: 'Dependencies', include_special_cased_deps: 'bool' = False)
activated by : pants.engine.target
dependencies :
dependents :
returned by 0 rules:
consumed by 0 rules:
used in 10 rules : pants.backend.awslambda.python.target_types.infer_lambda_handler_dependency
pants.backend.project_info.dependencies.dependencies
pants.backend.project_info.dependents.map_addresses_to_dependents
pants.backend.project_info.paths.paths
pants.backend.project_info.peek.get_target_data
pants.backend.python.dependency_inference.rules.resolve_parsed_dependencies
pants.backend.python.goals.setup_py.get_requirements
pants.backend.python.target_types_rules.infer_pex_binary_entry_point_dependency
pants.backend.python.target_types_rules.infer_python_distribution_dependencies
pants.backend.shell.dependency_inference.infer_shell_dependencies
rich-london-74860
03/31/2023, 8:48 PMDependenciesRequest
!
I started taking a deeper look at this today, and this prompted 3 other issues for me.
1. This is DependenciesRequest
, but what I need is dependees, is it possible to go the other way?
2. Targets
from Addresses
? This Get
will returns Addresses
and I see that each Address
object has a target_name
field, but I would need the Target
object to get 2nd degree dependees.Get(Targets, DependenciesRequest(tgt.get(Dependencies)))
3. Assuming that it is not possible to request addresses for dependees, one alternative would get to collect all targets (or at least all python distribution targets), get the dependencies for every target, and then reverse the graph to get dependees. How do I get all targets? I tried to use AllTargets
and AllTargetsRequest
, but that gave me this error:
17:47:58.77 [ERROR] Encountered 12 rule graph errors:
No installed rules return the type BuildTargetIdentifier, and it was not provided by potential callers of @rule(pants.bsp.util_rules.targets:215:resolve_bsp_build_target_identifier(BuildTargetIdentifier, BSPBuildTargets) -> BSPBuildTargetInternal).
If that type should be computed by a rule, ensure that that rule is installed.
If it should be provided by a caller, ensure that it is included in any relevant Query or Get.
No installed rules return the type BuildTargetIdentifier, and it was not provided by potential callers of @rule(pants.bsp.util_rules.targets:215:resolve_bsp_build_target_identifier(BuildTargetIdentifier, BSPBuildTargets) -> BSPBuildTargetInternal).
If that type should be computed by a rule, ensure that that rule is installed.
If it should be provided by a caller, ensure that it is included in any relevant Query or Get.
No installed rules return the type BuildTargetIdentifier, and it was not provided by potential callers of @rule(pants.bsp.util_rules.targets:215:resolve_bsp_build_target_identifier(BuildTargetIdentifier, BSPBuildTargets) -> BSPBuildTargetInternal).
If that type should be computed by a rule, ensure that that rule is installed.
If it should be provided by a caller, ensure that it is included in any relevant Query or Get.
No installed rules return the type BuildTargetIdentifier, and it was not provided by potential callers of @rule(pants.bsp.util_rules.targets:215:resolve_bsp_build_target_identifier(BuildTargetIdentifier, BSPBuildTargets) -> BSPBuildTargetInternal).
If that type should be computed by a rule, ensure that that rule is installed.
If it should be provided by a caller, ensure that it is included in any relevant Query or Get.
No installed rules return the type _ParseOneBSPMappingRequest, and it was not provided by potential callers of @rule(pants.bsp.util_rules.targets:137:parse_one_bsp_mapping(_ParseOneBSPMappingRequest) -> BSPBuildTargetInternal).
If that type should be computed by a rule, ensure that that rule is installed.
If it should be provided by a caller, ensure that it is included in any relevant Query or Get.
No installed rules return the type _ParseOneBSPMappingRequest, and it was not provided by potential callers of @rule(pants.bsp.util_rules.targets:137:parse_one_bsp_mapping(_ParseOneBSPMappingRequest) -> BSPBuildTargetInternal).
If that type should be computed by a rule, ensure that that rule is installed.
If it should be provided by a caller, ensure that it is included in any relevant Query or Get.
No installed rules return the type _ParseOneBSPMappingRequest, and it was not provided by potential callers of @rule(pants.bsp.util_rules.targets:137:parse_one_bsp_mapping(_ParseOneBSPMappingRequest) -> BSPBuildTargetInternal).
If that type should be computed by a rule, ensure that that rule is installed.
If it should be provided by a caller, ensure that it is included in any relevant Query or Get.
No installed rules return the type _ParseOneBSPMappingRequest, and it was not provided by potential callers of @rule(pants.bsp.util_rules.targets:137:parse_one_bsp_mapping(_ParseOneBSPMappingRequest) -> BSPBuildTargetInternal).
If that type should be computed by a rule, ensure that that rule is installed.
If it should be provided by a caller, ensure that it is included in any relevant Query or Get.
No source of dependency BSPBuildTargets for @rule(pants.bsp.util_rules.targets:215:resolve_bsp_build_target_identifier(BuildTargetIdentifier, BSPBuildTargets) -> BSPBuildTargetInternal). All potential sources were eliminated: []
No source of dependency BSPBuildTargets for @rule(pants.bsp.util_rules.targets:215:resolve_bsp_build_target_identifier(BuildTargetIdentifier, BSPBuildTargets) -> BSPBuildTargetInternal). All potential sources were eliminated: []
No source of dependency BSPBuildTargets for @rule(pants.bsp.util_rules.targets:215:resolve_bsp_build_target_identifier(BuildTargetIdentifier, BSPBuildTargets) -> BSPBuildTargetInternal). All potential sources were eliminated: []
No source of dependency BSPBuildTargets for @rule(pants.bsp.util_rules.targets:215:resolve_bsp_build_target_identifier(BuildTargetIdentifier, BSPBuildTargets) -> BSPBuildTargetInternal). All potential sources were eliminated: []
AllUnexpandedTargets
to invoke this rule?dep_addresses = await Get(
Dependents,
DependentsRequest(
addresses=[target.address],
transitive=False,
include_roots=False,
),
)
curved-television-6568
04/02/2023, 1:15 PM