https://pantsbuild.org/ logo
Join Slack
Powered by
Hey all! I’m struggling with opaque dependency gra...
# plugins
c
Hey all! I’m struggling with opaque dependency graph errors and curious if folks have advice for how to debug. Threading the details below to save channel space…
I’m trying to extend our custom Docker plugin and finding that changing up the rule that creates a 
BuiltPackage
is throwing the following graph error on a basic package command (truncated to respect the Slack character limit):
Copy code
$./pants package --no-local-cache test_docker:dockerized_flask_app
15:10:38.24 [INFO] Initializing scheduler...
15:10:38.61 [ERROR] Encountered 51 rule graph errors:
  No installed rules return the type ArchiveFieldSet, and it was not provided by potential callers of @rule(pants.core.target_types:657:package_archive_target(ArchiveFieldSet) -> BuiltPackage, gets=[Get(Targets, UnparsedAddressInputs), Get(FieldSetsPerTarget, FieldSetsPerTargetRequest), Get(HydratedSources, HydrateSourcesRequest), Get(Snapshot, MergeDigests), Get(Digest, CreateArchive)]).
    If that type should be computed by a rule, ensure that that rule is installed.
    If it should be provided by a caller, ensure that it is included in any relevant Query or Get.
  No installed rules return the type DockerFilesFS, and it was not provided by potential callers of @rule(sendwave.pants_docker.sources:55:get_files(DockerFilesFS) -> DockerComponent, gets=[Get(StrippedSourceFiles, SourceFilesRequest)]).
    If that type should be computed by a rule, ensure that that rule is installed.
    If it should be provided by a caller, ensure that it is included in any relevant Query or Get.
 ...
The 
--print-stacktrace
and 
-ldebug
flags don’t add anything notable to me, but I’m happy to post the output if you think there might be clues there. Based on trial-and-error (commenting things out until it works), this is the code block in the rule that raises the graph error:
Copy code
pip_requirement_targets = []
for target in transitive_targets.dependencies:
    if isinstance(target, PythonRequirementTarget):
        for value in target.get(PythonRequirementsField).value:
            pip_requirement_targets.append(value)

pip_requirements_component = await Get(
    DockerComponent,
    PythonRequirementsFS(requirements=tuple(pip_requirement_targets)),
)
Adding some logging reveals that the first loop to generate the 
pip_requirement_targets
array works fine; the problem is the 
Get
call to turn a 
PythonRequirementsFS
into a 
DockerComponent
. Here’s the edits we made to that rule, which really don’t change anything other than using a subsystem option to switch up the way that we format the pip install command in the image definition.
Copy code
@rule
async def get_requirements(
    field_set: PythonRequirementsFS, setup: PythonSetup, repos: PythonRepos, docker: Docker
) -> DockerComponent:
    ...
Any thoughts on how we can proceed with debugging? TL;DR: We have a rule that generates 
DockerComponent
from 
PythonRequirementsFS
, but executing that rule via a 
Get
call seems to raise a graph error implying that no rules were registered at all. The fact that the error is reporting all the rules to be missing rather than the offending one, plus the general tendency of graph errors to be opaque, makes me think there’s some other problem here that’s being suppressed.
w
seems to raise a graph error implying that no rules were registered at all.
rule graph errors are unfortunately quite terrible: sorry for the trouble. it’s generally the case that a small breakage causes many dozens-to-hundreds of errors.
c
No worries @witty-crayon-22786, that matches my understanding! Any advice for digging deeper on what might cause the breakage?
w
one thing that can be helpful is using 
-ltrace --no-pantsd
, as it will render a dot graph for the error instead as well
🙌 1
if you can share that, i might be able to point to something useful
c
Cool, that does indeed print more output! Looks like the 
errored subgraph
is probably the most helpful part of that message, so I’ve attached it here in a txtfile.
digraph.txt
w
which version of Pants is this? i assume you didn’t also change the Pants version while making your edits?
c
This is pants 2.13, and you’re right, we didn’t change the Pants version while making these edits.
w
is the 
Docker
type a Subsystem?
…yes
c
Yup!
It also doesn’t seem to make a difference — the error persists even when removing it as an argument along with the logic that references it in the rule function body.
w
Based on trial-and-error (commenting things out until it works), this is the code block in the rule that raises the graph error:
assuming that this is accurate, then the relevant subgraph is here… unfortunately, a symptom of the error messages being terrible is that you have to take all of this with a grain of salt. but it looks like it is attempting to use 
sendwave.pants_docker.python_requirement:74:get_requirements
to compute 
Get(DockerComponent, PythonRequirementsFS)
, but failing for some reason. unfortunately, it is claiming that all of the things it needed are missing, which is unlikely.
(and regarding my negativity around the error messages: i wrote this code, and am very interested in fixing it… eventually)
c
That makes sense, and matches what I was thinking! Any thoughts on how we can figure out the reason why 
get_requirements
is failing?
w
and yea, i don’t see any issues with your edit.
…oh. hm. one slightly odd thing which _shouldn’_t matter, but might: 
UnionRule(DockerComponentFieldSet, PythonRequirementsFS)
that means that 
package_into_image
will already be solving 
Get(DockerComponent, ..)
for all members of 
DockerComponentFieldSet
you’ve added a 
Get(DockerComponent, PythonRequirementsFS)
but try changing it to 
Get(DockerComponent, DockerComponentFieldSet, PythonRequirementsFS(..))
👀 1
(as mentioned above, this shouldn’t matter, so if it does end up mattering a bug report would be great).
c
Great point, that causes a different error to be raised but based on the stack trace it does seem like it fixes the rule resolution problem!
Copy code
./pants --print-stacktrace package test_docker:dockerized_flask_app                                                                                                                                                      1 err | 7s | remit-srv py | 04:01:20 PM 
16:01:32.69 [INFO] Initializing scheduler...
16:01:33.06 [INFO] Scheduler initialized.
16:01:35.12 [ERROR] 1 Exception encountered:

Engine traceback:
  in select
  in pants.core.goals.package.package_asset
  in sendwave.pants_docker.package.package_into_image (test_docker:dockerized_flask_app)
Traceback (no traceback):
  <pants native internals>
Exception: Invalid Get. Because the second argument to `Get(DockerComponent, DockerComponentFieldSet, PythonRequirementsFS(requirements=(<pants.backend.python.pip_requirement.PipRequirement object at 0x10b8f84f0>, <pants.backend.python.pip_requirement.PipRequirement object at 0x10b8f8520>, <pants.backend.python.pip_requirement.PipRequirement object at 0x10b89caf0>)))` is annotated with `@union`, the third argument should be a member of that union. Did you intend to register `UnionRule(DockerComponentFieldSet, PythonRequirementsFS)`? If not, you may be using the wrong type (PythonRequirementsFS) for the third argument.
w
hm, yea… that’s runtime, so it got past rule graph solving. but that union should be installed, so … hm
c
Yeah, here’s the 
rules
definition, which hasn’t changed during this PR: https://github.com/waveremit/pants-docker/blob/e20083ffc4445f105fdf0ad0bef7aab4aa6[…]4deb0/pants_plugins/sendwave/pants_docker/python_requirement.py
Copy code
def rules():
    return [
        UnionRule(DockerComponentFieldSet, PythonRequirementsFS),
        *collect_rules(),
    ]
@witty-crayon-22786 I dropped a 
<http://logger.info|logger.info>("Running python_requirement rules")
call in there to see if the 
rules
function was getting executed, and I confirmed it is:
Copy code
$ ./pants --print-stacktrace package test_docker:dockerized_flask_app                                                                                                                                                      1 err | 7s | remit-srv py | 09:15:57 AM 
09:16:19.18 [INFO] Running python_requirement rules
...
Based on the error, I expect this is the block of Rust code that’s raising the exception: https://github.com/pantsbuild/pants/blob/2adf6ed862403d69ad053f29a36c7928f8cced93/src/rust/engine/src/nodes.rs#L1057-L1080 I’m not totally sure what’s going on here but it seems like the select query is being adjusted to include “in scope types” when searching the 
edges
object, which is turning up empty and hence falling through to the union exception. Is there any way I can inspect the 
edges
object directly to see where 
PythonREquirementsFS
is registered?
w
in scope types
these are only in 2.15.x + i think… shouldn’t be in any released versions
but yes, if you checkout the version of pants that matches the one used in your repository, then you can run pants from sources in your repository, and add additional debug output: https://www.pantsbuild.org/docs/running-pants-from-sources
🙌 1
there are branches for each major version: i.e. https://github.com/pantsbuild/pants/tree/2.14.x and tags for particular releases, i.e. https://github.com/pantsbuild/pants/tree/release_2.14.0
c
Thank you @witty-crayon-22786! We figured out the root cause of the error — turns out we were importing the 
PythonRequirementsFS
object from the installed package in the virtualenv rather than from the development source code, and the object had a different structure in the installed package 🤦‍♀️ We fixed the import and it works great now!
w
yikes. ok, so the @union thing was a red herring
glad you figured it out.
i’m pretty curious why you have a separate docker plugin from the built in one: if you have time to share an explanation for that somewhere, it could be helpful in improving the builtin one
Open in Slack
PreviousNext
Powered by