#21845 `lark.exceptions.UnexpectedToken` when creating alert rules via Grafana's Terraform Provider Issue created by riebecj Describe the bug When using Grafana's Terraform Provider with Pants to create some alert rules, the provider has a for keyword that represents "for how long" the condition is met before firing (e.g.

for = "5m"

). That

for

keyword is causing

KeyError: 'FOR'

in lark/larl and I'm not sure how to even go about addressing it. I've dug through the provider's and Lark's GitHub issues but came up empty. This seems like it's only an issue when Pants uses lark to parse terraform (i.e. the perfect storm of variables). Running the same code with just Terraform outside of Pants works fine, and removing the

for

keyword and running it with Pants also works. Pants version 2.23.0 OS 5.15.167.4-microsoft-standard-WSL2 SMP Tue Nov 5 002155 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux Have not tested it with Mac. Additional info Full Traceback:

20:26:35.05 [ERROR] 1 Exception encountered:

Engine traceback:
  in `experimental-deploy` goal

ProcessExecutionFailure: Process 'Parse Terraform module sources: src/grafana' failed with exit code 1.
stdout:

stderr:
Traceback (most recent call last):
  File "/home/riebecj/.cache/pants/named_caches/pex_root/venvs/s/af356360/venv/lib/python3.12/site-packages/lark/parsers/lalr_parser_state.py", line 77, in feed_token
    action, arg = states[state][token.type]
                  ~~~~~~~~~~~~~^^^^^^^^^^^^
KeyError: 'FOR'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/tmp/pants-sandbox-KrIDnV/.cache/pex_root/venvs/c686c5a7b985a929356241e22bf584422497e4d9/0de1795ad4486f45ee94cecd983e9905b6b11dc9/pex", line 334, in <module>
    boot(
  File "/tmp/pants-sandbox-KrIDnV/.cache/pex_root/venvs/c686c5a7b985a929356241e22bf584422497e4d9/0de1795ad4486f45ee94cecd983e9905b6b11dc9/pex", line 317, in boot
    runpy.run_module(module_name, run_name="__main__", alter_sys=True)
  File "<frozen runpy>", line 226, in run_module
  File "<frozen runpy>", line 98, in _run_module_code
  File "<frozen runpy>", line 88, in _run_code
  File "/home/riebecj/.cache/pants/named_caches/pex_root/venvs/s/af356360/venv/lib/python3.12/site-packages/__pants_tf_parser.py", line 71, in <module>
    main(sys.argv[1:])
  File "/home/riebecj/.cache/pants/named_caches/pex_root/venvs/s/af356360/venv/lib/python3.12/site-packages/__pants_tf_parser.py", line 64, in main
    paths |= extract_module_source_paths(PurePath(filename).parent, content)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/riebecj/.cache/pants/named_caches/pex_root/venvs/s/af356360/venv/lib/python3.12/site-packages/__pants_tf_parser.py", line 36, in extract_module_source_paths
    parsed_content = hcl2.loads(content)
                     ^^^^^^^^^^^^^^^^^^^
  File "/home/riebecj/.cache/pants/named_caches/pex_root/venvs/s/af356360/venv/lib/python3.12/site-packages/hcl2/api.py", line 27, in loads
    tree = hcl2.parse(text + "\n")
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/riebecj/.cache/pants/named_caches/pex_root/venvs/s/af356360/venv/lib/python3.12/site-packages/lark/lark.py", line 658, in parse
    return self.parser.parse(text, start=start, on_error=on_error)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/riebecj/.cache/pants/named_caches/pex_root/venvs/s/af356360/venv/lib/python3.12/site-packages/lark/parser_frontends.py", line 104, in parse
    return self.parser.parse(stream, chosen_start, **kw)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/riebecj/.cache/pants/named_caches/pex_root/venvs/s/af356360/venv/lib/python3.12/site-packages/lark/parsers/lalr_parser.py", line 42, in parse
    return self.parser.parse(lexer, start)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/riebecj/.cache/pants/named_caches/pex_root/venvs/s/af356360/venv/lib/python3.12/site-packages/lark/parsers/lalr_parser.py", line 88, in parse
    return self.parse_from_state(parser_state)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/riebecj/.cache/pants/named_caches/pex_root/venvs/s/af356360/venv/lib/python3.12/site-packages/lark/parsers/lalr_parser.py", line 111, in parse_from_state
    raise e
  File "/home/riebecj/.cache/pants/named_caches/pex_root/venvs/s/af356360/venv/lib/python3.12/site-packages/lark/parsers/lalr_parser.py", line 102, in parse_from_state
    state.feed_token(token)
  File "/home/riebecj/.cache/pants/named_caches/pex_root/venvs/s/af356360/venv/lib/python3.12/site-packages/lark/parsers/lalr_parser_state.py", line 80, in feed_token
    raise UnexpectedToken(token, expected, state=self, interactive_parser=None)
lark.exceptions.UnexpectedToken: Unexpected token Token('FOR', 'for') at line 108, column 5.
Expected one of: 
        * RBRACE
        * __ANON_3

It looks like lark is parsing it as a

for

block instead of the keyword inside the

rule

block. pantsbuild/pants