LLM generates text that the grammar does not allow

[mattiasarro]

When running the example code, the following exception gets thrown:

UnexpectedCharacters: No terminal matches '"' in the current parser context, at line 1 col 7

The model had generated output {"a": " and the parser did not parse this particular output. Since {"a": " is a valid start of a JSON object, the error is a bit confusing. But playing around with different grammar, prompts and models a bit shows that the model can (and often does) generate text the parser can not parse, which results in lark.lexer.UnexpectedCharacters exception.

[mattiasarro]

This is actually a bug in rellm, created a PR there.