Use library for vimscript parsing instead of regexes #110

Activity

1. dbarnett

   added

   enhancement

   on Jan 6, 2020
2. dbarnett commented on Mar 31, 2020

   

   dbarnett

   on Mar 31, 2020

   ContributorAuthor

   More actions

   I did a little poking around here and it doesn't look too promising so far. It seems to parse expressions only, not commands, and doesn't seem to handle comments or line continuations either.

   Here are some samples of output I got invoking it from inside vim

   :echo nvim_parse_expression("s:plugin.Flag('foo', 'bar')", '', 0)
   {'ast':
    {'children':
     [{'children':
       [{'ident': 'plugin',
         'len': 8,
         'scope': 115,
         'start': [0, 0],
         'type': 'PlainIdentifier'},
        {'ident': 'Flag', 'len': 4, 'start': [0, 9], 'type': 'PlainKey'}],
       'len': 1,
       'start': [0, 8],
       'type': 'ConcatOrSubscript'},
      {'children':
       [{'len': 5,
         'start': [0, 14],
         'svalue': 'foo',
         'type': 'SingleQuotedString'},
        {'len': 6,
         'start': [0, 20],
         'svalue': 'bar',
         'type': 'SingleQuotedString'}],
       'len': 1,
       'start': [0, 19],
       'type': 'Comma'}],
     'len': 1,
     'start': [0, 13],
     'type': 'Call'},
    'len': 27}

   :Verbose PP nvim_parse_expression("call s:plugin.Flag('foo', 'bar')", '', 0)
   {'ast':
    {'children':
     [{'ident': 'call', 'len': 4, 'scope': 0, 'start': [0, 0], 'type': 'PlainIdentifier'},
      {'children':
       [{'children':
         [{'ident': 'plugin', 'len': 9, 'scope': 115, 'start': [0, 4], 'type': 'PlainIdentifier'},
          {'ident': 'Flag', 'len': 4, 'start': [0, 14], 'type': 'PlainKey'}],
         'len': 1,
         'start': [0, 13],
         'type': 'ConcatOrSubscript'},
        {'children':   
         [{'len': 5, 'start': [0, 19], 'svalue': 'foo', 'type': 'SingleQuotedString'},
          {'len': 6, 'start': [0, 25], 'svalue': 'bar', 'type': 'SingleQuotedString'}],
         'len': 1,
         'start': [0, 24],
         'type': 'Comma'}],
       'len': 1,
       'start': [0, 18],
       'type': 'Call'}],  
     'len': 0,
     'start': [0, 4],
     'type': 'OpMissing'},
    'error': {'arg': "s:plugin.Flag('foo', 'bar')", 'message': 'E15: Missing operator: %.*s'},
    'len': 32}

   :Verbose PP nvim_parse_expression("\"\n\" @public\ncall s:plugin.Flag(\n    \\ 'foo', 'bar')", '', 0)
   {'ast':
    {'children':
     [{'len': 3, 'start': [0, 0], 'svalue': "\n", 'type': 'DoubleQuotedString'},
      {'children':
       [{'len': 3, 'name': 112, 'start': [0, 3], 'type': 'Register'},
        {'ident': 'ublic', 'len': 5, 'scope': 0, 'start': [0, 6], 'type': 'PlainIdentifier'}],    
       'len': 0,
       'start': [0, 6],
       'type': 'OpMissing'}],
     'len': 0,
     'start': [0, 3],  
     'type': 'OpMissing'},
    'error': {'arg': "@public\ncall s:plugin.Flag(\n    \\ 'foo', 'bar')", 'message': 'E15: Missing operator: %.*s'},
    'len': 11}
3. dbarnett commented on Mar 31, 2020

   

   dbarnett

   on Mar 31, 2020

   ContributorAuthor

   More actions

   I did find that https://github.com/vim-jp/vim-vimlparser does a pretty good job parsing arbitrary vimscript. We'd probably want to use that instead, but I can't figure out how to declare a dependency on it for fetch it from pypi yet (vim-jp/vim-vimlparser#170).
4. dbarnett

   changed the title [-]Use libnvim for vimscript parsing instead of regexes[/-] [+]Use library for vimscript parsing instead of regexes[/+] on Mar 31, 2020
5. martindemello commented on Mar 20, 2022

   

   martindemello

   on Mar 20, 2022

   Contributor

   More actions

   there's a treesitter parser for viml and python bindings to treesitter - might be worth exploring.
6. flwyd commented on Oct 19, 2026

   

   flwyd

   on Oct 19, 2026

   More actions

   It looks like the treesitter parser and presumably nvim_parse_expression don't support Vim 9 script (#144). vim-jp/vim-vimlparser does handle vim9. The latter also has a Vim implementation (Python and JS code is generated from the authoritative Vim code), so it would be possible to rewrite vimdoc in vimscript itself, simplifying the release distribution story.