Skip to content

Vertically align self-spec for better readability #1

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
eugenesvk wants to merge 2 commits into
base: main
Choose a base branch
from
Open

Vertically align self-spec for better readability #1

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
7769537
vertically align spec
eugenesvk Aug 1, 2023
9938de5
update comment syntax
eugenesvk Aug 2, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
277 changes: 57 additions & 220 deletions self-spec.ebnf
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,220 +1,57 @@
/**
* A grammar is a list of one or more rules.
*/
grammar ::= rule+ ;

/**
* A rule can be either a grammar or a tokenizer rule. If a rule is defined
* more than once, each definition will just add to the alternatives.
*/
rule

/**
* A tokenizer rule, of the form `symbol :== pattern ;`. The tokenizer (if
* any) is expected to tokenize the input as follows:
*
* - try matching the _ token rule and all named token rules that are
* *directly* referred to from grammar production rules on the remainder
* of the input;
* - emit the token corresponding to the longest match to the parser, or
* drop it if said longest match is the _ token;
* - if there is a tie in match lengths, emit the token that was defined
* earlier in the grammar (the first definition is what counts, if
* there are multiple);
* - if no token matched, emit an unrecognized character error and (maybe)
* try to recover.
*
* Parsers that do not have a tokenizer (i.e. their tokens are simply the
* characters on the input) can treat these as normal rules.
*
* Tokenizer rules may not directly or indirectly refer to themselves, i.e.
* they may not be recursive. This allows them to be mapped onto regular
* expressions.
*
* Tokenizer rules that are not directly used within grammar rules are not
* treated as actual tokens; rather, they are treated as reusable bits of a
* regular expression. Borrowing ANTLR terminology, they are called
* fragments (but unlike in ANTLR, they are detected automatically). This is
* just a readability thing; it makes no difference for the matched language.
*/
::= doc? symbol ":==" alternation ";" -> token-rule

/**
* A whitespace rule, of the form `_ :== pattern ;`. Works like a normal
* tokenizer rule, but the token is not emitted to the parser. Parsers that
* do not have a tokenizer (i.e. their tokens are simply the characters on
* the input) must implicitly match _* at the start of the grammar and after
* every symbol in grammar production rules (i.e. `::=` rules).
*/
| doc? "_" ":==" alternation ";" -> whitespace-rule

/**
* A grammar production rule, for the form `symbol ::= pattern ;`. The syntax
* here is a bit convoluted because the toplevel alternations can be given
* docstrings and can be named, but the syntax is otherwise the same as for
* token rules.
*
* Whenever literal patterns are used in a grammar rule, they are implicitly
* converted to a token rule. Be mindful of this when writing the grammar:
* they can cause tokenizer conflicts just like any other token!
*/
| doc? symbol first-alter subseq-alter* ";" -> grammar-rule
;

/**
* All symbol names are written in kebab-case. GrammarSpec will convert the
* names to the appropriate case conventions for the target language.
*/
symbol :== [a-z] [a-z0-9#-]* ;

/**
* `a | b | ...` preferentially matches `a`, matches `b` if `a` does not
* match, and so on.
*/
alternation ::= concatenation ("|" concatenation)* ;

/**
* The alternatives at the root of a grammar production rule can be annotated
* with a variant name (if not specified it is auto-generated when needed) at
* the end and can be given a docstring before the ::= or | that starts the
* variant. They are functionally indistinguishable from a normal, nested
* alternation, though the parse tree may be generated in a different way.
*/
first-alter ::= doc? "::=" concatenation alter-name? ;
subseq-alter ::= doc? "|" concatenation alter-name? ;

/**
* A toplevel alternative can be named by appending `-> name`. The names do not
* have semantical meaning (they cannot be referred to within the grammar),
* but are used to attach names to variants in the generated ASTs.
*/
alter-name ::= "->" symbol ;

/**
* `a b ...` matches `a` followed by `b` and so on.
*/
concatenation ::= repetition+ ;

/**
* The usual ?*+ characters can be used to specify repetition.
*/
repetition

/**
* `a?` greedily matches `a` zero times or once.
*/
::= singular "?" -> maybe

/**
* `a*` greedily matches `a` zero or more times.
*/
| singular "*" -> any

/**
* `a+` greedily matches `a` one or more times.
*/
| singular "+" -> many

/**
* `a` matches `a` exactly once. Note that this one must come last in the
* list of alternatives, because otherwise the other alternatives would
* never match (because alternatives pick the *first* match, not the longest
* one).
*/
| singular -> one
;

/**
* A single literal pattern or referenced rule to match, or a parenthesized
* expression.
*/
singular
::= "(" alternation ")" -> nested
| symbol
| string-literal
| character-set
| character-literal
;

/**
* `'...'` and `"..."` match `...` literally. Empty string literals are not
* supported; instead use `?`.
*/
string-literal
:== "'" ( [^'##] | ## [^] )+ "'"
| '"' ( [^"##] | ## [^] )+ '"'
;

/**
* `[...]` matches a single character within the specified set. `a-z` may be
* used to select a whole range of code points, and a `^` at the front of the
* `...` inverts the set.
*/
character-set :== "[" "^"? ( character-set-char ( "-" character-set-char )? )* "]" ;

/**
* Within a character set, the characters `^-]#` have special meaning. Prefix
* them with a `#` to match them literally.
*/
character-set-char :== [^#^#-#]##] | escape-sequence ;

/**
* Hash escape sequences for characters can also be matched outside the context
* of a string or character range.
*/
character-literal :== escape-sequence ;

/**
* A hash escape sequence for a special character:
*
* - `#t`: short for `#x09` (tab).
* - `#n`: short for `#x0A` (LF/newline).
* - `#r`: short for `#x0D` (CR/carriage return).
* - `##`: matches `#` literally.
* - `#'`: matches `'` literally (for use within `'`-delimited strings).
* - `#"`: matches `"` literally (for use within `"`-delimited strings).
* - `#-`: matches `-` literally (for use within character sets).
* - `#^`: matches `^` literally (for use within character sets).
* - `#]`: matches `]` literally (for use within character sets).
* - `#xHH`: matches the given 8-bit code point, expressed with two hex
* digits.
* - `#uHHHH`: matches the given 16-bit code point, expressed with four hex
* digits.
* - `#UHHHHHHHH`: matches the given 32-bit code point, expressed with eight
* hex digits.
*/
escape-sequence :== ## (
[^xuU] |
"x" hex hex |
"u" hex hex hex hex |
"U" hex hex hex hex hex hex hex hex
) ;

/**
* A hex digit.
*/
hex :== [0-9a-fA-F] ;

/**
* /** comments are treated as docstrings. They may only appear immediately in
* front of a rule or toplevel alternative in a grammar production rule.
*/
doc :== "/**" comment-body "*/" ;

/**
* Normal /* comments can appear anywhere.
*/
_ :== "/*" ( [^*] comment-body )? "*/" ;

/**
* The body of a comment. Some magic is needed here to ensure that the body
* doesn't match the comment end sequence anywhere in it; we can't just match
* any character because the start of the first comment in the file would
* greedily match right up to the end of the last comment in the file.
*/
comment-body :== ( [^*] | "*"+ [^*/] )* "*"* ;

/**
* Whitespace can appear anywhere.
*/
_ :== [ #t#n#r]+ ;
grammar ::= rule+ ; /// A grammar is a list of one or more rules
rule // A rule can be either a grammar or a tokenizer rule. If a rule is defined more than once, each definition will just add to the alternatives
::= doc? symbol ":==" alternation ";" -> token-rule /** A tokenizer rule, of the form `symbol :== pattern ;`. The tokenizer (if any) is expected to tokenize the input as follows:
- try matching the _ token rule and all named token rules that are *directly* referred to from grammar production rules on the remainder of the input;
- emit the token corresponding to the longest match to the parser, or drop it if said longest match is the _ token;
- if there is a tie in match lengths, emit the token that was defined earlier in the grammar (the first definition is what counts, if there are multiple);
- if no token matched, emit an unrecognized character error and (maybe) try to recover.
Parsers that do not have a tokenizer (i.e. their tokens are simply the characters on the input) can treat these as normal rules.
Tokenizer rules may not directly or indirectly refer to themselves, i.e. they may not be recursive. This allows them to be mapped onto regular expressions.
Tokenizer rules that are not directly used within grammar rules are not treated as actual tokens; rather, they are treated as reusable bits of a regular expression. Borrowing ANTLR terminology, they are called fragments (but unlike in ANTLR, they are detected automatically). This is just a readability thing; it makes no difference for the matched language */
| doc? "_" ":==" alternation ";" -> whitespace-rule // A whitespace rule, of the form `_ :== pattern ;`. Works like a normal tokenizer rule, but the token is not emitted to the parser. Parsers that do not have a tokenizer (i.e. their tokens are simply the characters on the input) must implicitly match _* at the start of the grammar and after every symbol in grammar production rules (i.e. `::=` rules).
| doc? symbol first-alter subseq-alter* ";" -> grammar-rule // A grammar production rule, for the form `symbol ::= pattern ;`. The syntax here is a bit convoluted because the toplevel alternations can be given docstrings and can be named, but the syntax is otherwise the same as for token rules.
// Whenever literal patterns are used in a grammar rule, they are implicitly converted to a token rule. Be mindful of this when writing the grammar: they can cause tokenizer conflicts just like any other token!
;

symbol :== [a-z] [a-z0-9#-]* ; /// name-is-kebab-case, autoconverted to the appropriate case conventions for the target language
alternation ::= concatenation  
("|" concatenation)* ; /// `a | b | ...` preferentially matches `a`, matches `b` if `a` does not match, and so on
first-alter ::= doc? "::=" concatenation alter-name? ; /// alternatives at the root of a grammar production rule can be annotated with a variant name (if not specified it is auto-generated when needed) at the end and can be given a docstring before the ::= or | that starts the variant. They are functionally indistinguishable from a normal, nested alternation, though the parse tree may be generated in a different way.
subseq-alter ::= doc? "|" concatenation alter-name? ; ///
alter-name ::= "->" symbol ; /// A toplevel alternative can be named by appending `-> name`. The names do not have semantical meaning (they cannot be referred to within the grammar), but are used to attach names to variants in the generated ASTs.
concatenation ::= repetition+ ; /// `a b ...` matches `a` followed by `b` and so on.
repetition ::= singular "?" -> maybe // `a?` greedily matches `a` zero times or once
| singular "*" -> any // `a*` greedily matches `a` zero or more times
| singular "+" -> many // `a+` greedily matches `a` one or more times
| singular -> one ; /// `a` matches `a` exactly once. Must come last in the list of alternatives, because otherwise the other alternatives would never match (because alternatives pick the *first* match, not the longest one)
singular ::= "(" alternation ")" -> nested // A single literal pattern or referenced rule to match, or a parenthesized expression.
| symbol  
| string-literal  
| character-set  
| character-literal ;
string-literal :== "'" ( [^'##] | ## [^] )+ "'"  
| '"' ( [^"##] | ## [^] )+ '"' ; // `'...'` and `"..."` match `...` literally. Empty string literals are not supported; instead use `?`.
character-set :== "[" "^"? ( character-set-char  
( "-" character-set-char )?)*  
"]" ; /// `[...]` matches a single character within the specified set. `a-z` may be used to select a whole range of code points, and a `^` at the front of the `...` inverts the set.
character-set-char :== [^#^#-#]##] | escape-sequence ; /// Within a character set, the characters `^-]#` have special meaning. Prefix them with a `#` to match them literally.
character-literal :== escape-sequence ; /// Hash escape sequences for characters can also be matched outside the context of a string or character range.

escape-sequence :== ## (   // A hash escape sequence for a special character
[^xuU] |   //
"x" hex hex |  
"u" hex hex hex hex |  
"U" hex hex hex hex hex hex hex hex  
) ; /**  
`#t` short for `#x09` (tab)
`#n` short for `#x0A` (LF/newline)
`#r` short for `#x0D` (CR/carriage return)
`##` matches `#` literally  
`#'` `#"` matches `'` `"` literally (for use within `'`/`"`-delimited strings)
`#-` `#^` `#]` matches `-` ^` `]` literally (for use within character sets)
`#xHH` `#uHHHH` `#UH{8}` matches the given 8/16/32-bit code point , expressed with 2 / 4 / 8 hex digits */
hex :== [0-9a-fA-F] ; /// A hex digit
doc :== "/**" comment-body "*/" ; /// /** comments are treated as docstrings. They may only appear immediately in front of a rule or toplevel alternative in a grammar production rule.
_ :== "/*" ( [^*] comment-body )? "*/" ; /// Normal /* comments can appear anywhere
comment-body :== ( [^*] | "*"+ [^*/] )* "*"* ; /// The body of a comment. Some magic is needed here to ensure that the body doesn't match the comment end sequence anywhere in it; we can't just match any character because the start of the first comment in the file would greedily match right up to the end of the last comment in the file
_ :== [ #t#n#r]+ ; /// Whitespace can appear anywhere