Skip to content

Commit 80196ed

Browse files
pirapiraclaude
andcommitted
Implement Phase 2 parser: PEG recursive-descent parser for Python 3.12
Build a complete parser that transforms the lexer's token stream into a Python 3.12 AST. This establishes the source → tokens → AST pipeline needed for the interpreter. AST types (Lython/AST/Types.lean): - Expr (27 variants), Stmt (25 variants) in mutual inductive blocks - Supporting types: Comprehension, Arguments, Arg, CallKeyword, ExceptHandler, FunctionDef, ClassDef, Alias, WithItem, Module - Dump functions for test assertions via string comparison Parser infrastructure: - ParserM monad (StateT over Except) with token array + index - PEG combinators: attempt/backtrack, many, sepBy, optional - Token matching helpers for keywords, operators, delimiters, literals Expression parser (18 precedence levels): - Atoms through named expressions, full operator precedence - Primaries: attribute access, calls, subscripts, slicing - Comprehensions (list/set/dict), implicit string concatenation - Boolean/comparison chaining, ternary if, lambda, yield, starred Statement parser: - Simple: assignment, augmented/annotated assignment, return, pass, break, continue, raise, assert, global, nonlocal, del, import - Compound: if/elif/else, while, for, with, try/except/finally, def, class, async, decorators - Block parsing via NEWLINE/INDENT/DEDENT tokens - Function parameter parsing (*args, **kwargs, defaults, annotations) 70+ parser tests covering expressions, statements, and integration. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
1 parent 632d406 commit 80196ed

13 files changed

Lines changed: 2122 additions & 9 deletions

File tree

CLAUDE.md

Lines changed: 15 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -24,21 +24,34 @@ Lython/
2424
StringLit.lean — String/bytes/raw/f-string literal lexing
2525
Indent.lean — INDENT/DEDENT generation from leading whitespace
2626
Core.lean — Main tokenization loop, operator dispatch
27-
Parser.lean — PEG parser (stub)
28-
AST.lean — Python AST node types (stub)
27+
AST.lean — AST umbrella import
28+
AST/
29+
Types.lean — Expr, Stmt, Module, and supporting types (mutual inductives)
30+
Parser.lean — parser umbrella import
31+
Parser/
32+
Types.lean — ParseError, ParserState, ParserM monad
33+
Combinators.lean — PEG combinators (attempt, many, sepBy, etc.)
34+
Tokens.lean — Token matching helpers (expectKeyword, parseName, etc.)
35+
Expr.lean — Expression parser (18 precedence levels)
36+
Stmt.lean — Statement parser (simple + compound statements)
37+
Core.lean — Entry point: parse : String → Except String Module
2938
Interpreter.lean — tree-walking interpreter (stub)
3039
Runtime.lean — runtime support (stub)
3140
Main.lean — CLI entry point
3241
LythonTest.lean — test driver root
3342
LythonTest/
3443
Basic.lean — lexer tests (keywords, operators, numbers, strings, indent)
44+
Parser.lean — parser tests (expressions, statements, integration)
3545
```
3646

3747
## Code Style
3848

3949
- `set_option autoImplicit false` at the top of every file
4050
- No trailing whitespace
4151
- Follow existing patterns in the codebase
52+
- Do NOT use `/-! ... -/` module doc comments inside `mutual` blocks (they are commands, not comments; use `-- ...` instead)
53+
- Use `partial def` for all recursive parser functions
54+
- Structures cannot be in `mutual` blocks; use `inductive Foo where | mk : ...` instead
4255

4356
## Development Plan
4457

Lython/AST.lean

Lines changed: 2 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,3 @@
1-
set_option autoImplicit false
1+
import Lython.AST.Types
22

3-
namespace Lython.AST
4-
end Lython.AST
3+
set_option autoImplicit false

0 commit comments

Comments
 (0)