Simple lambda

Makefile

@@ -252,7 +252,7 @@ $(TEST_DEP): $(DEPDIR)/%.d: $(TSTDIR)/src/%.c .generated | $(DEPDIR)
 
 test: $(TEST_TARGETS) $(TARGET) $(UNIDIR)/unicode.db
 	for t in $(TSTDIR)/fn/test_*.fn ; do echo '***' $$t '***' ; ./$(TARGET) --include=fn --assertions-accumulate $$t || exit 1 ; done
-	for t in $(TSTDIR)/fn/fail_*.fn ; do echo '***' $$t '(expect to see an error) ***' ; ! ./$(TARGET) --include=fn --assertions-accumulate $$t || exit 1 ; done
+	for t in $(TSTDIR)/fn/fail_*.fn ; do echo '***' $$t '***' ; ! ./$(TARGET) --include=fn --assertions-accumulate $$t >/dev/null 2>&1 || exit 1 ; done
 	for t in $(TEST_TARGETS) ; do echo '***' $$t '***' ; $$t || exit 1 ; done
 	@echo "All tests passed."
 

README.md

@@ -84,7 +84,7 @@ oi --> scanner
 parser --> ast(AST) -->
 lc([Lambda Conversion]):::process --> tpmc([Pattern Matching Compiler]):::process
 lc <---> pg([Print Function Generator]):::process
-lc <---> me([Macro Expansion]):::process
+lc <---> me([Lazy Function Expansion]):::process
 tpmc --> vs([Variable Substitution]):::process
 vs --> lc
 lc <--> des([Desugaring]):::process
@@ -97,7 +97,9 @@ tc --> lambda2(Plain Lambda Form)
 lambda2 --> ci([Constructor Inlining]):::process
 ci --> lambda3(Inlined Lambda)
 subgraph anf-rewrite-2
-   alpha(["ɑ-Conversion"]):::process
+   desugaring(["Desugaring"]):::process
+   desugaring --> lambda_ds(desugared lambda)
+   lambda_ds --> alpha(["ɑ-Conversion"]):::process
    alpha --> lambda_a(alphatized lambda)
    lambda_a --> anfr([ANF Rewrite]):::process
    anfr --> lambda_b(New ANF)
@@ -109,7 +111,7 @@ subgraph anf-rewrite-2
    lambda_c --> betar(["β-Reduction WiP"]):::process
    betar --> lambda_d(simplified)
 end
-lambda3 --> alpha
+lambda3 --> desugaring
 lambda3 --> anfc
 lambda_a --> anfc([A-Normal Form Conversion]):::process
 anfc --> anf(ANF)
@@ -124,14 +126,15 @@ bc --> cekf([CEKF Runtime VM]):::process
 The "anf-rewrite-2" section is a WiP on the `anf-rewrite-2` branch. Although that branch started as a rewrite of the ANF transform, it became apparent that the CEK machine itself was blocking optimizations and so the intention is to target a more "traditional" register machine with an eye towards LLVM in the longer term. On that branch the ɑ-conversion is complete and incorporated (though it achieves nothing for the ANF path it is required for CPS.) The ANF rewrite is complete but abandoned, and the CPS transform is also complete.
 
 The various components named in the diagram above are linked to their implementation entry point here:
+
 * Scanner [pratt_scanner.c](src/pratt_scanner.c)
 * Parser [pratt_parser.c](src/pratt_parser.c)
 * AST [ast.yaml](src/ast.yaml)
 * Lambda Conversion [lambda_conversion.c](src/lambda_conversion.c)
 * Tpmc [tpmc_logic.c](src/tpmc_logic.c)
 * Print Function Generator [print_generator.c](src/print_generator.c)
 * Variable Substitution [lambda_substitution.c](src/lambda_substitution.c)
-* Macro Expansion [macro_substitution.c](src/macro_substitution.c)
+* Lazy Function Expansion [lazy_substitution.c](src/lazy_substitution.c)
 * Plain Lambda Form [lambda.yaml](src/lambda.yaml)
 * Simplification [lambda_simplify.c](src/lambda_simplify.c)
 * Type Checking [tc_analyze.c](src/tc_analyze.c)

docs/BETTER-EXTERNAL.md

@@ -0,0 +1,41 @@
+# Proposal for Replacing the `external` Section in Yaml Files
+
+Currenty, and embarrassingly, the `external` section in the yaml files is just a synonym for the `primitives` section. The intention of the `external` section was to describe specifically memory-managed types from other yaml files, but that requires adding information that is already available from those files.
+
+The idea then is quite simple: rather than the `external` section listing details of those other types:
+
+```yaml
+external:
+    TcType:
+        meta:
+            brief: external type from the type-checker
+            description: A type-checker type referenced by the ANF code.
+        data:
+            cname: "struct TcType *"
+            printFn: printTcType
+            markFn: markTcType
+            valued: true
+    IntMap:
+        meta:
+        ...
+```
+
+it would just contain references to the files themselves:
+
+```yaml
+external:
+- !include tc.yaml
+```
+
+Those includes could be parsed recursively by the existing parser and added to the current catalog.
+This causes a couple of problems, but they should be easy enough to solve:
+
+1. How to avoid generating code for the external nodes? Each entry in the catalog has an additional `external` flag, if true it is skipped over by the catalog dispatchers.
+2. How to avoid mutually recursive includes? The `include` feature of the yaml is written in `generate/loader.py` and that code could be extended to quietly return an empty object if it sees a file it has already (started to) load.
+3. How to avoid primitives being re-entered and causing duplicates? Actually the same solution for recursive includes, `primitives.yaml` would only be parsed once.
+
+I'd imagine that the entire parse and catalog-injection section in `generate.py` would become a function that gets handed an object resulting from a yaml file, along with an `external` boolean flag. It would call itself recursively with each element of any `internal` section that it finds.
+
+The `config` section could be ignored or only partially used if it is `external`.
+
+Of course there wil likely be other problems, probably `#include` directives will need looking at, but the advantages of having direct access to the original definitions is obvious.

docs/OPERATORS.md

@@ -1,5 +1,9 @@
 # Operators (and macros)
 
+Important update if you are reading this: the keyword `macro` has since been
+replaced with the sequence `lazy fn` in the parser to better reflect its
+nature. However the semantics are unchanged.
+
 Some issues with the initial implementation.
 
 I'd thought I could get away with a pure parser-only implementation of

docs/TODO.md

@@ -2,8 +2,15 @@
 
 More of a wish-list than a hard and fast plan.
 
+* Simplify
+  * Add a beta-reduction pass (after cps-transform).
+  * Add an eta-reduction pass.
+  * Add a constant/operator folding pass after beta and eta reduction.
 * Target LLVM
 * `syntax` construct that allows large-scale syntactic structures to be defined by the user.
+* Clean Up.
+  * Generate
+    * Move all signatures into `signature_helper.py`, not just the shared ones.
 * More numbers:
   * NaN for division by Zero etc.
   * Matrices.
@@ -29,4 +36,3 @@ More of a wish-list than a hard and fast plan.
 * (internal) have a NEWZ variant of NEW that bzero's its result.
 * Builtins
   * `now()` builtin returns current time in milliseconds.
-  * `iter()` returns consecutive integers starting from 0

docs/agent/anf.md

@@ -0,0 +1,138 @@
+# ANF (Administrative Normal Form) Conversion
+
+Converts lambda expressions to A-Normal Form where all intermediate computations are named.
+
+## What ANF Does
+
+Transforms nested expressions into a flat sequence of let-bindings where:
+
+- **Atomic expressions (aexp)**: Variables, constants, lambdas - always terminate, never error
+- **Complex expressions (cexp)**: Function applications, conditionals - may not terminate or may error
+- All complex subexpressions become let-bound temporary variables
+
+Example transformation:
+
+```scheme
+(a (b c) (d e))
+=>
+(let (t$1 (d e))
+  (let (t$2 (b c))
+    (a t$2 t$1)))
+```
+
+## The Algorithm (from Matt Might's blog)
+
+**Core Idea**: Walk expressions depth-first, replacing complex subexpressions with fresh variables, accumulating let-bindings on the way back out.
+
+**Key functions** (all in `src/anf_normalize.c`):
+
+- `normalize(LamExp, tail)` - Main entry point, dispatches on LamExp type
+- `replaceLamExp(LamExp, replacements)` - Converts LamExp to Aexp, accumulating replacements
+- `letBind(body, replacements)` - Wraps body in let-bindings from replacements table
+- `wrapTail(exp, tail)` - Optionally wraps expression in additional let-binding
+
+**The `tail` parameter**: Continuation-like - represents the "rest of the computation" to wrap the current expression in. NULL means this is the final result.
+
+## Implementation Pattern
+
+Most normalize functions follow this pattern:
+
+1. Create a `LamExpTable` for tracking replacements (hash table mapping fresh symbols to LamExps)
+2. Call `replaceLamExp()` on subexpressions, which:
+   - If subexpr is atomic (var/constant), return it as Aexp
+   - If subexpr is complex (application), generate fresh symbol, add to replacements, return symbol as Aexp
+3. Build the ANF construct with replaced Aexps
+4. Call `wrapTail(exp, tail)` to optionally wrap in outer binding
+5. Call `letBind(exp, replacements)` to wrap in all accumulated let-bindings
+6. Return the wrapped expression
+
+## Critical Data Flow
+
+```text
+LamExp (lambda.yaml)
+  ↓ normalize()
+  → replaceLamExp() + LamExpTable
+    → Aexp (atomic expressions)
+  → Build ANF structure (Exp/Cexp)
+  → wrapTail()
+  → letBind() - wraps in let-bindings
+  ↓
+Exp (anf.yaml)
+```
+
+## Known Complexity Issues
+
+1. **Deeply Nested Functions**: The normalize functions have 30+ dispatch cases, one per LamExp type. Each follows slightly different logic.
+
+2. **GC Protection Overhead**: Extensive use of PROTECT/UNPROTECT macros throughout due to allocations during traversal. Easy to get wrong.
+
+3. **Tail Threading**: The `tail` parameter threads through recursion but its purpose isn't always clear. Sometimes NULL, sometimes accumulated let-bindings.
+
+4. **Dual Type System**: Must track both LamExp (input) and Aexp/Cexp/Exp (output) simultaneously. Easy to confuse which type is which.
+
+5. **Replacements Table**: The `LamExpTable` accumulates symbol→LamExp mappings that become let-bindings, but lifetime and scope isn't always obvious.
+
+## Debugging ANF
+
+```bash
+# Enable debug output
+# Uncomment DEBUG_ANF in src/common.h
+
+# Dump ANF for inspection
+./bin/fn --dump-anf path/to/file.fn
+```
+
+**Watch for**:
+
+- Incorrect nesting of let-bindings
+- Fresh symbol collisions (shouldn't happen but indicates `freshSymbol()` issues)
+- GC crashes (usually from missing PROTECT/UNPROTECT)
+- Type mismatches between LamExp and ANF structures
+
+## Potential Improvements
+
+1. **Simplify normalize dispatch**: Could the 30+ cases share more common code?
+2. **Clearer tail semantics**: Document when tail is NULL vs. non-NULL
+3. **Reduce PROTECT overhead**: Could intermediate allocations be batched?
+4. **Better error messages**: When ANF conversion fails, why?
+5. **Refactor replacements**: The hash table approach works but is it the clearest?
+
+## Key Files
+
+- `src/anf_normalize.c` - The implementation (1100+ lines)
+- `src/anf.yaml` - ANF data structures (Exp, Aexp, Cexp)
+- `src/lambda.yaml` - Input lambda structures
+- `docs/ANF.md` - Original algorithm notes
+
+## References
+
+- [Matt Might's ANF blog post](https://matt.might.net/articles/a-normalization/)
+
+## Tail Recursion & Wrapping Pitfalls
+
+**Correct tail wrapping is critical**. The `tail` parameter in `normalize` functions represents the "continuation" or "context" that the current expression should return to.
+
+### Incorrect Wrapping (Breaks Tail Recursion)
+
+Wraps the result `t$1` in a new `let` *after* the recursive call returns, forcing a stack frame.
+
+```scheme
+;; Source: tail_call(x)
+;; Bad ANF:
+(let (t$1 (tail_call x))
+  t$1) 
+;; This is NOT a tail call!
+```
+
+### Correct Wrapping (Preserves Tail Recursion)
+
+If `tail` is passed down correctly, the recursive call becomes the body of the `let` chain.
+
+```scheme
+;; Source: tail_call(x)
+;; Good ANF:
+(tail_call x)
+;; No wrapping, jumps directly
+```
+
+**Rule of Thumb**: When normalizing a function call, if it is in tail position (i.e., `tail` parameter is NULL or empty identity), **do not bind it to a variable** just to return that variable. Return the `AppExp` (or `Cexp`) directly.