feat: replace custom filter engine with tokf-filter crate

CHANGELOG.md

@@ -122,6 +122,12 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ### Features
 
+* **toml-dsl:** replace RTK's custom 8-stage filter engine with `tokf-filter` crate
+  * Full backward compatibility — all 47 built-in filters and user-authored `.rtk/filters.toml` files work unchanged
+  * RTK keeps its TOML parsing/registry/matching layer; delegates filtering to `tokf-filter::apply()`
+  * RTK handles `head_lines`/`tail_lines`/`max_lines` with omission markers (tokf silently truncates)
+  * `truncate_lines_at` now uses `…` (unicode ellipsis) instead of `...` (3 ASCII dots)
+  * Unlocks tokf's advanced features for user-authored filters: sections, chunks, aggregates, templates, JSON extraction, `on_success`/`on_failure` branches, `dedup`
 * **toml-dsl:** declarative TOML filter engine — add command filters without writing Rust ([#299](https://github.com/rtk-ai/rtk/issues/299))
   * 8 primitives: `strip_ansi`, `replace`, `match_output`, `strip/keep_lines_matching`, `truncate_lines_at`, `head/tail_lines`, `max_lines`, `on_empty`
   * lookup chain: `.rtk/filters.toml` (project-local) → `~/.config/rtk/filters.toml` (user-global) → built-in filters

Cargo.toml

@@ -32,6 +32,8 @@ ureq = "2"
 hostname = "0.4"
 flate2 = "1.0"
 quick-xml = "0.37"
+tokf-filter = { version = "0.2.33", default-features = false }
+tokf-common = "0.2.33"
 which = "8"
 
 [build-dependencies]

docs/filter-workflow.md

@@ -40,14 +40,15 @@ flowchart TD
         S -->|"✅ match"| U["exec command\ncapture stdout"]
         U --> V
 
-        subgraph PIPELINE ["8-stage filter pipeline"]
-            V["strip_ansi"] --> W["replace"]
-            W --> X{"match_output\nshort-circuit?"}
-            X -->|"✅ pattern matched"| Y[["emit message\nstop pipeline"]]
-            X -->|"no match"| Z["strip/keep_lines"]
-            Z --> AA["truncate_lines_at"]
-            AA --> AB["tail_lines"]
-            AB --> AC["max_lines"]
+        subgraph PIPELINE ["tokf-filter pipeline + RTK post-processing"]
+            V{"match_output\nshort-circuit?\n(strip_ansi applied\nif enabled)"} -->|"✅ pattern matched"| Y[["emit message\nstop pipeline"]]
+            V -->|"no match"| W["replace\n(replace_all mode)"]
+            W --> WA["strip_ansi\ntrim_lines"]
+            WA --> Z["skip/keep lines"]
+            Z --> ZA["dedup"]
+            ZA --> AA["truncate_lines_at"]
+            AA --> AB["head/tail_lines\n(RTK adds omission\nmarkers)"]
+            AB --> AC["max_lines\n(RTK adds truncation\nmarker)"]
             AC --> AD{"output\nempty?"}
             AD -->|"yes"| AE[["emit on_empty"]]
             AD -->|"no"| AF[["print filtered\noutput + exit code"]]

src/filters/README.md

@@ -3,6 +3,9 @@
 Each `.toml` file in this directory defines one filter and its inline tests.
 Files are concatenated alphabetically by `build.rs` into a single TOML blob embedded in the binary.
 
+Filtering is powered by the [tokf-filter](https://crates.io/crates/tokf-filter) engine.
+RTK handles the TOML registry, command matching, and `head/tail/max_lines` omission markers.
+
 ## Adding a filter
 
 1. Copy any existing `.toml` file and rename it (e.g. `my-tool.toml`)
@@ -37,15 +40,27 @@ expected = "expected filtered output"
 | `description` | string | Human-readable description |
 | `match_command` | regex | Matches the command string (e.g. `"^docker\\s+inspect"`) |
 | `strip_ansi` | bool | Strip ANSI escape codes before processing |
-| `strip_lines_matching` | regex[] | Drop lines matching any regex |
-| `keep_lines_matching` | regex[] | Keep only lines matching at least one regex |
-| `replace` | array | Regex substitutions (`{ pattern, replacement }`) |
-| `match_output` | array | Short-circuit rules (`{ pattern, message }`) |
-| `truncate_lines_at` | int | Truncate lines longer than N characters |
-| `max_lines` | int | Keep only the first N lines |
-| `tail_lines` | int | Keep only the last N lines (applied after other filters) |
+| `strip_lines_matching` | regex[] | Drop lines matching any regex (alias: `skip`) |
+| `keep_lines_matching` | regex[] | Keep only lines matching at least one regex (alias: `keep`) |
+| `replace` | array | Regex substitutions (`{ pattern, replacement }`) — all occurrences per line |
+| `match_output` | array | Short-circuit rules (`{ pattern, message, unless }`) |
+| `truncate_lines_at` | int | Truncate lines longer than N characters (uses `…` ellipsis) |
+| `head_lines` | int | Keep only the first N lines (with `"... (N lines omitted)"` marker) |
+| `tail_lines` | int | Keep only the last N lines (with omission marker) |
+| `max_lines` | int | Absolute line cap after head/tail (with `"... (N lines truncated)"` marker) |
 | `on_empty` | string | Fallback message when filtered output is empty |
 
+## Pipeline order
+
+1. `match_output` — short-circuit if pattern matches (ANSI-stripped when `strip_ansi = true`)
+2. `replace` — regex substitutions, all occurrences per line, rules chained sequentially
+3. `strip_ansi` — remove ANSI escape codes
+4. `skip`/`keep` — filter lines by regex
+5. `truncate_lines_at` — truncate long lines
+6. `head_lines`/`tail_lines` — keep first/last N lines (with omission markers)
+7. `max_lines` — absolute line cap
+8. `on_empty` — message if result is empty
+
 ## Naming convention
 
 Use the command name as the filename: `terraform-plan.toml`, `docker-inspect.toml`, `mix-compile.toml`.