Skip to content

Commit c5ca65b

Browse files
authored
fix: parse top-level SQL clauses (#28)
## Summary - add an offset-preserving, quote- and parenthesis-aware scanner for top-level ABAP SQL structure - parse `SELECT`, `FROM`, tail clauses, `UNION SELECT`, `UP TO`, and caller targets only at proven top level - preserve the original query as the only executable representation and keep full-query source authorization unchanged - add red/green coverage for literals, doubled quotes, nesting, multiline whitespace, comments, caller targets, authorization, and generated-source boundaries ## Root cause and invariant `QUERY_PARSE` searched raw user text with single-space substring and regex patterns. Literal, nested, comment, and multiline lookalikes could therefore move or remove clause boundaries before authorization and generation. The fix builds a same-length structural view, masks literal and nested content, and maps every structural offset back to the untouched original query. Invalid quote/parenthesis state, surviving comments, backticks, and templates fail closed. The structural view is never executed. ## TDD evidence - red commit `8ec4040`: NPL activated the original production code with 78 inherited tests green and exactly six new parser regressions red - exact source candidate `1b76755c1af07325ff50f76ffa12661bc45213d1` - source SHA-256 `8897954b5235e9c88c7e4bcd9653eccdd9f4b5235958792fe2c7e5b40cfbec1f` - local `npm ci`, `npm test`, and `git diff --check`: green - configured abaplint: 0 findings - diagnostic abaplint: 2,355 findings across 62 rules versus master 2,230; no increase in `dangerous_statement`, `ambiguous_statement`, `sql_escape_host_variables`, or `strict_sql` ## Live validation ### SAP_BASIS 750 / NPL - explicit activation and active syntax: green, zero errors - ABAP Unit: 91 passed, 0 failed - active/inactive source equal at server SHA-256 `fa0332e5679236d636bece64cbb3e28b3dccf2e8b21112c69efadd550ea78d0b` - no inactive ZTOAD main or child part - complete DEFAULT ATC unchanged: 88 findings (3 P1, 4 P2, 81 P3) ### S/4HANA 2023 / A4H - explicit activation and active syntax: green, zero errors - seven POSIX-regex deprecation warnings are the documented 758 signal for the 7.50-compatible API - ABAP Unit: 91 passed, 0 failed - active/inactive source equal at the same server hash; no inactive ZTOAD part - ABAP_CLOUD_READINESS unchanged: 682 findings (474 P1, 208 P2) - S4HANA_READINESS_2023 returned zero rows but remains prerequisite-incomplete - fresh WebGUI session received a multiline `CONCAT( 'FROM', ... )` query via real typing and blur, returned five expected `FROM...` values, and showed the backend success message - complete post-smoke ST22 set contained no ID newer than the `2026-08-07T06:26:10Z` snapshot ## Shared-system cleanup NPL and A4H were restored to exact master `0f056e8`. Both are active/inactive equal at server SHA-256 `36ebfcba68733ff91655ec552298af8ec5d19a203767aa88f34dedc67d7e6f83`, have no inactive ZTOAD part, and pass all 78 master tests. ## Compatibility and rollback The implementation uses only SAP_BASIS 750 syntax/APIs. Rollback is the source-only revert documented in `docs/plans/base-bug-004.md`; no DDIC metadata, table data, customizing, screen, transaction, or authorization object changed.
1 parent 0f056e8 commit c5ca65b

6 files changed

Lines changed: 805 additions & 56 deletions

File tree

‎AGENTS.md‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -46,6 +46,8 @@ If ARC-1's pre-write linter selects the wrong ABAP release, bypass only that loc
4646

4747
ZTOAD's report source is large enough that streaming a complete `SAPWrite` JSON payload through stdin can return `EAGAIN`. If that occurs, create a bounded temporary JSON payload file, pass its path to ARC-1, and verify the subsequent write before activation; never interpret the failed streamed request as a deployed candidate. Capture very large ATC JSON to a temporary file and emit an exact count/priority/prerequisite summary so terminal truncation cannot corrupt the evidence.
4848

49+
When automating the ARC-1 CLI, parse the returned MCP envelope as well as the process status: a tool failure can be returned as JSON with `isError: true` while the CLI process still exits with status 0. Treat that envelope as a failed gate and never continue to activation. `SAP_ALLOWED_PACKAGES` is the canonical allowlist variable, and a long-running MCP process does not acquire later environment/configuration changes; restart it or use a bounded fresh process after an allowlist change, then verify the target system before writing.
50+
4951
If ARC-1 authentication or a live system is unavailable, local checks may continue, but do not claim live validation. Record the missing gate in the PR and complete it before merge unless the maintainer explicitly accepts the risk.
5052

5153
NPL now has the complete ZTOAD object set in offline native-abapGit package `$ZTOAD2`, including the real transparent table. Never replace that table with a DDIC structure. Use ARC-1 for exact source deployment, explicit activation, syntax, Unit, ATC, and inactive-object checks. Structural refreshes require another reviewed native-abapGit ZIP import; do not export the local root package or accept release-normalized table metadata as a source change without a dedicated structural review. The missing table enhancement category is tracked separately as `BASE-DDIC-001`.

‎docs/baseline-findings.md‎

Lines changed: 11 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
# ZTOAD baseline findings
22

3-
_Snapshot: 2026-08-07 · branch `codex/fix-base-bug-003-v2` from merged master `e5ae759` · live system A4H client 001, SAP_BASIS 758 SP02 · secondary target NPL client 001, SAP_BASIS 750 SP02_
3+
_Snapshot: 2026-08-07 · branch `codex/fix-base-bug-004` from merged master `0f056e8` · live system A4H client 001, SAP_BASIS 758 SP02 · secondary target NPL client 001, SAP_BASIS 750 SP02_
44

55
This is the ordered work list for incremental TDD. It records observed defects separately from broad style debt so a new failure cannot disappear inside the legacy backlog. A finding is closed only after its regression test is green locally where possible, green on A4H, and green on the SAP_BASIS 750 target.
66

@@ -9,16 +9,16 @@ This is the ordered work list for incremental TDD. It records observed defects s
99
| Gate | Result | Interpretation |
1010
|---|---|---|
1111
| Repository `npm test` | Passed, 0 configured abaplint findings | Fast compatibility/XML gate is green |
12-
| Full abaplint quality profile | 2,230 findings across 62 rules | Exact master is 2,168; the candidate's +62 is limited to the default profile's conflicting clean-name/Hungarian-name rules; configured abaplint remains the zero-finding merge gate |
13-
| A4H SAP syntax | 0 errors, 2 POSIX warnings | Exact `BASE-BUG-003` candidate `6a2971b` activated successfully; both warnings are pre-existing compatibility findings |
14-
| A4H ABAP Unit | 78 passed, 0 failed | Exact candidate is green, including the new scanner and function generator corpus |
12+
| Full abaplint quality profile | 2,355 findings across 62 rules | Exact master is 2,230; the candidate delta is dominated by diagnostic style/naming rules from the focused scanner/test corpus. `dangerous_statement`, `ambiguous_statement`, `sql_escape_host_variables`, and `strict_sql` did not increase; configured abaplint remains the zero-finding merge gate |
13+
| A4H SAP syntax | 0 errors, 7 POSIX warnings | Exact `BASE-BUG-004` candidate `1b76755` activated successfully; the warnings are the 758 deprecation signal for the 7.50-compatible regex API and the candidate has no syntax error |
14+
| A4H ABAP Unit | 91 passed, 0 failed | Exact candidate is green, including clause-boundary, multiline, literal-lookalike, source-authorization, generator, and representation-boundary tests |
1515
| A4H ATC `S4HANA_READINESS_2023` | Initial run displayed 0 finding rows | Later evidence shows missing prerequisites; this is not an authoritative zero gate |
1616
| A4H ATC `ABAP_CLOUD_READINESS` | 682 findings: 474 P1, 208 P2 | Complete exact-candidate run; unchanged from merged master and retained as an architectural burn-down signal rather than a zero gate |
17-
| A4H WebGUI smoke | Separate fresh sessions executed sanitized `SUBSTRING` and nested `CONCAT`/`SUBSTRING` queries, each returning five DD03L rows with the expected aliases | Exact BASE-BUG-003 candidate is green; no new ST22 entry appeared after the pre-smoke dump-set marker |
18-
| NPL SAP syntax / ABAP Unit | 0 errors; 78 passed, 0 failed | Complete offline native-abapGit installation is green for the exact candidate on SAP_BASIS 750 |
17+
| A4H WebGUI smoke | A fresh session received a multiline `SELECT CONCAT( 'FROM', TABNAME ) AS LABEL FROM DD02L UP TO 5 ROWS` through real editor typing and blur; the result grid returned five `FROM...` values and the backend success message | Exact BASE-BUG-004 candidate is green; the complete post-smoke ST22 set had no ID newer than the `2026-08-07T06:26:10Z` marker |
18+
| NPL SAP syntax / ABAP Unit | 0 errors; 91 passed, 0 failed | Complete offline native-abapGit installation is green for the exact candidate on SAP_BASIS 750 |
1919
| NPL ATC `DEFAULT` | 88 findings: 3 P1, 4 P2, 81 P3 | Complete exact-candidate run; unchanged from the restored master baseline and tracked as legacy debt |
2020

21-
All three P0 security fixes, the installation closure, and the WebGUI capability adapter are merged and preserved in this candidate. Exact source commit `6a2971bded5ff273b82b1db4c4b15b84485612b6` has local SHA-256 `4f07726e262d06a42fa56257197bd22ce5d1654fbf64919e9069f2a28deb3256`; both systems reported equal active/inactive source at server-normalized SHA-256 `36ebfcba68733ff91655ec552298af8ec5d19a203767aa88f34dedc67d7e6f83`. NPL and A4H each passed 78/78 Unit tests. After evidence, both systems were restored and activated to exact master `e5ae759`, server SHA-256 `c7d4cbcfcb9a3af1d322e3f912c586dfee12b8af0fe608049807cfa61e94cccd`, with no inactive ZTOAD part and 67/67 master tests. `S4HANA_READINESS_2023` returned no rows but remains incomplete because its prerequisite checks are unavailable.
21+
All three P0 security fixes, the installation closure, the WebGUI capability adapter, and `BASE-BUG-003` are merged and preserved in this candidate. Exact source commit `1b76755c1af07325ff50f76ffa12661bc45213d1` has local SHA-256 `8897954b5235e9c88c7e4bcd9653eccdd9f4b5235958792fe2c7e5b40cfbec1f`; both systems reported equal active/inactive source at server-normalized SHA-256 `fa0332e5679236d636bece64cbb3e28b3dccf2e8b21112c69efadd550ea78d0b`. NPL and A4H each passed 91/91 Unit tests. After evidence, both systems were restored and activated to exact master `0f056e8`, server SHA-256 `36ebfcba68733ff91655ec552298af8ec5d19a203767aa88f34dedc67d7e6f83`, with no inactive ZTOAD part and 78/78 master tests. `S4HANA_READINESS_2023` returned no rows but remains incomplete because its prerequisite checks are unavailable.
2222

2323
## Ordered findings register
2424

@@ -33,7 +33,7 @@ Priority meanings: **P0** blocks the requested test workflow or is an immediate
3333
| BASE-BUG-001 | P1 | [Issue #6](https://github.com/marianfoo/ztoad/issues/6): a reported `SELECT DISTINCT ... COUNT ... MAX ... GROUP BY ... HAVING` query produces “The INTO/APPENDING clause must be at the end of the SELECT.” The first regression fixture used `COUNT( field )`, which is valid on 758 but rejected by the 750 compiler before clause ordering is checked. | Sanitized `DD03L` reproduction with the 7.50-compatible `COUNT( DISTINCT field )`; check exact generated ordering around `HAVING`, `ORDER BY`, `UP TO`, and `INTO TABLE`. | Fixed candidate: exact source is 67/67 on both 750 and 758; syntax/state and ATC deltas are green |
3434
| BASE-BUG-002 | P1 | [Issue #7](https://github.com/marianfoo/ztoad/issues/7): a reported `SUM( CASE ... END ) AS ...` over `EKBE` produces “No component exists with the name CASE.” The expression is split into false result components. | Sanitized aggregate with multi-line `CASE`, qualified fields, multiplication, alias, WHERE, and GROUP BY; assert one aggregate expression and the expected generated row type. | Fixed in PR #21: all 67 tests, including the CASE corpus, now pass on both 750 and 758; A4H WebGUI/ST22 green; Cloud ATC unchanged, S/4 ATC incomplete |
3535
| BASE-BUG-003 | P1 | [Issue #4](https://github.com/marianfoo/ztoad/issues/4): `SUBSTRING`/`CONCAT` produce “No component exists with the name SUBSTRING(”, exposing naïve select-list tokenization. | Nested function arguments, commas inside functions, aliases, literals containing spaces/commas, the complete 7.50 function family, and function composition. | Fixed in PR #27 candidate: 78/78 on both releases; live `SUBSTRING` and nested `CONCAT` results green; ATC unchanged; no new ST22 dump |
36-
| BASE-BUG-004 | P1 | Keyword detection uses string searches and space splitting rather than quote/parenthesis-aware lexical tokens. Keywords in literals, comments, functions, or subqueries can move clause boundaries. | Quoted `FROM`, `WHERE`, `UNION`, `UP TO`; escaped quotes; comments; nested parentheses; multiline queries. | Open |
36+
| BASE-BUG-004 | P1 | Keyword detection used raw string searches and single-space delimiters rather than quote/parenthesis-aware top-level structure, so literal, nested, comment, and multiline lookalikes could move clause boundaries. | Quoted/doubled-quote `FROM`, tail-keyword matrix, literal and real `UNION`/`UP TO`, comments, nested parentheses, caller targets, multiline queries, preserved source authorization, and live result execution. | Fixed candidate `1b76755`: 91/91 on both releases, syntax/state and ATC baselines green, real A4H WebGUI result green, no new ST22 dump |
3737
| BASE-BUG-005 | P1 | `UNION` splitting is whitespace-sensitive and does not model `UNION ALL`; branches are appended while assuming compatible result layouts. | `UNION`, `UNION ALL`, mismatched branch types/columns, per-branch limits, and keywords in literals/subqueries. | Open |
3838
| BASE-BUG-006 | P1 | [Issue #2](https://github.com/marianfoo/ztoad/issues/2): no serialized `TRAN ZTOAD` existed, so FLP/WebGUI could not launch a stable ZTOAD transaction directly. | Repository contract, native-abapGit round trip, direct FLP and standalone WebGUI launch, ST22 delta, and ADT-only 7.50 availability check. | Fixed: local contract green; native A4H object clean; both A4H launch paths reach ZTOAD with no new dump; NPL offline installation includes `TRAN ZTOAD` and UI execution remains intentionally out of scope. |
3939
| BASE-BUG-007 | P1 | [Issue #5](https://github.com/marianfoo/ztoad/issues/5): source-only/manual installation was incomplete. A4H had seven inactive ZTOAD composite parts even though main-source equality was green. Native-abapGit activation restored source, CUA/GUI status, four dynpros, and texts; current abapGit also moved dynpro flow logic into four separate files. | Local repository/flow-logic closure contract plus live inactive-inventory gate; coordinated native-abapGit pull/activation; selective clean round trip; `STATUS010`/read-only dispatch and ST22 smoke. | Fixed and merged in PR #22: both systems now have the complete object set and zero inactive ZTOAD parts. The separate post-dispatch runtime defect is fixed by `BASE-RUN-006`. |
@@ -62,12 +62,13 @@ This diagnostic profile is intentionally not the merge gate yet. Run `npm run li
6262

6363
## Initial executable tests
6464

65-
The program now ends with eight harmless, short ABAP Unit classes:
65+
The program now ends with nine harmless, short ABAP Unit classes:
6666

67-
- `LTC_QUERY_PARSER`: 23 tests for the original SELECT contract plus authorized/unauthorized nested sources, two-level nesting, ordinary and parenthesized joins, literal/identifier keyword lookalikes, scanner-level comment rejection, later `UNION`/`UNION ALL` sources, and fail-closed dynamic/CTE/attached-path/detached-path/hierarchy sources.
67+
- `LTC_QUERY_PARSER`: 31 tests for the original SELECT contract plus top-level clause boundaries, multiline input, literal keyword matrices, authorized/unauthorized nested sources, two-level nesting, ordinary and parenthesized joins, scanner-level comment rejection, later `UNION`/`UNION ALL` sources, and fail-closed dynamic/CTE/attached-path/detached-path/hierarchy sources.
6868
- `LTC_QUERY_INPUT_VALIDATOR`: 12 tests for the supported SQL-fragment alphabet, literal state, decimals/minus, and source-boundary rejection.
6969
- `LTCL_SELECT_EXPR_ANALYZER`: 7 tests for simple/searched CASE result references, qualified/unqualified columns, quoted `THEN` lookalikes, and fail-closed literal-only results.
7070
- `LTCL_SELECT_LIST_SCANNER`: 7 tests for top-level versus nested commas, doubled quotes, the complete 7.50 function family, literal lookalikes, expression closure, and fail-closed unbalanced/trailing input.
71+
- `LTCL_SQL_CLAUSE_SCANNER`: 5 tests for same-length offset preservation, literals and nested parentheses, multiline whitespace normalization, and fail-closed quote/parenthesis/comment/backtick/template states.
7172
- `LTC_QUERY_GENERATOR`: 12 tests for function and CASE generation/metadata/fail-closed typing, strict clause ordering, escaped-count and legacy-select compatibility, statement-injection rejection, and doubled-quote wrapping rejection.
7273
- `LTC_LINE_SPLITTER`: 6 tests for short/exact/long safe lines, unsafe hard-cut and literal-space rejection, and column-one comment prevention.
7374
- `LTC_COMMAND_PARSER`: 7 tests for UPDATE, DELETE FROM, default/legacy-flag Native SQL rejection, DML statement-injection rejection, invalid-syntax return semantics, and preserved valid DML generation.

‎docs/development.md‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -96,6 +96,8 @@ If ARC-1's pre-write linter reports an incorrect ABAP release, do not weaken all
9696

9797
For the large ZTOAD report, an ARC-1 stdin write can fail with `EAGAIN` before SAP receives the source. Retry through a bounded temporary JSON payload file, then require a successful write response and explicit activation. Large ATC result JSON can exceed terminal output limits; capture the complete result first and derive exact finding counts, priority counts, and prerequisite errors from that complete payload rather than parsing truncated console text.
9898

99+
ARC-1 CLI automation must validate both layers of its result. A tool-level failure can be serialized as an MCP JSON envelope with `isError: true` even when the CLI process exits with status 0; treat either signal as failure and do not issue the subsequent activation. Use the canonical `SAP_ALLOWED_PACKAGES` variable for the write allowlist. Because an already-running MCP process keeps the environment it started with, restart it or launch a bounded fresh process after changing an allowlist or target configuration, and verify the intended system before the first write.
100+
99101
## 5. Structural object changes
100102

101103
Examples include adding issue #2's transaction code, changing the table definition, changing dynpros or text elements, and adding a global class.

0 commit comments

Comments
 (0)