Skip to content

Add schema inference and parser test workbench #34

Description

@jusso-dev

Problem

Custom logs are hard to onboard safely. Analysts and admins need a workbench that can inspect sample logs, infer a draft mapping, test parser output, and show rejected records before a source is enabled.

Scope

Build a local schema/parser workbench for sample JSON, JSONL, CSV-with-header, syslog/CEF, and key-value logs. The first version can infer field paths and propose a mapping to Tawny-SOC canonical event shape.

Implementation Notes

  • Add /settings/ingestion or extend the existing ingestion settings section.
  • Inputs: paste sample logs, upload sample file if supported by current stack, choose stream type or auto-detect.
  • Outputs: detected format, inferred field list, proposed canonical mappings, parser result preview, rejected record reasons, canonical indicators extracted.
  • Add a server-side parser test endpoint that reuses lib/ingestion/normalizers.ts.
  • Store draft parser configs but require explicit enablement before accepting source traffic.

Acceptance Criteria

  • A user can paste sample logs and preview normalized Tawny-SOC events.
  • Rejected records include actionable reasons.
  • The workbench shows which canonical indicators would be extracted.
  • Draft mappings are editable before saving.
  • Saved parser configs are disabled until explicitly enabled.

Tests

  • Unit test format detection for supported sample types.
  • Unit test parser preview returns accepted and rejected samples.
  • API test invalid sample input is handled without throwing.
  • Component test preview renders normalized fields and rejection reasons.

Metadata

Metadata

Assignees

No one assigned

    Labels

    area:ingestionLog ingestion sources and transportsarea:normalizationParsing and canonical event normalizationarea:uiUser interface workpriority:mediumMedium priority implementation work

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions