Extract canonical indicator fields during ingestion

[jusso-dev]

Problem

Hunt and triage workflows benefit from consistent pivots across source types. Today, important entities are available in specific fields, but Tawny-SOC should extract canonical indicator arrays during ingestion for fast cross-source search and correlation.

Scope

Add canonical indicator extraction for IP addresses, domains, URLs, hashes, usernames, emails, hostnames, process names, cloud account IDs, and MITRE techniques.

Implementation Notes

• Extend normalized event shape with canonical arrays: anyIps, anyDomains, anyUrls, anyHashes, anyUsers, anyEmails, anyHosts, anyProcesses, anyCloudAccounts, anyMitreTechniques.
• Extract from known first-class fields and selected payload paths.
• Normalize casing and dedupe values.
• Avoid storing invalid indicators.
• Update YAAQL aliases so has:ip, ip:, domain:, hash:, user:, and mitre: prefer canonical fields.
• Backfill seed data or make extraction happen at read time for old records until migration is available.

Acceptance Criteria

• New ingested records include canonical indicator arrays.
• YAAQL indicator aliases search canonical arrays across all supported source types.
• Alert detail and Hunt pivots use canonical indicators where possible.
• Duplicate indicator values are stored once.
• Invalid IPs, malformed hashes, and empty strings are ignored.

Tests

• Unit test indicator extraction across generic JSON, Windows, CloudTrail, Azure, M365, and firewall records.
• Unit test dedupe and validation.
• YAAQL tests for canonical indicator aliases.
• Regression test existing normalizer outputs remain compatible.