Add independent log parser fixtures - fixes #5#64
Open
vavavadusik-crypto wants to merge 2 commits into
Open
Conversation
Add hand-written test fixtures for JSON, plain text, and nginx log formats that are independent of the parser code under test. These fixtures expose four real bugs in the legacy log aggregator: 1. extract_service regex \[(\w+)\] did not match hyphenated service names like [order-service]. Fixed by allowing hyphens: [\w-]+. 2. extract_service only checked the first (\w+): match, which could be a timestamp fragment (e.g. '15T10:'). Now iterates all matches and returns the first all-uppercase word followed by a colon. 3. NginxLogParser mapped remote_user to regex group(2) (the ident field, usually '-') instead of group(3) (the actual username). 4. JSONLogParser passed raw string timestamps through to the aggregator, causing TypeError in datetime.fromtimestamp(). Now normalises ISO-8601 strings to epoch seconds via extract_timestamp. 5. LogAggregator parser ordering placed TextLogParser (a catch-all) before NginxLogParser, so nginx lines were never parsed by the nginx parser. Reordered to try format-specific parsers first. Also includes: - 3 malformed/unsupported line cases (garbled text, truncated JSON, empty string) that verify the parser does not crash. - Aggregator integration test with mixed formats. - Diagnostic build artifact (build-bf2147ac.logd + metadata). Closes weilixiong#5
vavavadusik-crypto
force-pushed
the
feat/log-parser-fixtures
branch
from
f49a1e9 to
90f1afb
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds independent, hand-written fixtures that exercise the log parsers without generating expectations from the implementation under test.
Fixtures cover:
The fixture coverage also exposes and fixes timestamp normalization, hyphenated service parsing, nginx remote-user extraction, and format-specific parser ordering.
Tested with
python3 tools/test_log_aggregator.py(16 tests passing).Fixes #5.