[Streams 🌊] Enrichment - Fix broken results due to condition and add skipped metric #212757
Conversation
tonyghiani
added
release_note:skip
|
Pinging @elastic/obs-ux-logs-team (Team:obs-ux-logs) |
| @@ -609,6 +637,11 @@ const isSuccessfulProcessor = ( | |||
| ): processor is WithRequired<IngestPipelineSimulation, 'doc' | 'tag'> => | |||
| processor.status === 'success' && !!processor.tag; | |||
|
|
|||
| const isSkippedProcessor = ( | |||
| processor: IngestPipelineSimulation | |||
| // @ts-expect-error Looks like the IngestPipelineSimulation.status is not typed correctly and misses the 'skipped' status | |||
There was a problem hiding this comment.
Looks like it's using WatcherActionStatusOptions which seems very wrong... Could you report this?
There was a problem hiding this comment.
Yep this is wrong, I'll report that! 👌
There was a problem hiding this comment.
I'll look into this today, if I see it's a simple change I'll probably fix it here and ping you for a final check |
|
@flash1293 I added the changes to handle nested fields in the condition and seems to work well, this is ready for another check 👌 |
Yeah I did it cause we introduces the new status of "skipped" document, and this wasn't accounted for anywhere, so I added there to at least put it in a team. I already discussed with Luca offline about this, and we might want to improve the filters on the table to match all of them: parsed, partially parsed, skipped, failed. It's a temporary change to get the matched/unmatched rates compensate each other correctly. |
|
But you just changed this in the last commit, right? 950fca0 |
💚 Build Succeeded
Metrics [docs]Async chunks
History
|
|
Starting backport for target branches: 8.x |
…skipped metric (elastic#212757) ## 📓 Summary When the condition is not met, the processing simulation reports wrong metrics and fails on a unhandler error. This work fix the issue and also update the document simulation metrics, reporting how many documents are skipped by a processor during the simulation. A follow-up work will update the filters on the date to better reflect the available states of the documents (parsed, partially parsed, skipped, failed). <img width="701" alt="Screenshot 2025-02-28 at 12 47 10" src="https://github.com/user-attachments/assets/1b6979e4-78a1-4db3-af72-faaf06c0e249" /> (cherry picked from commit 6e2a103)
💚 All backports created successfully
Note: Successful backport PRs will be merged automatically after passing CI. Questions ?Please refer to the Backport tool documentation |
📓 Summary
When the condition is not met, the processing simulation reports wrong metrics and fails on a unhandler error.
This work fix the issue and also update the document simulation metrics, reporting how many documents are skipped by a processor during the simulation.
A follow-up work will update the filters on the date to better reflect the available states of the documents (parsed, partially parsed, skipped, failed).