#1 (comment)
Improve error handling and error logging all over the codebase to ensure that:
- Data ingestion pipeline doesn't keep going when critical errors arise
- ...but also doesn't terminate for simple errors
(maybe we can implement a logic to configure this in a modular manner, e.g. should_break["..."] = True for errors that arise in specific places within the pipeline (not specific errors, but rather specific functions and/or places))
- Most importantly, all errors and warnings should explicitly appear and be traceable
#1 (comment)
Improve error handling and error logging all over the codebase to ensure that:
(maybe we can implement a logic to configure this in a modular manner, e.g.
should_break["..."] = Truefor errors that arise in specific places within the pipeline (not specific errors, but rather specific functions and/or places))