-
Notifications
You must be signed in to change notification settings - Fork 3
Description
Summarize the feature in question.
Tablestakes member of the suite of built-in tests. Low effort task to provide more complete conventional functionality.
imperative description for clarity of understanding:
Create a variable ‘watermark’.
Read table metadata (engine specific) for most_recent_update or equivalent. Revert and option to calculate the maximum value of a nominated column
Calculate the interval between the watermark and current_date()/current_datetime()/current_timestamp(). if interval > error_interval:
raise error
elsif interval > warn_interval
warn
**What team(s) will benefit from this feature request?
Conventional test for developers familiar with other sql frameworks.
Is this an enhancement to an existing feature? If so, briefly describe the update.
add another macro
Describe alternatives you've considered if applicable.
Technically something like this would work, especially is all incremental models had a start and end
select count(*) as cnt from my_tbl where my_col between start and end having not cnt between minimum_batch_size and maximum_batch_size
On the principal that the simple thing should be simple, a macro that compares a maximum date/dt/ts to the relevant variation of current_datatype() and compares the interval to warning/error tolerances is easier.
Additional context
Happy to contribute some code.