Skip to content

Add freshness test #5

@dbrtly

Description

@dbrtly

Summarize the feature in question.
Tablestakes member of the suite of built-in tests. Low effort task to provide more complete conventional functionality.

imperative description for clarity of understanding:
Create a variable ‘watermark’.
Read table metadata (engine specific) for most_recent_update or equivalent. Revert and option to calculate the maximum value of a nominated column
Calculate the interval between the watermark and current_date()/current_datetime()/current_timestamp(). if interval > error_interval:
raise error
elsif interval > warn_interval
warn

**What team(s) will benefit from this feature request?
Conventional test for developers familiar with other sql frameworks.

Is this an enhancement to an existing feature? If so, briefly describe the update.
add another macro

Describe alternatives you've considered if applicable.

Technically something like this would work, especially is all incremental models had a start and end
select count(*) as cnt from my_tbl where my_col between start and end having not cnt between minimum_batch_size and maximum_batch_size

On the principal that the simple thing should be simple, a macro that compares a maximum date/dt/ts to the relevant variation of current_datatype() and compares the interval to warning/error tolerances is easier.

Additional context
Happy to contribute some code.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions