v1.0.8

What's Changed

• updating dbt connection timeout by @shourya116 in #243

Full Changelog: v1.0.7...v1.0.8

v1.0.7

What's Changed

• fixed complex_rule_validation_error_count by @shourya116 in #236
• docs: replace Google internal URL with the public ones by @shuuji3 in #233
• Update build-test.yml by @thinhha in #239

New Contributors

• @shuuji3 made their first contribution in #233

Full Changelog: v1.0.6...v1.0.7

v1.0.6

What's Changed

• Removed requests_oauth2 dependency from composer DAG by @shourya116 in #228
• Fixed cloud build issue for pytz dependency by @AmandeepSinghCS in #230
• Updated the composer dag by @shourya116 in #229
• Fixed the incremental validation for default partitioned bigquery tables referenced with entity_uri by @shourya116 in #232
• Updated the bigquery dry run timeout by @shourya116 in #234
• Fixed Dataplex URI for Big Lake Tables by @shourya116 in #235

Full Changelog: v1.0.5...v1.0.6

v1.0.5

What's Changed

• fix bug with bq client by @thinhha in #227

Full Changelog: v1.0.4...v1.0.5

v1.0.4

What's Changed

• added reference columns to views by @shourya116 in #221
• remove optional project_id argument from BigQuery client by @thinhha in #225
• fixed ambiguous columns by @shourya116 in #224

Full Changelog: v1.0.3...v1.0.4

v1.0.3

What's Changed

• fix regex rule by @AmandeepSinghCS in #195
• updated the documentation for reference columns by @AmandeepSinghCS in #196
• Update REFERENCE.md by @thinhha in #200
• updated python and dbt dependencies by @AmandeepSinghCS in #197
• updated dbt_project.yml by @AmandeepSinghCS in #201
• Update REFERENCE.md by @thinhha in #208
• fixed reference column yaml by @shourya116 in #214
• updated the warning message by @shourya116 in #215
• Bump setuptools from 60.6.0 to 65.5.1 by @dependabot in #218
• fixed complex_rule_validation_success_flag by @shourya116 in #217

New Contributors

• @shourya116 made their first contribution in #214
• @dependabot made their first contribution in #218

Full Changelog: v1.0.2...v1.0.3

v1.0.2

What's Changed

• fix for incremental validation by @AmandeepSinghCS in #184
• fixed incremental validation test that assumes dq_summary table to be… by @AmandeepSinghCS in #185
• removed execution_ts from failed_records sql query by @AmandeepSinghCS in #187
• Fix test cli integration by @AmandeepSinghCS in #189
• Updated google license by @AmandeepSinghCS in #188
• removed reference columns from custom sql statement rule by @AmandeepSinghCS in #190
• fix the dataplex entity for multiple matches for bigquery uri by @AmandeepSinghCS in #192
• Include star for ref columns by @AmandeepSinghCS in #193

Full Changelog: v1.0.1...v1.0.2

v1.0.1

This release includes the following changes:

1. Added the new feature which adds failed_records_query field to the dq_summary and target table . The query will give the records from source table that failed the data quality check.
2. Added the feature to specify reference_columns in input configs and include them from source table to failed_records_query in dq_summary and target table. This will help in getting the required information from source table by joining it on referenced columns. The reference columns will mostly include unique identifier fields from source table.
3. Bug-Fixes for incremental data validation.

What's Changed

• added get high watermark fields and current execution timestamp to co… by @AmandeepSinghCS in #173
• Update composer docs by @AmandeepSinghCS in #176
• Save failed records query by @AmandeepSinghCS in #177
• fix to make reference columns optional by @AmandeepSinghCS in #178
• Update custom sql rule failed records logic by @AmandeepSinghCS in #179
• Save failed records query per rule by @AmandeepSinghCS in #181
• updated failed records query field names. by @AmandeepSinghCS in #182
• Add invocation id to failed records query by @AmandeepSinghCS in #183

Full Changelog: v1.0.0...v1.0.1

v1.0.0

We're happy to announce that Cloud Data Quality (CloudDQ) project has reached the stable v1.0.0 version. This release includes the following changes:

1. Officially deprecating support for CLI flags --dbt_path and --dbt_profiles_dir. Please migrate to using CLI flags --gcp_project_id, --gcp_bq_dataset_id, --gcp_service_account_key_path (if using exported SA keys), --gcp_impersonation_credentials (if using SA impersonation) instead. If you are still using --dbt_path and --dbt_profiles_dir, existing pipelines will break and you are advised to migrate to using the native connection configurations flags described above as soon as possible.
2. Officially graduating CLI flags --enable_experimental_dataplex_gcs_validation and --enable_experimental_bigquery_entity_uris from experimental status. These flags respectively allow validation of Dataplex GCS Assets via BigQuery External Tables and referencing BigQuery tables directly via entity_uri, without needing to first register them as Dataplex Assets. If used, the experimental flags will not throw an error, however they are redundant and can be removed as their behaviors are now enabled by default.
3. Officially making CLI flag --target_bigquery_summary_table a required argument. Users are recommended to only consume Data Quality summary results from the target table of their choice instead of relying on the dq_summary table or any intermediate data stored in the BigQuery dataset specified in --gcp_bq_dataset_id. The --target_bigquery_summary_table cannot be the same table as the dq_summary table automatically created in the BigQuery dataset specified in --gcp_bq_dataset_id.
4. Exposes new CLI flag --num_threads for tuning performance. This flag allows increasing number of concurrent BigQuery jobs for calculating data quality summary data.
5. Exposes new CLI flag --intermediate_table_expiration_hours for tuning storage expiration for intermediate entity-level data quality summary calculations data.
6. bug-fixes for allowing more BigQuery data types such as GEOGRAPHY and RECORD when using Dataplex and BigQuery entity_uri.
7. bug-fixes for allowing case-insensitive entity-ids.
8. bug-fixes for improving error messages from parsing invalid YAML configurations.

What's Changed

• fixed entity-id-uuid bug by @AmandeepSinghCS in #164
• fixed geography type by @AmandeepSinghCS in #163
• add RECORD type by @thinhha in #165
• added a flag for dbt intermediate table expiration hours by @AmandeepSinghCS in #158
• Dbt threads cli flag by @AmandeepSinghCS in #161
• removed dbt flags and updated dbt runner by @AmandeepSinghCS in #166
• updated user manual for intermediate_table_expiration_hours and num_t… by @AmandeepSinghCS in #168
• Update docs for v1.0.0 by @thinhha in #167
• Target table req arg by @AmandeepSinghCS in #169
• validate configs before loading into cache by @thinhha in #170

Full Changelog: v0.5.3...v1.0.0

v0.5.3

What's Changed

• clarify that entities configs are not needed if users only use entity_uri by @thinhha in #148
• Advanced Rules: NIT Update in USERMANUAL.md by @hejnal in #149
• reenable test_advanced_dq_rules tests by @thinhha in #150
• fix schema for dataplex gcs entities by @AmandeepSinghCS in #151
• updated release pipeline to copy executables for testing by @AmandeepSinghCS in #154
• fixed release pipeline by @AmandeepSinghCS in #155
• Bigquery Native Tables support by @AmandeepSinghCS in #153
• Bq uri enhancements and tests by @AmandeepSinghCS in #159
• removed bq uri experimental flag by @AmandeepSinghCS in #157
• Remove gcs validation flag by @AmandeepSinghCS in #162

Full Changelog: v0.5.2...v0.5.3