-
Notifications
You must be signed in to change notification settings - Fork 8
/
Copy pathCHANGELOG
493 lines (373 loc) · 22.9 KB
/
CHANGELOG
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
snowplow-utils 0.17.1 (2024-10-29)
---------------------------------------
## Summary
This release is aiming to optimize base_create_snowplow_events_this_run macro for spark.
## Fix
- Optimize base_create_snowplow_events_this_run for spark
## Upgrading
Update the snowplow-utils version in your `packages.yml` file.
snowplow-utils 0.17.0 (2024-10-14)
---------------------------------------
## Summary
This release focuses on expanding support for Apache Spark with the Iceberg file format and enhancing integration tests.
## Features
- Add support for Apache Spark with Iceberg file format
## Under the hood
- Modify integration tests to support Spark
- Changed return_limits_from_model macro default limits
## Upgrading
Update the snowplow-utils version in your `packages.yml` file.
snowplow-utils 0.16.8 (2024-07-29)
---------------------------------------
## Summary
This release extends support for those users who would like to use the Snowplow incremental base macros but are using specific older versions of the AMP tracker or the Pixel tracker which do not send dvce_created / dvce_sent_tstamp fields. It also fixes a bug in the apply_grands() macro when the users does not have a grants config defined.
## Features
- Add allow_null_dvce_tstamps var
## Fixes
- Add default for grant_config macro
## Upgrading
To upgrade, bump the package version in your `packages.yml` file.
snowplow-utils 0.16.7 (2024-06-10)
---------------------------------------
## Summary
This release extends support for the combine_column_versions macro to support column prefixes not ending in a major version number (when used in combination with exclude column versions).
## Features
- Extend functionality of combine_column_versions
## Under the hood
- Update some of our test to pass with newer versions of dbt utils
## Upgrading
To upgrade, bump the package version in your `packages.yml` file.
snowplow-utils 0.16.6 (2024-05-24)
---------------------------------------
## Summary
This release adds support for a list of unique keys when using the snowplow optimized incremental approach with the delete+insert strategy.
## Features
- Support list of unique keys when using delete+insert strategy with snowplow optimize
## Upgrading
To upgrade, bump the package version in your `packages.yml` file.
snowplow-utils 0.16.5 (2024-05-09)
---------------------------------------
## Summary
This release moves the custom sql part of the events this run macro for redshift to the same level as the custom entities or SDEs are joined so that they can be used in any custom sql.
## Features
- Allow usage of entity and SDE fields in custom SQL for redshift.
## Upgrading
To upgrade, bump the package version in your `packages.yml` file.
snowplow-utils 0.16.4 (2024-04-15)
---------------------------------------
## Summary
This release adds more flexibility to the `exclude_versions` argument of the `combine_column_versions` macro for BigQuery to allow removal of any column with the suffix of the specified version.
## Fixes
- Remove hardcoded structure requirement to exclude column versions.
## Upgrading
To upgrade, bump the package version in your `packages.yml` file.
snowplow-utils 0.16.3 (2024-03-11)
---------------------------------------
## Summary
This releases adds the database argument to the `get_sde_or_context` macro.
## Features
- Add `database` argument to `get_sde_or_context` macro
## Upgrading
To upgrade, bump the package version in your `packages.yml` file.
snowplow-utils 0.16.2 (2024-02-26)
---------------------------------------
## Summary
This releases adds two new sets of macros, one is `parse_agg_dict` that generates aggregation sql, and the other is `apply_grants` which allows the usage of the `snowplow__grant_select_to` variable to grant select on models to any user/role in that list (does not support BigQuery). We also fix a rare edge case where custom user identifiers made from sql could lead to duplicate sessions in the lifecycle table.
## Features
- Add new `parse_agg_dict` macro for use to generate aggregation sql in other packages
- Overwrite default dbt `apply_grants` macro to enable using a variable to define grant targets
- Add new `default__grant_usage_on_schemas_built_into` macro to add as a post-hook in package to grant usage for schemas
- Ensure only one user id even when a custom user sql is provided for the lifecycles manifest table
## Upgrading
To upgrade, bump the package version in your `packages.yml` file.
snowplow-utils 0.16.1 (2024-01-29)
---------------------------------------
## Summary
This release contains a fix to the unnest macro for redshift, which meant the query wasn't complete when using it without the index.
## Fixes
- Fix bug in redshift unnest macro
## Upgrading
To upgrade, bump the package version in your `packages.yml` file.
snowplow-utils 0.16.0 (2024-01-26)
---------------------------------------
## Summary
This release contains a new array size macro as well as the optional ability for the unnest macros to also surface the index of the original array element for preserving the order. It also contains some fixes for redshift. Please note that from this version onwards this package is under the SPAL license.
# Features
- Add get_array_size macro
- Add optional index to unnest
## Fixes
- Fix existing tests for redshift
- Fix issue with multiple end hooks failing (Close #152)
## Under the hood
- Update license to SPAL
## Upgrading
To upgrade, bump the package version in your `packages.yml` file.
snowplow-utils 0.15.2 (2023-10-13)
---------------------------------------
## Summary
This release contains a bug fix in our new `base` macros, which did not affect default behaviour. This also does not impact behavior in any of the Snowplow packages that rely on the utils package.
## Fixes
- Fix an incorrect default value in the `base_create_snowplow_events_this_run` macro.
## Upgrading
To upgrade, bump the package version in your `packages.yml` file.
snowplow-utils 0.15.1 (2023-09-29)
---------------------------------------
## Summary
This release combines a few bug fixes in our new `base` macros, including an issue with redshift query size limits.
## Fixes
- Fix issue with 8001 redshift error
- Add condition for late loading events to fix issue where session events before session start timestamp were processed
- Fix the manifest table not using the specified session timestamp field
- Provide correct error message when no user identifier is present
## Under the hood
- Fix some issues in our integration tests
## Upgrading
To upgrade, bump the package version in your `packages.yml` file.
snowplow-utils 0.15.0 (2023-09-19)
---------------------------------------
## Summary
This full release introduces new "base" macros which are used to recreate the `snowplow_base_events_this_run` and related tables. It gives you the option to customise your session and user identifiers, your partitioning timestamps, and much more. Additionally, we remove the deprecated `snowplow_incremental` materialization that was removed in `0.14.0`, and we add multi-version support for the `get_field` macro for BigQuery.
We have documentation explaining the outlined changes in detail [here](https://docs.snowplow.io/docs/modeling-your-data/modeling-your-data-with-dbt/) that you can use to explore the base macro changes in particular. For information about how to do this, see our [Advanced Usage of the Utils package](https://docs.snowplow.io/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-utils-data-model/dbt-utils-advanced-operation/) page and the [utils configuration](https://docs.snowplow.io/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-configuration/utils/) variables.
Finally, before we dig into the details, starting with this version all future releases of our dbt packages will be released under the [Snowplow Community License](https://docs.snowplow.io/community-license-1.0/) instead of the Apache 2 license. For what this means for you, check out our [FAQ](https://docs.snowplow.io/docs/contributing/community-license-faq/), and our [announcement blog](https://snowplow.io/blog/license-changes-for-snowplow-open-source/).
## Features
- Create the "base" macros to enable greater flexibility
- Update the package to be under the Snowplow Community License
- Released a fix where `get_optional_fields` returned field aliases even when `include_field_alias` was set to false ([#132](https://github.com/snowplow/dbt-snowplow-utils/issues/132))
- Add multi-version support to the `get_field` macro for BigQuery
- Remove deprecated `snowplow_incremental` materialization
## Under the hood
- Introduce integration tests to check customization
## Upgrading
To upgrade, bump the package version in your `packages.yml` file.
snowplow-utils 0.15.0-rc1 (2023-08-31)
---------------------------------------
## Summary
**This is a pre-release version of the package, we believe it to be in working condition but you may encounter bugs and some features may change before the final release.**
This release introduces new "base" macros which are used to recreate the `snowplow_base_events_this_run` and related tables. It gives you the option to customise your session and user identifiers, your partitioning timestamps, and much more. We have temporary docs hosted [here](https://deploy-preview-445--snowplow-docs.netlify.app/docs/modeling-your-data/modeling-your-data-with-dbt/) that you can use to explore docs related to these changes. Once this is fully released, these docs will migrate over to the offical [docs site](https://docs.snowplow.io/)
For information about how to do this, see our [Advanced Usage of the Utils package](https://deploy-preview-445--snowplow-docs.netlify.app/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-advanced-usage/dbt-utils-advanced-operation/) page and the [utils configuration](https://deploy-preview-445--snowplow-docs.netlify.app/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-configuration/utils/) variables.
Finally, before we dig into the details, starting with this version all future releases of our dbt packages will be released under the [Snowplow Community License](https://docs.snowplow.io/community-license-1.0/) instead of the Apache 2 license. For what this means for you, check out our [FAQ](https://docs.snowplow.io/docs/contributing/community-license-faq/), and our [announcement blog](https://snowplow.io/blog/license-changes-for-snowplow-open-source/).
## Features
- Create the "base" macros to enable greater flexibility
- Update the package to be under the Snowplow Community License
- Released a fix where `get_optional_fields` returned field aliases even when `include_field_alias` was set to false ([#132](https://github.com/snowplow/dbt-snowplow-utils/issues/132))
## Under the hood
- Introduce integration tests to check customization
## Upgrading
To upgrade, bump the package version in your `packages.yml` file.
snowplow-utils 0.14.3 (2023-07-04)
---------------------------------------
## Summary
This version adds the ability to specify which package is returning the run limits in the `print_run_limits` macro, and fixes an issue with using string agg in Databricks with many `null` values.
## Features
- Add ability to tag package in limits (Close #133)
## Fixes
- Fix issue with large volume of nulls in databricks string agg
## Upgrading
To upgrade bump the package version in your `packages.yml` file.
snowplow-utils 0.14.2 (2023-04-19)
---------------------------------------
## Summary
This version adds the `get_field` macro to enable extracting fields from our self-describing-event and context columns easier and more consistent across warehouses.
## Features
- Add new macro `get_field` to make it easier to extract fields from unstruct/context columns across warehouses
## Upgrading
To upgrade bump the package version in your `packages.yml` file.
snowplow-utils 0.14.1 (2023-04-06)
---------------------------------------
## Summary
This version makes changes to the new `get_sde_or_context` macro to support returning multiple entities per context, and changes the join column names to reduce the risk of clashes.
## 🚨 Breaking Changes 🚨
- Column names output by the `get_sde_or_context` macro have been changed from `<context_name>_id/tstamp` to `<context_name>__id/tstamp`. There is no need for you to make any changes unless you have used this macro in a custom model.
## Features
- Add support for contexts with multiple values per event to the `get_sde_or_context` macro
## Upgrading
To upgrade bump the package version in your `packages.yml` file.
snowplow-utils 0.14.0 (2023-03-28)
---------------------------------------
## Summary
This version makes some big changes by deprecating our `snowplow_incremental` materialization and a few of our other macros, with the goal of providing a simpler usage experience and easier maintenance of the package going forward. We've also added a brand new macro, `get_sde_or_context` to aid working with Self Describing Events or Contexts for our Redshift/Postgres users.
## 🚨 Breaking Changes 🚨
### Deprecated `snowplow_incremental` materialization
We have deprecated the `snowplow_incremental` materialization and will be removing it entirely in a future version. In place we are providing an optimization on top of the built-in `incremental` materialization. To use this optimization for `incremental` materialized models, each model config must have `snowplow_optimize=true` and the following must be added to the top level of your `dbt_project.yml` file:
```yaml
# dbt_project.yml
...
dispatch:
- macro_namespace: dbt
search_order: ['snowplow_utils', 'dbt']
```
For more information see [here](https://docs.snowplow.io/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-advanced-usage/dbt-incremental-materialization/).
### Deprecated macros
The `get_cluster_by` and `get_partition_by` macros have also been deprecated and will be removed in a future version. These should be replaced by `get_value_by_target_type` which offers the same functionality but more generally.
### `type_string` and `type_max_string`
`type_string` has been entirely removed from the package and should be replaced with calls direct to `dbt.type_string()` instead. In all cases except redshift this should be suitable, for redshift when you need a column of length greater than 256 we provide `type_max_string` instead. For all other warehouses this is just a wrapper to `dbt.type_string()`.
## Features
- Deprecate `get_cluster_by` and `get_partition_by` macro in favor of `get_value_by_target_type`
- Remove `type_string()` and rework `type_max_string()` to prioritize dbt logic where possible
- Deprecated old materialization
- Add new `get_sde_or_context` macro
## Under the hood
- Remove all internal references to `snowplow_incremental` materialization
- Migrate tests to new materialization approach
## Docs
- Update readme
## Upgrading
To upgrade bump the package version in your `packages.yml` file, and follow our [migration guide](https://docs.snowplow.io/docs/modeling-your-data/modeling-your-data-with-dbt/migration-guides/utils/#upgrading-to-0140) for the above breaking changes.
snowplow-utils 0.14.0-rc2 (2023-03-10)
---------------------------------------
## Summary
**This is a pre-release version of the package, we believe it to be in working condition but you may encounter bugs and some features may change before the final release.**
This version fixes a few issues from the first release candidate, including failing when you had no data in a scratch table, and deprecates some macros.
As a reminder Users will need to add the following to their `dbt_project.yml` to benefit from the enhancements:
```yaml
# dbt_project.yml
...
dispatch:
- macro_namespace: dbt
search_order: ['snowplow_utils', 'dbt']
```
For custom models and more details, please refer to our temporary docs page: https://docs.snowplow.io/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-advanced-usage/dbt-incremental-logic-pre-release/
## Features
Deprecate `get_cluster_by` and `get_partition_by` macro in favor of `get_value_by_target_type`
Remove `type_string()` and rework `type_max_string()` to prioritize dbt logic where possible
Fix inability to progress when scratch table contained no data
Ensure type consistency for the `upsert_date_key` throughout query
## Under the hood
Remove all internal references to `snowplow_incremental` materialization
Migrate tests to new materialization approach
snowplow-utils 0.14.0-rc1 (2023-03-06)
---------------------------------------
## Summary
**This is a pre-release version of the package, we believe it to be in working condition but you may encounter bugs and some features may change before the final release.**
This version of the package begins the migration away from our `snowplow_incremental` materialization and instead provides an overwrite to the standard `incremental` materialization to provide the same performance improvements but in a simpler way. We expect users should see little to no performance change from the previous version, please let us know if you see performance degradation for large volumes of data.
Users will need to add the following to their `dbt_project.yml` to benefit from the enhancements:
```yaml
# dbt_project.yml
...
dispatch:
- macro_namespace: dbt
search_order: ['snowplow_utils', 'dbt']
```
For custom models and more details, please see more details on our temporary docs page: https://docs.snowplow.io/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-advanced-usage/dbt-incremental-logic-pre-release/
## Features
Deprecated old materialization
Add get_merge_sql for materialization
Fix a broken github action for our github pages
snowplow-utils 0.13.2 (2023-02-21)
---------------------------------------
## Summary
This release fixes a compilation error raised if dbt compile is ran on a fresh installation of one of the dbt-snowplow packages. Under the hood we also fix the GitHub pages generation automation and update the pr template.
## Features
Fix initial dbt compile error (Close #69)
Fix utils gh pages generation
Update pr template
snowplow-utils 0.13.1 (2023-02-20)
---------------------------------------
## Summary
This release introduces a new cross-db macro - get_array_to_string - to harmonise array to string transformations, as well as adds more features / optimisations to some of the existing macros. There are some automations and simplifications made under the hood for easier maintenance and clarity.
## Features
Get string agg optimisations (Close #101)
Add get_array_to_string macro
Fix unnest macro for Postgres and snowflake (Close #105)
Add delimiter parameter to get_split_to_array (Close #106)
Tidy macro inheritance
Add warning for no data returned for model limits
Document macros in yaml, add new macro, prepare to depeciate
Add action for generating docs for pages
snowplow-utils 0.13.0 (2022-12-08)
---------------------------------------
## Summary
This release bumps the dependency of `dbt-utils` to support v1, and therefore will remove all of the deprecation warnings being displayed for users on later versions of dbt. This version also requires dbt version 1.3 at least
## Features
Bump compatibility to [email protected] as a minimum (Close #95)
Add standard actions and templates + use utils for databricks connection
snowplow-utils 0.12.3 (2022-11-30)
---------------------------------------
Add unnest macro (Close #99)
Add Media Player macros (Close #79)
snowplow-utils 0.12.2 (2022-10-26)
---------------------------------------
Bump dbt-utils version for patch fix (Close #92)
Fix incremental_strategy default argument bug (Close #97)
snowplow-utils 0.12.1 (2022-09-22)
---------------------------------------
Exclude specific entity versions from combine_column_version macro (Close #91) (Thanks to @bgraf)
snowplow-utils 0.12.0 (2022-08-11)
---------------------------------------
Add ability to create indexes from config (Close #83)
Add support for Databricks runtimes below 10.4 (Close #85)
snowplow-utils 0.11.0 (2022-07-19)
---------------------------------------
Add support for databricks via Spark (Close #80)
snowplow-utils 0.10.0 (2022-06-22)
---------------------------------------
Add support for databricks (Close #77)
snowplow-utils 0.9.0 (2022-05-05)
---------------------------------------
Add `set_query_tag` functionality (Close #71)
Update the codeowner file (Close #73)
snowplow-utils 0.8.0 (2022-03-11)
---------------------------------------
Add a type_string macro to generate a varchar type (Close #65)
Add cross_db macro documentation (Close #66)
snowplow-utils 0.7.0 (2022-03-09)
---------------------------------------
Add support for dbt v1.0.0+ (Close #63)
Bump copyright to 2022 (Close #60)
Support dbt_utils 0.8.0 (Close #58)
snowplow-utils 0.6.0 (2022-01-31)
---------------------------------------
Add snowplow_mobile_delete_from_manifest macro (Close #54)
snowplow-utils 0.5.0 (2021-12-16)
---------------------------------------
Add dbt v1 compatibility (Close #52)
snowplow-utils 0.4.1 (2021-12-06)
---------------------------------------
Fix for unexpired sessions (Close #48)
snowplow-utils 0.4.0 (2021-11-29)
---------------------------------------
Update README (Close #45, Close #35)
Refactor combine_column_versions (Close #44)
Add CI Action (Close #43)
Improve filtering for long sessions (Close #42)
Refactor snowplow_delete_from_manifest (Close #46)
Restructure snowplow-utils project (Close #41)
Transition DDL manifest creation to native dbt (Close #40)
Add support for Postgres (Close #39)
snowplow-utils 0.3.1 (2021-10-12)
---------------------------------------
Fix snowplow_merge for dbt 0.19.2 (Close #33)
snowplow-utils 0.3.0 (2021-10-11)
---------------------------------------
Update README (Close #31)
Update integration tests for Github Actions (Close #32)
Remove create_snowplow_manifest_schema step (Close #30)
Fix get_successful_models for dbt 0.18.0 (Close #23)
Add dynamic variables based on target name (Close #28)
Add Snowflake support to snowplow_incremental (Close #27)
Fix BigQuery snowplow_incremental expected dataset (Close #25)
Add Snowflake support to macros (Close #26)
snowplow-utils 0.2.0 (2021-08-20)
---------------------------------------
Update READMEs (Close #21)
Update dbt_utils URL to new dbt-labs repo (Close #20)
Improve is_run_with_new_events performance (Close #19)
Refactor & add tests snowplow_delete_from_manifest (Close #18)
Refactor create_incremental_manifest_table (Close #17)
Add integrations_tests script (Close #16)
Replace get_successful_snowplow_models (Close #15)
Fix snowplow_teardown_all macro (Close #14)
Refactor & add tests get_enabled_snowplow_models (Close #13)
Remove update_current_incremental_tstamp_table (Close #12)
Add BigQuery support to snowplow_incremental (Close #7)
Refactor & add tests app_id_filter (Close #11)
Refactor & add tests update_incremental_manifest_table (Close #10)
Refactor & add tests get_run_limits (Close #9)
Add functionality to support BigQuery (Close #8)
snowplow-utils 0.1.1 (2021-07-26)
---------------------------------------
Fix app_id_filter macro (Close #4)
Fix README formatting for dbt hub (Close #3)
snowplow-utils 0.1.0 (2021-07-26)
---------------------------------------
Add snowplow-utils v0.1.0 (Close #2)