Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Feature] Make timeout for bigquery temp table dynamic. #805

Open
3 tasks done
gjskibinski opened this issue Feb 9, 2025 · 0 comments · May be fixed by #806
Open
3 tasks done

[Feature] Make timeout for bigquery temp table dynamic. #805

gjskibinski opened this issue Feb 9, 2025 · 0 comments · May be fixed by #806
Labels
pkg:dbt-bigquery Issue affects dbt-bigquery type:enhancement New feature request

Comments

@gjskibinski
Copy link

gjskibinski commented Feb 9, 2025

Is this your first time submitting a feature request?

  • I have read the expectations for open source contributors
  • I have searched the existing issues, and I could not find an existing issue for this feature
  • I am requesting a straightforward extension of existing dbt functionality, rather than a Big Idea better suited to a discussion

Describe the feature

Current State
At the moment temp tables created in big query dbt models have arbitrary timeout set to 12h.

Code Reference

opts["expiration_timestamp"] = "TIMESTAMP_ADD(CURRENT_TIMESTAMP(), INTERVAL 12 hour)"

Idea for the feature
It would be useful to customize this timeout and align it with actual timeout for processing.

Idea for follow up
Once we customize timeout - it will give us foundation to also customize temp table name and make it per run - at the moment it's always <original-table>_temp which will prevent e.g. concurrent runs targeting different partitions with insert_overwrite per partition strategy.
I think in order to get there though - it's essential to align timeouts.

Describe alternatives you've considered

No response

Who will this benefit?

People who run the same model many times, with high failures rate (since if query fails temp table won't be dropped).

Are you interested in contributing this feature?

Yup, I'll add PR shortly + I signed CLA

Anything else?

No response

@gjskibinski gjskibinski added triage:product In Product's queue type:enhancement New feature request labels Feb 9, 2025
@gjskibinski gjskibinski linked a pull request Feb 9, 2025 that will close this issue
4 tasks
@amychen1776 amychen1776 added pkg:dbt-bigquery Issue affects dbt-bigquery and removed triage:product In Product's queue labels Feb 11, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
pkg:dbt-bigquery Issue affects dbt-bigquery type:enhancement New feature request
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants