UX Feedback on Logging and stdout

[dataders]

What would you like to see from logs that you aren't yet seeing?

Please try to make a comment for each distinct idea so that they can be addressed individually and also prioritized with voting.

[dataders]

✅ from @pempey (community slack thread)

dbt.log is overwritten with every invocation today with Fusion. in Core, new invocation logs append to the end of the file
Right now I am trouble shooting parsing errors running different commands and the dbt.log and the query_log.sql are getting rest with every command. That makes it hard to track the errors as I have to copy pasty my findings out to other fiels where they lose there context

[dataders]

this was shipped in 2.0.0-preview.47

[pempey]

✅ I would like to see timestamps in the text version of the log output. There are timestamps in the json output and while json is human parsable it is not really human readable and timestamps can be a critical piece of information when debugging.

[dataders]

timestamps were added for most in preview.56 released on October 31.

If there's things missing still please open a ticket so we can have them added!

[pempey]

✅ I would like the log to be more than a copy of stdout. With the current fancy stdout output processes are happening, durations are counting, but none of that shows up in the logs. I think that the logs should reflect the events that transpired, even at the INFO level, so even if the duration of a process is not in stdout once it is complete the time the process ended (or stated) and the process duration would show up in the log.

[dataders]

This was the case during the beta phase, but was (IIRC) rectified in preview.56 released on October 31.

[pempey]

I would like the stdout and the log to show more information on the invocation processing. Currently there is nothing but the query log to tell me if by on-run-start and on-run-end were processed. In general there needs to be more information about the invocation made available in the logs.

[dataders]

can you say more about this? you mean to say you'd like INFO/stdout to mention when it's executing on-run-start and on-run-end hooks? Is there anything else that you think is missing?

[pempey]

✅ I would like there to be a link between the dbt.log output and the query_log.sql. In general I would like to be able to trace when a query is executed and link it to the sequence of events of the invocation. I would like to know why a query was run and not just what query was run.

[dataders]

so we haven't shipped exactly what you're asking for. however, all queries are also logged in logs/dbt.log. Does that meet your requirements?

[dataders]

@pempey in Slack says

why does test output include a schema?

dbtf test -s dim_date                                                                                                                       
dbt-fusion 2.0.0-preview.3
   Loading ~/.dbt/profiles.yml
   Loading packages.yml
    Passed [  0.40s] test  dbt_test__audit.not_null_dim_date_date_id
    Passed [  0.72s] test  dbt_test__audit.unique_dim_date_date_id

[dataders]

@pempey initially I thought

yeah there shouldn't be a schema

but now i'm thinking

oh wait, what if store_failures: true? I should see FQN in that scenario, right?

what say you?

[pempey]

Yes it makes sense if store_failures: true. When store_failures: false the query only lives in the target folder so there should not be a schema associated with it.

[jeff-skoldberg-gmds]

The two longer running models have 2+ minutes of render time. I believe the render time should be included in the final result, otherwise the model looks faster than it actual is. The grand total of 2:53 is correct, but I would say the model times shown are false.

[dataders]

🚧 https://github.com/dbt-labs/dbt-fusion/issues/1559

[pempey]

🎫 In the log a message that lists models and appends to the end ...and n others is not helpful. I would expect the log to list every model the message refers to.

warning: dbt1000: Skipping unit_tests for models with detected 'unsafe' and 'static_analysis' set to 'off': 'unit_test.gitlab_snowflake.snowplow_event_counts.test_snowplow_event_count_event_filtering', 'unit_test.gitlab_snowflake.snowplow_gitlab_events.test_snowplow_gitlab_events_web_page_id_deduplication', 'unit_test.gitlab_snowflake.snowplow_gitlab_events.test_snowplow_gitlab_events_tracker_filter', 'unit_test.gitlab_snowflake.snowplow_event_counts.test_snowplow_event_count_last_uploaded_at', 'unit_test.gitlab_snowflake.snowplow_gitlab_events_context_flattened.test_snowplow_gitlab_events_context_flattened' and 5 others

[jtcohen6]

I'd expect:

• debug-level logs where I can see the full list - probably logs/dbt.log, or --debug if I want them on stdout
• a verbose flag to "expand" specific warnings (like we have with --show-all-deprecations in dbt Core)

[dataders]

the full list is still not present in debug level logs

[dataders]

https://github.com/dbt-labs/dbt-fusion/issues/1115

[pempey]

🎫 The error trace in the stdout can make it more challenging to find the error then actually help the developer. for example:

    Failed [  1.86s] model common.dim_date (table)
error: dbt1501: Error executing materialization macro 'dbt_snowflake.materialization_table_snowflake' for model model.gitlab_snowflake.dim_date: Failed to eval the compiled Jinja expression invalid operation: AdapterError: [Snowflake] 001003 (42000): SQL compilation error:
syntax error line 90 at position 0 unexpected ')'. (SQLSTATE: 42000, Vendor code: 1003)
(in compiled/models/common/dimensions_shared/dim_date.sql:1:14)
(in dbt_internal_packages/dbt-snowflake/macros/materializations/table.sql:26:11)
(in dbt_internal_packages/dbt-adapters/macros/etc/statement.sql:14:35)
  --> compiled/models/common/dimensions_shared/dim_date.sql:1:14

The model in question had a SQL error and while the model fails and the trace leads back to the model and the first line of the error message has the name of the model in it there are some problems with clarity in the messaging.

The styling actually hides the name of the model in the error message:

In the first line the eyes are drawn to the name of the macro. This could be helpful if the macro was acutely in the model but this leads to another clarity problem: the primary error and most of the trace are for internal macros. The trace feels like it is sending me in the wrong direction to fix the error with the information it is giving me. The real problem, and where a developer should be fixing it is buried within the full text of the message.

I think that better styling would help a lot here. But I also wonder how much of the trace should just go to the logs? Or maybe it should be 'smarter' and better indicate when the trace goes into the internal macros (I do not think that the file path is a clear indicator as they can easily be noisy when you see them all the time)

[dataders]

https://github.com/dbt-labs/dbt-fusion/issues/1116

[jeff-skoldberg-gmds]

✅ The summary info at the end of a dbt Core run should be added to fusion.

dbt Core:

vs Fusion:

Core is much more informative! Must have :)

[dataders]

great point -- this is the same as @bruno-szdl's point made on LinkedIn (i.e. re-emit all errors again at the bottom)

I'd love to see all the errors again at the bottom, so I wouldn't need to scroll up to see what failed.

note: I've deleted the comment i made 4 days ago in this discussion and instead am commenting Bruno's point here because @jeff-skoldberg-gmds's picture makes it much more clear

[jeff-skoldberg-gmds]

Cool, I thought "re-emit all errors at bottom" was JUST that, and not also summarize the total score:
Pass / Warn / Error / Skip etc.

So, I would like to see the error messages at the bottom as well as the summary "score board".

[jtcohen6]

Yes! I think it's both things -

1. Fix the summary line so that it is correct (https://github.com/dbt-labs/dbt-fusion/issues/416)
2. Re-display all the warnings / errors again at the bottom, so users don't need to go hunting for them. (It's a good thing that Fusion displays the warnings / errors during the run. This is a longstanding complaint with dbt Core, you need to wait for the build to complete before knowing what went wrong.)

[dataders]

AFAICT, this request has been resolved, specifically the two things @jtcohen6 calls out above

[dataders]

On a dbt build I think I'd like to see Testing on it's own line?

In the current paradigm, my understanding that a model completes the "Running" phase when both the model is materialized and all of it's tests are currently running.

If testing were it's own line, I'm torn on if I'd rather see:

1. the unit of testing be the model: aligns with the deonmoinators for Rendering, Analyzing, and Running
2. the unit be each individual test: actually reflects the number of tests to be run

[dataders]

as soon as one model failure is detected, log messages turn from yellow to red even though the log message is exactly the same.

before run failure

after run failure

[pempey]

The color of the running model names appear to change to yellow when the run time exccesds 60 seconds. It would be great if the could be configured.

[dataders]

often models run for more than 999 seconds. in those cases we need more timing info than LONG!!!

[pempey]

I think I would prefer changing units rather an mixed units.

[  0.76s]
[ 52.76s]
[  5.01s]
[   5.7m]
[  1.08h]
[ 23.16h]

[jeff-skoldberg-gmds]

I think I would prefer changing units rather an mixed units.

[  0.76s]
[ 52.76s]
[  5.01s]
[   5.7m]
[  1.08h]
[ 23.16h]

Your proposal is potentially more accurate too. 2.16H is more precise than "2 hours 9 minutes".

[joellabes]

I don't love decimalised hours/minutes - this feels like prioritising consistency over clarity. The most precise way to represent 2.16 hrs is 7,776,000 ms but the whole point of this is to sacrifice a bit of precision in favour of human readability.

[jeff-skoldberg-gmds]

I don't love decimalised hours/minutes - this feels like prioritising consistency over clarity. The most precise way to represent 2.16 hrs is 7,776,000 ms but the whole point of this is to sacrifice a bit of precision in favour of human readability.

can you tell us what you do like? Hours minutes seconds?

[joellabes]

Oh sorry - didn't explicitly say it, but I liked jerco's original set

[pempey]

✅ I would expect that the time format between logs is consistent. In the beta_cache.log epoch time appears to be used but in the query_log.sql ISO format is used.

[COMPLETE] SCHEMA
   ╭─ Name: "PEMPEY"."COMMON"
   ├─ Relations: 0
   ├─ Cached: 1756490408015ms
   ╰─ Status: All relations cached as a batch

-- created_at: 2025-08-29T17:59:52.588928+00:00
-- dialect: snowflake
-- node_id: not available
-- desc: List relations
SHOW OBJECTS IN SCHEMA "PEMPEY_PROD"."WORKSPACE_DATA" LIMIT 10000;

[dataders]

I'm giving this a ✅ if only because we've deprecated the beta_cache.log!

[pempey]

🎫 When an error give a list and overflows(?), truncating a list and appending and n others it defeats the usefulness of the list. This may be reasonable in the stdout but it should be supported in the log with the full list.

error: dbt0227: No column HIGH_PRIORITY__C found. Available are SOURCE.HIGH_PRIORITY_REASON__C, SOURCE.COGNISM_CITY__C, SOURCE.ISPRIORITYRECORD, SOURCE.ABM_TIER__C, SOURCE.FO_INQUIRY__C, SOURCE.HIGH_PRIORITY_TIMESTAMP__C, SOURCE.LEAD_FROM__C ..(and 351 more)

[dataders]

this is the same as this request, right?

[dataders]

https://github.com/dbt-labs/dbt-fusion/issues/1115

[pempey]

✅ When the compile of a test fails, for example because of a missing column, I would like the error to direct me to the yaml where the test is defined in yaml and not the referents test macro in the target\generic_tests directory as the rendered test is not something I can change to fix the problem.

error: dbt0227: No column CUSTOMER_ID found. Available are PREP.CUSTOMERS.CUSTOMERS_DB_CLOUD_ACTIVATIONS_SOURCE.CLOUD_ACTIVATION_ID, PREP.CUSTOMERS.CUSTOMERS_DB_CLOUD_ACTIVATIONS_SOURCE.SUBSCRIPTION_NAME, PREP.CUSTOMERS.CUSTOMERS_DB_CLOUD_ACTIVATIONS_SOURCE.BILLING_ACCOUNT_ID, PREP.CUSTOMERS.CUSTOMERS_DB_CLOUD_ACTIVATIONS_SOURCE.CLOUD_ACTIVATION_CODE, PREP.CUSTOMERS.CUSTOMERS_DB_CLOUD_ACTIVATIONS_SOURCE.CLOUD_ACTIVATION_CREATED_AT, PREP.CUSTOMERS.CUSTOMERS_DB_CLOUD_ACTIVATIONS_SOURCE.CLOUD_ACTIVATION_UPDATED_AT, PREP.CUSTOMERS.CUSTOMERS_DB_CLOUD_ACTIVATIONS_SOURCE.SEAT_UTILIZATION_REMINDER_SENT_AT ..(and 1 more)
  --> target/generic_tests/not_null_customers_db_cloud_activations_source_customer_id.sql:7:8 (target/compiled/generic_tests/not_null_customers_db_cloud_activations_source_customer_id.sql:7:8)

[dataders]

https://github.com/dbt-labs/dbt-fusion/issues/1117

[pempey]

🎫 On theme with the formatting of errors I want to mention the information provided in this error:

warning: dbt1014: Failed to download source schema for 'source.gitlab_snowflake.zuora_revenue_share.zuora_revenue_share_calendar'. Setting 'static_analysis' to off. Skipping analysis for 'model.gitlab_snowflake.zuora_revenue_calendar_source': [Snowflake] 002003 (02000): SQL compilation error:
Database 'ZUORA_SNOW_USW2_PROD_001_ZUORA_INHEVIE' does not exist or not authorized. (SQLSTATE: 02000, Vendor code: 2003)

For a while I thought that there was an error while while creating the error. After reading it a few times I surmised that the end of the error was trying to tell me why it failed to download the schema. I think that the repeated information at the end was really confusing me and leading me to not fully understand what was happening. To be fair I am currently getting hundreds of these errors/warnings so I never really stopped to read them before now. But, the formatting and they way the information is connected does not help with the communication.

[dataders]

https://github.com/dbt-labs/dbt-fusion/issues/615

[jtcohen6]

We support dbtf as an unambiguous alias for Fusion, so that users who already have dbt Core / dbt Cloud CLI in their PATH's dbt namespace can install/try Fusion without making any changes.

There are a few places where we suggest users run commands - for example:

suggestion: Run `dbt init` to create a profiles.yml file and connect to your database. Cloud CLI credentials from `dbt_cloud.yml` are not yet supported.

New version available 2.0.0-preview.22 (run `dbt system update`)

We should change these from dbt to dbtf, since then it will copy-paste-just-work for all users.

(Also - "Cloud CLI credentials" are supported! #23)

[karenderer]

✅ Logs should give feedback the model selected does not exist, core's log is more useful is this scenario.

Fusion

dbtf build -s my_model
dbt-fusion 2.0.0-preview.45
   Loading ~/.dbt/profiles.yml
   Loading packages.yml
warning: dbt0255: Microbatch incremental strategy is not supported. Use --exclude config.incremental_strategy:microbatch to exclude these models.
  --> ...
warning: dbt1000: Detected unsafe introspection which may lead to non-deterministic static analysis. ...
================================== Execution Summary ==================================
Finished 'build' with 2 warnings for target 'dev' [1m 18s]

Core

dbt build -s my_model
Running with dbt=1.10.13
Registered adapter: snowflake=1.10.2
...
The selection criterion 'my_model' does not match any enabled nodes

[dataders]

https://github.com/dbt-labs/dbt-fusion/issues/215 is the ticket -- trying to get this shipped soon!

[dataders]

It's shipped now in preview.72!

[karenderer]

Group the warnings differently - one section for "this is wrong with your project" and one for "this is wrong with your current dbt command" since some projects may have a lot of warnings.

The example below correctly shows I tried to run a model that doesn't exist, but it's buried amongst the other project level warnings.

warning: dbt1000: The selection criterion 'fqn:my_model' does not match any enabled nodes
warning: dbt1000: Nothing to do. Try checking your model configs and model specification args

Full log below:

$ dbtf build -s my_model                                                                           git:(karen/fusion-fixes-20251120) ✗ 
dbt-fusion 2.0.0-preview.72
   Loading ~/.dbt/profiles.yml
  Fetching packages.yml
   Loading ⠐ [2s] 1 in-progress packages [0.3s]                                                                                                                                  WARNING: Python models feature is enabled via environment variable
WARNING: Python models feature is enabled via environment variable
WARNING: Python models feature is enabled via environment variable
WARNING: Python models feature is enabled via environment variable
WARNING: Python models feature is enabled via environment variable
WARNING: Python models feature is enabled via environment variable
WARNING: Python models feature is enabled via environment variable
WARNING: Python models feature is enabled via environment variable
WARNING: Python models feature is enabled via environment variable
WARNING: Python models feature is enabled via environment variable
WARNING: Python models feature is enabled via environment variable
WARNING: Python models feature is enabled via environment variable
WARNING: Python models feature is enabled via environment variable
 Resolving ⠠ [39s]                                                                                                                                                    
 Resolving ⠤ [39s]                                                                                                                                                    
 Resolving ⠦ [39s]                                                                                                                                                    
 Resolving ⠖ [39s]                                                                                                                                                    
 Resolving ⠒ [39s]                                                                                                                                                    

============================================================================== Errors and Warnings ==============================================================================
warning: dbt1065: Package 'audit_helper' requires dbt version [>=1.2.0, <2.0.0], but current version is 2.0.0-preview.72. This package may not be compatible with your dbt version.
warning: dbt1065: Package 'dbt_artifacts' requires dbt version [>=1.3.0, <1.11.0], but current version is 2.0.0-preview.72. This package may not be compatible with your dbt version.
warning: dbt1065: Package 'codegen' requires dbt version [>=1.1.0, <2.0.0], but current version is 2.0.0-preview.72. This package may not be compatible with your dbt version.
warning: dbt1065: Package 'dbt_utils' requires dbt version [>=1.3.0, <2.0.0], but current version is 2.0.0-preview.72. This package may not be compatible with your dbt version.
warning: dbt1065: Package 'dbt_expectations' requires dbt version [>=1.7.0, <2.0.0], but current version is 2.0.0-preview.72. This package may not be compatible with your dbt version.
warning: dbt1065: Package 'salesforce_formula_utils' requires dbt version [>=1.3.0, <2.0.0], but current version is 2.0.0-preview.72. This package may not be compatible with your dbt version.
warning: dbt1065: Package 'fivetran_utils' requires dbt version [>=1.3.0, <2.0.0], but current version is 2.0.0-preview.72. This package may not be compatible with your dbt version.
warning: dbt1065: Package 'dbt_date' requires dbt version [>=1.6.0, <2.0.0], but current version is 2.0.0-preview.72. This package may not be compatible with your dbt version.
warning: dbt1065: Package 'fivetran_utils' requires dbt version [>=1.3.0, <2.0.0], but current version is 2.0.0-preview.72. This package may not be compatible with your dbt version.
warning: dbt0255: Microbatch incremental strategy is not supported. Use --exclude config.incremental_strategy:microbatch to exclude these models.
  --> [path redacted]
warning: dbt1000: The selection criterion 'fqn:my_model' does not match any enabled nodes
warning: dbt1000: Nothing to do. Try checking your model configs and model specification args
warning: dbt1000: Detected unsafe introspection which may lead to non-deterministic static analysis. To suppress this warning, set static_analysis to 'unsafe' in the nodes' configuration. Learn more: https://docs.getdbt.com/docs/fusion/new-concepts. Nodes: [nodes redacted]

=============================================================================== Execution Summary ===============================================================================
Finished 'build' with 13 warnings for target 'snowflake_dev' [1m 42s]

[sophieml]

I was working on some logging tooling and noticed that execution_time calculation differs between Core and Fusion for models that are run in parallel. Looks like Fusion tracks cumulative time for parallel models (e.g. you'll see timing from <1s to >1m if running a large number of models), while Core uses individual model timings.

Is there anything else in Fusion's logging that is intentionally defined differently from Core?

[dataders]

seems similar to https://github.com/dbt-labs/dbt-fusion/discussions/584#discussioncomment-14211407 which we're tracking in https://github.com/dbt-labs/dbt-fusion/issues/1559. stay tuned

[civitaspo]

Please include the database name in Fusion console log lines (not just schema.alias)

In Fusion’s default console output, completed/running nodes are shown as schema.alias (e.g. Succeeded [0.12s] model analytics.customers). The database/catalog is omitted.

In projects that use multiple databases with the same schema and table names, those lines are ambiguous. We can’t tell which database a node landed in from stdout alone.

dbt Core’s start/result lines typically include the database (e.g. my_database.my_schema.my_model). Fusion’s telemetry already carries database / schema / identifier on node events, and --log-format json / OTEL attributes include node_relation.database, but the human-readable console message still drops it.

Ask: Please include the database in the default console display (e.g. database.schema.alias), or make it configurable (always / when multi-database / opt-in flag). Even showing the full relation name only when databases differ across the selected nodes would help a lot.