GraphQL Error: RunStepStatsQuery Fails Due to Message: 'NoneType' object has no attribute 'value'

[jimjeffers]

What's the issue?

We are running dagster 1.9.8 open source on kubernetes. I have a graph that triggers a few hundred operations in total. It works fine at first but around 4 hours into the job I can no longer check on the status. The UI returns the following error:

Operation name: {RunStepStatsQuery}

Message: 'NoneType' object has no attribute 'value'

Path: ["pipelineRunOrError","stepStats"]

Locations: [{"line":14,"column":3}]

Stack Trace:
  File "/usr/local/lib/python3.10/site-packages/graphql/execution/execute.py", line 523, in execute_field
    result = resolve_fn(source, info, **args)
  File "/usr/local/lib/python3.10/site-packages/dagster_graphql/schema/pipelines/pipeline.py", line 490, in resolve_stepStats
    return get_step_stats(graphene_info, self.run_id)
  File "/usr/local/lib/python3.10/site-packages/dagster_graphql/implementation/fetch_runs.py", line 369, in get_step_stats
    return [GrapheneRunStepStats(stats) for stats in step_stats]
  File "/usr/local/lib/python3.10/site-packages/dagster_graphql/implementation/fetch_runs.py", line 369, in <listcomp>
    return [GrapheneRunStepStats(stats) for stats in step_stats]
  File "/usr/local/lib/python3.10/site-packages/dagster_graphql/schema/logs/events.py", line 644, in __init__
    status=stats.status.value,

If I keep the run open for about 5 minutes the view eventually populates.

What did you expect to happen?

I do not believe we should be seeing a graphql error display. I'm also uncertain as to why the client cannot jump straight to the current step vs loading all of the prior log / job data for an extended amount of time before populating the view.

How to reproduce?

I'm not sure how to replicate this without replicating my job that causes the issue. Would it be best to supply the debug file from the job? It's 128mb so far.

Dagster version

dagster, version 1.9.8

Deployment type

Dagster Helm chart

Deployment details

Dagster is installed on GKE with autopilot enabled via helm.

Additional information

No response

Message from the maintainers

Impacted by this issue? Give it a 👍! We factor engagement into prioritization.

[gibsondan]

Hi @jimjeffers - I believe we have a fix for this going live in 1.9.9 this week: #27089

[jimjeffers]

Awesome! Thanks @gibsondan