Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Port Registering of Asset Changes to Task SDK #45752

Closed
kaxil opened this issue Jan 17, 2025 · 1 comment · Fixed by #45924
Closed

Port Registering of Asset Changes to Task SDK #45752

kaxil opened this issue Jan 17, 2025 · 1 comment · Fixed by #45924
Assignees
@amoghrajesh
Labels
area:task-execution-interface-aip72 AIP-72: Task Execution Interface (TEI) aka Task SDK
Milestone
Airflow 3.0.0

Comments

@kaxil
Copy link
Member

kaxil commented Jan 17, 2025

Assets are stored in DB after a Task completes execution with SUCCESS state as shown in the following code.

airflow/airflow/models/taskinstance.py

Line 359 in 051e617

ti._register_asset_changes(events=context["outlet_events"], session=session)

This includes extracting outlets from the task and storing them in the DB.

We should port this and related changes to the Task SDK's task_runner.py

airflow/task_sdk/src/airflow/sdk/execution_time/task_runner.py

Line 600 in 051e617

def finalize(log: Logger): ...

@kaxil kaxil added the area:task-execution-interface-aip72 AIP-72: Task Execution Interface (TEI) aka Task SDK label Jan 17, 2025
@kaxil kaxil added this to the Airflow 3.0.0 milestone Jan 17, 2025
@kaxil kaxil self-assigned this Jan 17, 2025
kaxil added a commit that referenced this issue Jan 17, 2025
@kaxil
AIP-72: Add support for outlet_events in Task Context (#45727)
418b701 
part of #45717 and #45752

This PR adds support for `outlet_events` in Context dict within the Task SDK by adding an endpoint on the API Server which is fetched when outlet_events is accessed.
@kaxil kaxil assigned amoghrajesh and unassigned kaxil Jan 20, 2025
@kaxil
Copy link
Member Author

kaxil commented Jan 20, 2025

@amoghrajesh To test this you could run, https://github.com/apache/airflow/blob/main/airflow/example_dags/example_asset_alias.py example DAG

@kaxil kaxil moved this from Todo to In Progress in AIP-72 - Task Execution Interface and SDK Jan 21, 2025
@amoghrajesh amoghrajesh mentioned this issue Jan 22, 2025
Merged
dauinh pushed a commit to dauinh/airflow that referenced this issue Jan 24, 2025
@kaxil @dauinh
AIP-72: Add support for outlet_events in Task Context (apache#45727)
981985d 
part of apache#45717 and apache#45752

This PR adds support for `outlet_events` in Context dict within the Task SDK by adding an endpoint on the API Server which is fetched when outlet_events is accessed.
got686-yandex pushed a commit to got686-yandex/airflow that referenced this issue Jan 30, 2025
@kaxil @got686-yandex
AIP-72: Add support for outlet_events in Task Context (apache#45727)
430cb64 
part of apache#45717 and apache#45752

This PR adds support for `outlet_events` in Context dict within the Task SDK by adding an endpoint on the API Server which is fetched when outlet_events is accessed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@amoghrajesh amoghrajesh

Labels
area:task-execution-interface-aip72 AIP-72: Task Execution Interface (TEI) aka Task SDK
Projects
AIP-72 - Task Execution Interface and SDK
Status: Done
Milestone
Airflow 3.0.0
Development

Successfully merging a pull request may close this issue.

AIP-72: Port Registering of Asset Changes to Task SDK on task completion astronomer/airflow
2 participants
@kaxil @amoghrajesh