Removal of AIP-44 code #44436

potiuk · 2024-11-27T22:57:54Z

Body

We eventually decided to drop AIP-44. We are about to remove it and here is the list of things that should be removed:

removal of "enble-aip-44" controlled parts of CI workflows and breeze
removal of --database-isolation Breeze flag (and corresponding scripts/test shennigans
removal of AIRFLOW_ENABLE_AIP_44 occurrences (ci/code) and _ENABLE_AIP_44 , InternalApiConfig, database_access_isolation
Swap internal RPC server for API server in the helm chart #44463
removal of "internal_api_v1.yaml" and "internal-api" CLI
removal of @internal_api_call decorators (82 methods) - re-join back methods that were separated out from the main code - when methods start with _:
- ./task_sdk/src/airflow/sdk/definitions/asset/init.py: expand_alias_to_assets
- ./airflow/secrets/metastore.py: _fetch_connection
- ./airflow/secrets/metastore.py: _fetch_variable
- ./airflow/jobs/job.py: _kill
- ./airflow/jobs/job.py: _fetch_from_db
- ./airflow/jobs/job.py: _add_to_db
- ./airflow/jobs/job.py: _update_in_db
- ./airflow/jobs/job.py: _update_heartbeat
- ./airflow/jobs/job.py: most_recent_job
- ./airflow/api/common/trigger_dag.py: trigger_dag
- ./airflow/assets/manager.py: register_asset_change
- ./airflow/dag_processing/processor.py: update_import_errors
- ./airflow/dag_processing/processor.py: _validate_task_pools_and_update_dag_warnings
- ./airflow/dag_processing/processor.py: execute_callbacks
- ./airflow/dag_processing/processor.py: execute_callbacks_without_dag
- ./airflow/dag_processing/processor.py: _execute_task_callbacks
- ./airflow/dag_processing/processor.py: save_dag_to_db
- ./airflow/dag_processing/manager.py: deactivate_stale_dags
- ./airflow/dag_processing/manager.py: _fetch_callbacks
- ./airflow/dag_processing/manager.py: _get_priority_filelocs
- ./airflow/dag_processing/manager.py: clear_nonexistent_import_errors
- ./airflow/models/dagrun.py: fetch_task_instances
- ./airflow/models/dagrun.py: _check_last_n_dagruns_failed
- ./airflow/models/dagrun.py: fetch_task_instance
- ./airflow/models/dagrun.py: get_previous_dagrun
- ./airflow/models/dagrun.py: get_previous_scheduled_dagrun
- ./airflow/models/dagrun.py: _get_log_template
- ./airflow/models/trigger.py: from_object
- ./airflow/models/trigger.py: bulk_fetch
- ./airflow/models/trigger.py: clean_unused
- ./airflow/models/trigger.py: submit_event
- ./airflow/models/trigger.py: submit_failure
- ./airflow/models/trigger.py: ids_for_triggerer
- ./airflow/models/trigger.py: assign_unassigned
- ./airflow/models/xcom_arg.py: _get_task_map_length
- ./airflow/models/renderedtifields.py: _update_runtime_evaluated_template_fields
- ./airflow/models/serialized_dag.py: get_serialized_dag
- ./airflow/models/dag.py: fetch_callback
- ./airflow/models/dag.py: fetch_dagrun
- ./airflow/models/dag.py: get_current
- ./airflow/models/dag.py: get_paused_dag_ids
- ./airflow/models/dag.py: deactivate_deleted_dags
- ./airflow/models/skipmixin.py: _skip
- ./airflow/models/skipmixin.py: _skip_all_except
- ./airflow/models/xcom.py: set
- ./airflow/models/xcom.py: get_value
- ./airflow/models/xcom.py: get_one
- ./airflow/models/xcom.py: clear
- ./airflow/models/taskinstance.py: _merge_ti
- ./airflow/models/taskinstance.py: _add_log
- ./airflow/models/taskinstance.py: _update_ti_heartbeat
- ./airflow/models/taskinstance.py: _xcom_pull
- ./airflow/models/taskinstance.py: _get_template_context
- ./airflow/models/taskinstance.py: _handle_failure
- ./airflow/models/taskinstance.py: _record_task_map_for_downstreams
- ./airflow/models/taskinstance.py: _update_rtif
- ./airflow/models/taskinstance.py: _defer_task
- ./airflow/models/taskinstance.py: _handle_reschedule
- ./airflow/models/taskinstance.py: get_task_instance
- ./airflow/models/taskinstance.py: _clear_xcom_data
- ./airflow/models/taskinstance.py: _set_state
- ./airflow/models/taskinstance.py: _get_dagrun
- ./airflow/models/taskinstance.py: _check_and_change_state_before_execution
- ./airflow/models/taskinstance.py: _register_asset_changes_int
- ./airflow/models/taskinstance.py: save_to_db
- ./airflow/models/dagwarning.py: purge_inactive_dag_warnings
- ./airflow/models/variable.py: _set
- ./airflow/models/variable.py: _update
- ./airflow/models/variable.py: _delete
- ./airflow/cli/commands/task_command.py: _get_ti_db_access
- ./airflow/utils/log/file_task_handler.py: _render_filename_db_access
- ./airflow/utils/cli_action_loggers.py: _default_action_log_internal
- ./airflow/sensors/base.py: _orig_start_date
- ./providers/src/airflow/providers/edge/models/edge_logs.py: push_logs
- ./providers/src/airflow/providers/edge/models/edge_job.py: reserve_task
- ./providers/src/airflow/providers/edge/models/edge_job.py: set_state
- ./providers/src/airflow/providers/edge/models/edge_worker.py: register_worker
- ./providers/src/airflow/providers/edge/models/edge_worker.py: set_state
- ./tests/api_internal/test_internal_api_call.py: fake_method
- ./tests/api_internal/test_internal_api_call.py: fake_method_with_params
- ./tests/api_internal/test_internal_api_call.py: fake_class_method_with_params
- ./tests/api_internal/test_internal_api_call.py: fake_class_method_with_serialized_params
removal of *Pydantic models

BTW. Command to generate list of methods:

find . -name '*.py' | xargs grep -A 4 '@internal_api_call' | grep -v "@internal_api_call" | grep "def " | sed 's/(.*//' | sed "s/-/: /" | sed "s/^/   - [ ] /" | sed "s/def //"

Committer

I acknowledge that I am a maintainer/committer of the Apache Airflow project.

The text was updated successfully, but these errors were encountered:

potiuk · 2024-11-27T22:58:29Z

cc: @jscheffl @rawwar @kaxil @ashb

Part of apache#44436

Part of #44436

As part of removing AIP-44 we remove database isolation mode and remove pytest markers that skipped some tests. Part of apache#44436

As part of removing AIP-44 we remove database isolation mode and remove pytest markers that skipped some tests. Part of #44436 --------- Co-authored-by: Kalyan R <[email protected]>

This change removes all configuration that controls AIP-44 behaviour. It does not yet remove all the related code, this will be a follow up but it removes all the controls that determine if AIP-44 is enabled or not and removes all the Traceback Session/Disabling of DB session modifications that were used in "database isolation" mode. The "database isolation" mode has been disabled in apache#44441 so there was no easy way to enable it anyoway - this change removes the capability to use database isolation mode completely. Part of apache#44436

Part of apache#44436

As part of removing AIP-44 we remove database isolation mode and remove pytest markers that skipped some tests. Part of apache#44436 --------- Co-authored-by: Kalyan R <[email protected]>

This change removes all configuration that controls AIP-44 behaviour. It does not yet remove all the related code, this will be a follow up but it removes all the controls that determine if AIP-44 is enabled or not and removes all the Traceback Session/Disabling of DB session modifications that were used in "database isolation" mode. The "database isolation" mode has been disabled in apache#44441 so there was no easy way to enable it anyoway - this change removes the capability to use database isolation mode completely. Part of apache#44436

This change removes all configuration that controls AIP-44 behaviour. It does not yet remove all the related code, this will be a follow up but it removes all the controls that determine if AIP-44 is enabled or not and removes all the Traceback Session/Disabling of DB session modifications that were used in "database isolation" mode. The "database isolation" mode has been disabled in #44441 so there was no easy way to enable it anyoway - this change removes the capability to use database isolation mode completely. Part of #44436

This PR removes CLI and openapi specification for the API. Part of apache#44436

ashb · 2024-11-28T21:44:45Z

I'm currently re-working all the dag processing stuff to use TaskSDK code etc (but don't know when I'll have a PR for that -- likely Monday or Tuesday) so it might be worth skipping over those. But on the other hand the changes are quite simple to do so it maybe makes sense to do it anyway.

potiuk · 2024-11-28T21:52:29Z

I think we can progress on those regardless. One-by-one.

This PR removes CLI and openapi specification for the API. Part of #44436

jscheffl · 2024-11-28T22:44:52Z

I'd support - but probably I'm only having time on the weekend. As I was also preparing to move Edge-Worker off the internal API, I have 4 PR's (4 slices instead of one elephant) and asked @kaxil already for review. If somebody else has time... these 4 PRs implicitly also remove parts (5 hits) listed from the work and lift them to FastAPI.

https://github.com/apache/airflow/pulls?q=is%3Aopen+is%3Apr+author%3Ajscheffl+label%3Aprovider%3Aedge

potiuk · 2024-11-29T20:36:41Z

I take "airflow/secrets/" to get the founding removal PR :).

Part of apache#44436

potiuk · 2024-11-29T20:59:05Z

First small, exemplary PR: #44489

Part of #44436

The Pudanic models have been used in a number of places still and we are using them also for context passing for PythonVirtualEnv and ExternaPythonOperator - this PR removes all the models and their usages. Closes: apache#44436 # Please enter the commit message for your changes. Lines starting

potiuk · 2024-12-12T21:37:06Z

CLOSED !!!!

kaxil · 2024-12-12T21:39:51Z

Nice work 👏

The Pudanic models have been used in a number of places still and we are using them also for context passing for PythonVirtualEnv and ExternaPythonOperator - this PR removes all the models and their usages. Closes: apache#44436 # Please enter the commit message for your changes. Lines starting

Part of apache#44436

As part of removing AIP-44 we remove database isolation mode and remove pytest markers that skipped some tests. Part of apache#44436 --------- Co-authored-by: Kalyan R <[email protected]>

This change removes all configuration that controls AIP-44 behaviour. It does not yet remove all the related code, this will be a follow up but it removes all the controls that determine if AIP-44 is enabled or not and removes all the Traceback Session/Disabling of DB session modifications that were used in "database isolation" mode. The "database isolation" mode has been disabled in apache#44441 so there was no easy way to enable it anyoway - this change removes the capability to use database isolation mode completely. Part of apache#44436

This PR removes CLI and openapi specification for the API. Part of apache#44436

Part of apache#44436

Part of apache#44436 Co-authored-by: kalyanr <[email protected]>

Part of apache#44436

…che#44549) This is the last "internal_api_call" change from AIP-44 removal and it not only removes the dag_processsing/manager calls but also the internal_api_call itself. Part of apache#44436

Part of apache#44436

The Pudanic models have been used in a number of places still and we are using them also for context passing for PythonVirtualEnv and ExternaPythonOperator - this PR removes all the models and their usages. Closes: apache#44436 # Please enter the commit message for your changes. Lines starting

potiuk added the kind:meta High-level information important to the community label Nov 27, 2024

dosubot bot added the area:CI Airflow's tests and continious integration label Nov 27, 2024

potiuk added a commit to potiuk/airflow that referenced this issue Nov 27, 2024

Remove AIP-44 related CI configuration and workflows

b76f73b

Part of apache#44436

potiuk mentioned this issue Nov 27, 2024

Remove AIP-44 related CI configuration and workflows #44437

Merged

potiuk added a commit that referenced this issue Nov 28, 2024

Remove AIP-44 related CI configuration and workflows (#44437)

77912f8

Part of #44436

potiuk added a commit to potiuk/airflow that referenced this issue Nov 28, 2024

Get rid of database isolation option

1c3a6f2

As part of removing AIP-44 we remove database isolation mode and remove pytest markers that skipped some tests. Part of apache#44436

potiuk mentioned this issue Nov 28, 2024

Get rid of database isolation option #44441

Merged

potiuk added a commit to potiuk/airflow that referenced this issue Nov 28, 2024

Get rid of database isolation option

3e09bbf

As part of removing AIP-44 we remove database isolation mode and remove pytest markers that skipped some tests. Part of apache#44436

potiuk mentioned this issue Nov 28, 2024

Remove AIP-44 configuration from the code #44454

Merged

prabhusneha pushed a commit to astronomer/airflow that referenced this issue Nov 28, 2024

Remove AIP-44 related CI configuration and workflows (apache#44437)

dcf4b7d

Part of apache#44436

potiuk added a commit to potiuk/airflow that referenced this issue Nov 28, 2024

Remove internal-api command

f175b91

This PR removes CLI and openapi specification for the API. Part of apache#44436

potiuk mentioned this issue Nov 28, 2024

Remove internal-api command #44462

Merged

potiuk added a commit to potiuk/airflow that referenced this issue Nov 28, 2024

Remove internal-api command

38cefe0

This PR removes CLI and openapi specification for the API. Part of apache#44436

ashb mentioned this issue Nov 28, 2024

Swap internal RPC server for API server in the helm chart #44463

Merged

potiuk added a commit that referenced this issue Nov 28, 2024

Remove internal-api command (#44462)

1f18458

This PR removes CLI and openapi specification for the API. Part of #44436

shahar1 mentioned this issue Nov 29, 2024

[Mothership PR] Remove internal_api_call decorator #44486

Closed

potiuk added a commit to potiuk/airflow that referenced this issue Nov 29, 2024

Remove internal_api_call from airflow.secrets.metastore

0c2438a

Part of apache#44436

potiuk mentioned this issue Nov 29, 2024

Remove internal_api_call from airflow.secrets.metastore #44489

Merged

potiuk mentioned this issue Dec 1, 2024

Remove all remnants of "internal_api_call" #44551

Merged

potiuk added a commit that referenced this issue Dec 1, 2024

Remove all remnants of "internal_api_call" (#44551)

8c019cf

Part of #44436

potiuk mentioned this issue Dec 1, 2024

Remove Pydanitc models introduced for AIP-44 #44552

Merged

potiuk closed this as completed in #44552 Dec 12, 2024

potiuk closed this as completed in 8af1bbd Dec 12, 2024

eladkal mentioned this issue Dec 22, 2024

Status of testing Providers that were prepared on December 22, 2024 #45148

Closed

LefterisXefteris pushed a commit to LefterisXefteris/airflow that referenced this issue Jan 5, 2025

Remove AIP-44 related CI configuration and workflows (apache#44437)

b7d7583

Part of apache#44436

LefterisXefteris pushed a commit to LefterisXefteris/airflow that referenced this issue Jan 5, 2025

Remove internal-api command (apache#44462)

9ef2ae9

This PR removes CLI and openapi specification for the API. Part of apache#44436

LefterisXefteris pushed a commit to LefterisXefteris/airflow that referenced this issue Jan 5, 2025

remove internal_api_call from airflow.secrets.metastore (apache#44489)

4a6590a

Part of apache#44436

LefterisXefteris pushed a commit to LefterisXefteris/airflow that referenced this issue Jan 5, 2025

Remove AIP-44 from Job (apache#44493)

4d32873

Part of apache#44436

LefterisXefteris pushed a commit to LefterisXefteris/airflow that referenced this issue Jan 5, 2025

Remove AIP-44 from taskinstance (apache#44540)

0806683

Part of apache#44436 Co-authored-by: kalyanr <[email protected]>

LefterisXefteris pushed a commit to LefterisXefteris/airflow that referenced this issue Jan 5, 2025

Remove AIP-44 code from renderedtifields.py (apache#44546)

bd80719

Part of apache#44436

LefterisXefteris pushed a commit to LefterisXefteris/airflow that referenced this issue Jan 5, 2025

Remove all remnants of "internal_api_call" (apache#44551)

644a9e9

Part of apache#44436

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Removal of AIP-44 code #44436

Removal of AIP-44 code #44436

potiuk commented Nov 27, 2024 •

edited

Loading

potiuk commented Nov 27, 2024 •

edited

Loading

ashb commented Nov 28, 2024

potiuk commented Nov 28, 2024

jscheffl commented Nov 28, 2024

potiuk commented Nov 29, 2024

potiuk commented Nov 29, 2024

potiuk commented Dec 12, 2024

kaxil commented Dec 12, 2024

Removal of AIP-44 code #44436

Removal of AIP-44 code #44436

Comments

potiuk commented Nov 27, 2024 • edited Loading

Body

Committer

potiuk commented Nov 27, 2024 • edited Loading

ashb commented Nov 28, 2024

potiuk commented Nov 28, 2024

jscheffl commented Nov 28, 2024

potiuk commented Nov 29, 2024

potiuk commented Nov 29, 2024

potiuk commented Dec 12, 2024

kaxil commented Dec 12, 2024

potiuk commented Nov 27, 2024 •

edited

Loading

potiuk commented Nov 27, 2024 •

edited

Loading