Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Migrate Edge calls for Worker to FastAPI part 4 - Cleanup #44434

Merged

Conversation

jscheffl
Copy link
Contributor

@jscheffl jscheffl commented Nov 27, 2024

After PR #44311, #44330 and #44433 this PR is a cleanup and removes the old Internal API extensions.

For review: Only the last commit is relevant, the other 7 are from referenced base PRs

@jscheffl jscheffl added area:providers AIP-69 Edge Executor provider:edge Edge Executor / Worker (AIP-69) labels Nov 27, 2024
@jscheffl jscheffl requested review from kaxil and Copilot November 27, 2024 22:53
Copy link
Contributor

@Copilot Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot reviewed 11 out of 26 changed files in this pull request and generated 1 suggestion.

Files not reviewed (15)
  • docs/spelling_wordlist.txt: Language not supported
  • providers/src/airflow/providers/edge/CHANGELOG.rst: Language not supported
  • providers/src/airflow/providers/edge/worker_api/routes/_v2_routes.py: Evaluated as low risk
  • providers/src/airflow/providers/edge/worker_api/auth.py: Evaluated as low risk
  • providers/src/airflow/providers/edge/worker_api/datamodels.py: Evaluated as low risk
  • providers/src/airflow/providers/edge/worker_api/app.py: Evaluated as low risk
  • providers/src/airflow/providers/edge/worker_api/routes/rpc_api.py: Evaluated as low risk
  • providers/src/airflow/providers/edge/worker_api/routes/logs.py: Evaluated as low risk
  • providers/src/airflow/providers/edge/worker_api/routes/jobs.py: Evaluated as low risk
  • providers/src/airflow/providers/edge/init.py: Evaluated as low risk
  • providers/src/airflow/providers/edge/worker_api/routes/_v2_compat.py: Evaluated as low risk
  • providers/src/airflow/providers/edge/executors/edge_executor.py: Evaluated as low risk
  • providers/src/airflow/providers/edge/models/edge_logs.py: Evaluated as low risk
  • providers/src/airflow/providers/edge/models/edge_job.py: Evaluated as low risk
  • providers/src/airflow/providers/edge/cli/edge_command.py: Evaluated as low risk

@jscheffl jscheffl force-pushed the feature/separate-edge-api-from-internal-api-4 branch 2 times, most recently from b575110 to 2c127df Compare November 28, 2024 23:07
@jscheffl jscheffl force-pushed the feature/separate-edge-api-from-internal-api-4 branch from 2c127df to f6dc914 Compare November 30, 2024 19:40
@jscheffl jscheffl force-pushed the feature/separate-edge-api-from-internal-api-4 branch from 5a1db37 to 6ba555a Compare November 30, 2024 22:29
@jscheffl jscheffl requested review from Copilot and potiuk November 30, 2024 22:29

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot reviewed 5 out of 15 changed files in this pull request and generated no suggestions.

Files not reviewed (10)
  • providers/src/airflow/providers/edge/CHANGELOG.rst: Language not supported
  • providers/src/airflow/providers/edge/openapi/edge_worker_api_v1.yaml: Language not supported
  • providers/tests/edge/models/test_edge_job.py: Evaluated as low risk
  • providers/src/airflow/providers/edge/models/edge_logs.py: Evaluated as low risk
  • providers/src/airflow/providers/edge/models/edge_job.py: Evaluated as low risk
  • providers/src/airflow/providers/edge/cli/edge_command.py: Evaluated as low risk
  • providers/src/airflow/providers/edge/models/edge_worker.py: Evaluated as low risk
  • providers/tests/edge/cli/test_edge_command.py: Evaluated as low risk
  • providers/src/airflow/providers/edge/worker_api/app.py: Evaluated as low risk
  • providers/src/airflow/providers/edge/provider.yaml: Evaluated as low risk
Comments skipped due to low confidence (2)

providers/src/airflow/providers/edge/cli/api_client.py:113

  • The docstring is incorrect. It should be updated to: """Update the state of the worker in the central site and thereby implicitly heartbeat."""
"""Register worker with the Edge API."""

providers/src/airflow/providers/edge/cli/api_client.py:128

  • The method model_dump_json is not a standard Pydantic method. Use json() instead: WorkerQueuesBody(queues=queues, free_concurrency=free_concurrency).json(exclude_unset=True)
WorkerQueuesBody(queues=queues, free_concurrency=free_concurrency).model_dump_json(exclude_unset=True)
Copy link
Member

@potiuk potiuk left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is some devilish trickery with this one... Seems like it's no change at all:

Screenshot 2024-11-30 at 23 43 14

But ... It looks good:

@jscheffl jscheffl force-pushed the feature/separate-edge-api-from-internal-api-4 branch from 6ba555a to e176682 Compare December 1, 2024 11:05
@jscheffl
Copy link
Contributor Author

jscheffl commented Dec 1, 2024

There is some devilish trickery with this one... Seems like it's no change at all:
But ... It looks good:

Now after Part 3 is merged looks much better:

image

@jscheffl jscheffl merged commit 0d98e2b into apache:main Dec 1, 2024
65 checks passed
@potiuk
Copy link
Member

potiuk commented Dec 1, 2024

Now after Part 3 is merged looks much better:

:)

LefterisXefteris pushed a commit to LefterisXefteris/airflow that referenced this pull request Jan 5, 2025
* Remove internal API bindings after migration to FastAPI

* Move import to function preventing module import errors
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
AIP-69 Edge Executor area:providers kind:documentation provider:edge Edge Executor / Worker (AIP-69)
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants