AIP-81 Trasition of Remote Commands

[bugraoz93]

Description

Transition of Remote Commands, below are remote commands. We should integrate CLI API client and call API endpoints to eliminate sessions and direct database calls from remote commands.

• asset
• backfill
• config
• connection
• dag
• jobs
• pool
• provider
• task (This will be decided after discussion, that's why there is no sub-issue for it, it will be updated)
• variable
• version

Dependency

#45300 should be merged.

Use case/motivation

AIP-81

Related issues

No response

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[bugraoz93]

Hey @jason810496 @jx2lee @josix @Prab-27,

I still need to finalise a couple of CI and breeze steps for checking the unit test properly for capture problems and easier-to-ship code via local dev tools via breeze #48099. There are a few limitations at the moment, but I will spend time and finish that work. Then we can start creating PRs in airflow-ctl.

Auto generation update,
I did a couple of trials. For now, what most makes sense is to autogenerate via mapping (1-1) the command-operation relation within the commands. I mean let's say we have jobs operation (under airflow-ctl/src/airflowctl/api/operations.py) and list method, this means the autogeneration part will generate airflowctl jobs list with proper parameters (mandatory or not) and return the results in a standard output format.
From this point (automation implementation is not blocking anymore but CI works still is), we should focus on the features we cannot provide with this 1-1 mapping, for example, we have bulk operations where they need to be utilised for file operations we are providing under airflow-core/src/airflow/cli/commands. We should implement those behaviours with API calls. There are still a few elements we can not mimic via Public API such as the airflow dags test, we should skip these transitions under airflowctl.

Thanks for participating in making this happen! We will be unblocked in 1-2 days 🤞 and I will ping you from this umbrella issue.

[bugraoz93]

Hey @jx2lee @josix @Prab-27, @yunchipang,

The CI/Breeze tooling is merged. The issues are unblocked now. Please remember to include unit tests. A couple of things to keep in mind while working on the issues,

There are a couple of commands within the group commands (issue titles such as dags, pools, variables, etc.) that we cannot replicate through API. For example, airflow dags test. You can find a more fine-grained split in this PR (#47538), which can help answer if this command should be possible with API. Also, don't hesitate to ask. This was embedded in the airflow-core and the decisions were towards having and releasing this as a separate distribution airflowctl. You will see that the current airflow-core/cli/commands are already flattened under commands/ and we are going to populate remote commands under airflowctl.

For the CLI approach, we are going to replicate a couple of things for now, such as how CLI commands are created, so you can check airflow-core/cli for the baseline as well. For example, the commands are created in cli_config.py as the same. For sure, there will be main differences where this distribution doesn't depend on airflow-core but a running instance of the API server in airflow-core. We are going to use Client operations to communicate with the API. If you think an operation is missing, please include it.

Please install keyrings.alt into the reeze container for now if you are going to test it out. I am going to include this into devel-common soon (#48598).
pip install keyrings.alt

For now, we are going to use http://localhost:28080/auth/docs to generate our token. I am going to add an auth with a username and password soon (#47703). You should create your token and run the following to be able to use the client and communicate with the API.
airflowctl auth login --api-token=<JWT_TOKEN>
Then, you can execute your implemented commands to do development tests.

For the tests, there is already a created test suite baseline to interact with the test client. You can check this commit ( https://github.com/bugraoz93/airflow/blob/4158467766c0f5bbab284613fbbdf59277467876/tests/cli/commands/remote_commands/test_connection_command.py ) where I integrated with the connections command earlier with the client. You can check out how to use test_client.

One last small note: if you don't use airflowctl within the same breeze container with api server. Please include your api server URL to the auth command.
airflowctl auth login --api-token=<JWT_TOKEN> --api-url=<API_URL>

Please don't hesitate to ask any questions. Additionally, if you think your timeline does not match with doing these issues anymore, please let me know. Thanks!

cc: @jason810496 @jedcunningham @potiuk