Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Move apache.iceberg provider to new providers structure #45809

Merged
merged 1 commit into from
Jan 21, 2025

Conversation

potiuk
Copy link
Member

@potiuk potiuk commented Jan 20, 2025

This also add the feature of managing devel-dependencies via dependency groups in pyproject.toml and adjust tests to cover case where providers can be two levels deeper.


^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

@potiuk potiuk force-pushed the move-apache-iceberg branch from 95d347f to 6cd7f28 Compare January 20, 2025 21:05
@potiuk
Copy link
Member Author

potiuk commented Jan 20, 2025

Did I say I love 💔 sphinx .... ?

This also add the feature of managing devel-dependencies via
dependency groups in pyproject.toml and adjust tests to cover
case where providers can be two levels deeper.
@potiuk potiuk force-pushed the move-apache-iceberg branch from 6cd7f28 to 92c2331 Compare January 21, 2025 15:27
@potiuk
Copy link
Member Author

potiuk commented Jan 21, 2025

OK. I think I got to an agreement with Sphinx finally 🤞 ...

I also moved "example_dags" (i.e. system tests) from tests.system.providers.<provider_id> to tests.system.<provider_id>. I realized that tests.system. packages were previously exposed via inventory - and if there are any cross-providers references in the documentation, that would break the documentation links cross-providers. And we do not really need that extra "providers" package in the middle.

I moved the airbyte system tests back as well in this change.

This is the current apache.iceberg provider's inventory:

root@986a47c0f1f9:/opt/airflow/docs/_inventory_cache# python3 -m sphinx.ext.intersphinx apache-airflow-providers-apache-iceberg/objects.inv
py:attribute
    airflow.providers.apache.iceberg.hooks.iceberg.IcebergHook.conn_id                                         : _api/airflow/providers/apache/iceberg/hooks/iceberg/index.html#airflow.providers.apache.iceberg.hooks.iceberg.IcebergHook.conn_id
    airflow.providers.apache.iceberg.hooks.iceberg.IcebergHook.conn_name_attr                                         : _api/airflow/providers/apache/iceberg/hooks/iceberg/index.html#airflow.providers.apache.iceberg.hooks.iceberg.IcebergHook.conn_name_attr
    airflow.providers.apache.iceberg.hooks.iceberg.IcebergHook.conn_type                                         : _api/airflow/providers/apache/iceberg/hooks/iceberg/index.html#airflow.providers.apache.iceberg.hooks.iceberg.IcebergHook.conn_type
    airflow.providers.apache.iceberg.hooks.iceberg.IcebergHook.default_conn_name                                         : _api/airflow/providers/apache/iceberg/hooks/iceberg/index.html#airflow.providers.apache.iceberg.hooks.iceberg.IcebergHook.default_conn_name
    airflow.providers.apache.iceberg.hooks.iceberg.IcebergHook.hook_name                                         : _api/airflow/providers/apache/iceberg/hooks/iceberg/index.html#airflow.providers.apache.iceberg.hooks.iceberg.IcebergHook.hook_name
py:class
    airflow.providers.apache.iceberg.hooks.iceberg.IcebergHook                                         : _api/airflow/providers/apache/iceberg/hooks/iceberg/index.html#airflow.providers.apache.iceberg.hooks.iceberg.IcebergHook
py:data
    airflow.providers.apache.iceberg.__version__                                         : _api/airflow/providers/apache/iceberg/index.html#airflow.providers.apache.iceberg.__version__
    airflow.providers.apache.iceberg.hooks.iceberg.TOKENS_ENDPOINT                                         : _api/airflow/providers/apache/iceberg/hooks/iceberg/index.html#airflow.providers.apache.iceberg.hooks.iceberg.TOKENS_ENDPOINT
    tests.system.apache.iceberg.example_iceberg.bash_command                                         : _api/tests/system/apache/iceberg/example_iceberg/index.html#tests.system.apache.iceberg.example_iceberg.bash_command
    tests.system.apache.iceberg.example_iceberg.test_run                                         : _api/tests/system/apache/iceberg/example_iceberg/index.html#tests.system.apache.iceberg.example_iceberg.test_run
py:method
    airflow.providers.apache.iceberg.hooks.iceberg.IcebergHook.get_conn                                         : _api/airflow/providers/apache/iceberg/hooks/iceberg/index.html#airflow.providers.apache.iceberg.hooks.iceberg.IcebergHook.get_conn
    airflow.providers.apache.iceberg.hooks.iceberg.IcebergHook.get_token_macro                                         : _api/airflow/providers/apache/iceberg/hooks/iceberg/index.html#airflow.providers.apache.iceberg.hooks.iceberg.IcebergHook.get_token_macro
    airflow.providers.apache.iceberg.hooks.iceberg.IcebergHook.get_ui_field_behaviour                                         : _api/airflow/providers/apache/iceberg/hooks/iceberg/index.html#airflow.providers.apache.iceberg.hooks.iceberg.IcebergHook.get_ui_field_behaviour
    airflow.providers.apache.iceberg.hooks.iceberg.IcebergHook.test_connection                                         : _api/airflow/providers/apache/iceberg/hooks/iceberg/index.html#airflow.providers.apache.iceberg.hooks.iceberg.IcebergHook.test_connection
py:module
    airflow.providers.apache.iceberg                                                 : _api/airflow/providers/apache/iceberg/index.html#module-airflow.providers.apache.iceberg
    airflow.providers.apache.iceberg.hooks                                           : _api/airflow/providers/apache/iceberg/hooks/index.html#module-airflow.providers.apache.iceberg.hooks
    airflow.providers.apache.iceberg.hooks.iceberg                                         : _api/airflow/providers/apache/iceberg/hooks/iceberg/index.html#module-airflow.providers.apache.iceberg.hooks.iceberg
    tests.system.apache.iceberg                                                      : _api/tests/system/apache/iceberg/index.html#module-tests.system.apache.iceberg
    tests.system.apache.iceberg.example_iceberg                                         : _api/tests/system/apache/iceberg/example_iceberg/index.html#module-tests.system.apache.iceberg.example_iceberg
std:doc
    _api/airflow/providers/apache/iceberg/hooks/iceberg/index airflow.providers.apache.iceberg.hooks.iceberg: _api/airflow/providers/apache/iceberg/hooks/iceberg/index.html
    _api/airflow/providers/apache/iceberg/hooks/index airflow.providers.apache.iceberg.hooks  : _api/airflow/providers/apache/iceberg/hooks/index.html
    _api/airflow/providers/apache/iceberg/index airflow.providers.apache.iceberg        : _api/airflow/providers/apache/iceberg/index.html
    _api/tests/system/apache/iceberg/example_iceberg/index tests.system.apache.iceberg.example_iceberg: _api/tests/system/apache/iceberg/example_iceberg/index.html
    _api/tests/system/apache/iceberg/index   tests.system.apache.iceberg             : _api/tests/system/apache/iceberg/index.html
    changelog                                Changelog                               : changelog.html
    commits                                  Package apache-airflow-providers-apache-iceberg: commits.html
    connections                              Connecting to Iceberg                   : connections.html
    index                                    apache-airflow-providers-apache-iceberg : index.html
    installing-providers-from-sources        Installing from sources                 : installing-providers-from-sources.html
    security                                 Releasing security patches              : security.html
std:label
    genindex                                 Index                                   : genindex.html
    howto/connection:iceberg                 Connecting to Iceberg                   : connections.html#howto-connection-iceberg
    modindex                                 Module Index                            : py-modindex.html
    py-modindex                              Python Module Index                     : py-modindex.html
    search                                   Search Page                             : search.html

@potiuk potiuk merged commit 4280b83 into apache:main Jan 21, 2025
91 checks passed
@potiuk potiuk deleted the move-apache-iceberg branch January 21, 2025 16:26
dauinh pushed a commit to dauinh/airflow that referenced this pull request Jan 24, 2025
This also add the feature of managing devel-dependencies via
dependency groups in pyproject.toml and adjust tests to cover
case where providers can be two levels deeper.
@utkarsharma2 utkarsharma2 added the changelog:skip Changes that should be skipped from the changelog (CI, tests, etc..) label Jan 27, 2025
gpathak128 pushed a commit to gpathak128/airflow that referenced this pull request Jan 29, 2025
This also add the feature of managing devel-dependencies via
dependency groups in pyproject.toml and adjust tests to cover
case where providers can be two levels deeper.
got686-yandex pushed a commit to got686-yandex/airflow that referenced this pull request Jan 30, 2025
This also add the feature of managing devel-dependencies via
dependency groups in pyproject.toml and adjust tests to cover
case where providers can be two levels deeper.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area:dev-tools area:providers changelog:skip Changes that should be skipped from the changelog (CI, tests, etc..) kind:documentation provider:apache-iceberg
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants