Isolated plugins / operators / hooks

[jdu]

Description

Because of the complexity of some of the plugins that are starting to come out, i'd like to propose a long-term goal / feature of finding a way to isolate plugins to prevent dependency issues, and stabilize integration points and problems over the longer term.

We're starting to run into issues where plugins and extensions to airflow have really complex dependency trees which cause upgrades and installs to become much more complex as well as major versions can break previously working plugins triggering rewrites as well as version checking to decide which interface to work against.

I know i'm light on details, but doing something similar to other tools where plugins are isolated in a separate process / environment, like LSP implementations in an editor, or how Sublime handles python-based extensions (I think they run as a background service when needed), where they are spawned as a secondary process and integrate with airflow through gRPC or something similar.

Use case/motivation

This would make developing plugins and extensions easier as there can be a contractual interface that developers work against that can insulate them from changes in airflows code and classes. We're right now dealing with an issue in upgrading to 2.4.x where a plugin we rely on heavily has been a bit of a horror show on 2.4.x

a) Because it's own dependencies now conflict with airflows and some provider packages
b) Because the interface it worked against changed (albeit the interface it worked against was probably not the correct one in the first place).

Related issues

No response

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[boring-cyborg[bot]]

Thanks for opening your first issue here! Be sure to follow the issue template!

[jdu]

As an additional benefit this would free it up so that devs could develop plugins in other languages as well which would increase the potential number of maintainers and give a boost to the plugin ecosystem around airflow potentially.

[potiuk]

This would be a complete change in how plugin mechanism work, because plugins are essentially executed in the same interpreter as the "main part" of the process. For example plugins create Python objects that then are used to instantiate views in the UI, or plugins create Timetable objects that are used by scheduler to calculate when to schedule the next dag run. if we were to isolate plugins and run them in separate process, we would have to execute them separately, have separate virtualenv for plugins and serialize the objects back and forth remotely between independently runniing python interpreters. This could be done with either mutliprocessing (with different python executable) or even have a separate plugin server that would have to be used for such purpose and reused between different components).

It's not impossible, of course but it is rather complex when it comes to deployment and requires very, very careful thinking not only about the APIs and interfaces used but also about performance implications. This is a complex change that would require Airflow Improvement Proposal to be drafted, discussed and finalized in the devlist, reviewing and interating over the docs, reaching consensus and passing successful voting. If you are up to the task (just be aware that it will likely take months just to discuss it so this is a marathon, not a sprint) - feel free to start it.

Just a watch-out: it's not needed in 9x% of cases where airflow is used so this is a niche case, that's why we have not invested into it. So there is a risk that we might even decide that we do not want to implement it, if from the discussion it will come out that either complexity of the implementation, or number of changes, or potential backwards incompatibilities or performance implications will be too much of a problem. Simply the gains to achieve might be much lower than cost involved in maintenance and risks of breking existing installation. So any proposal there must take into account and discuss all the aspects that might lead to higher cost of maintenance, complexity, performance.

[potiuk]

converting it into discussion - as this is the stage it is in at the moment.

[uranusjr]

One thing I’m wondering is what kind of complexity exactly is in the plugin? Plugin code is run in the scheduler or webserver, so there’s really limited complexity in what it can do so to not clog up the entire Airflow process.

[potiuk]

There are few and the plugins are also usable in workers:

• Timetables are called directly from scheduler code (and potentially use python libraries)
• Extra links - they are currently called from webserver but they might need to use python libraries
• FAB view extensions (those are the worst as they can be pretty much anything and they can use pretty much anytthing
• macros - you can extend Airflow's jinja but providing custom macros. Those can be used in workers
• hooks, operators - discourage in Airflow 2 but users can use plugins to add extra hooks and operators that are not in DAGS/PLUGINS folder this way (also in workers)
• listeners - > they also work in workers (local task job).

Almost all of those heavily rely on the plugin components being python - either by adding Python object or by calling some python methods as callbacks. And for example if you would like to add macro or listener to local_task_job as "isolated" plugin, you would have to make such local_task_job ot have another process counterpart and marshal/unmarshall stuff - for local executor, that should likely be a subprocess, for Celery - more likely this should be a server running continuously (for performance reason) in K8S - likely you would like to run it as side container.

All that is I think possible but complexity of cross-exeucutor solution is I think too muhc for the benefit.

[jdu]

Thinking it over I think there's possible benefit in something like this in airflows case, but it might be less intrusive potentially if it was an extension to airflow in and of itself. Rather than ham-fistedly going in and refactoring large swathes of the airflow plugin, provider code, adding some additional hooks or interfaces to core, and then creating a plugin that acts as an extension manager and marshaller of some sort?

Some of the benefits I could see (keeping in mind I haven't dug into internals enough yet to have real sense of scale here):

1. Insulate extension developers from changes in core by providing a set interface to program against
2. If isolation is good enough, allow developers to create plugins/extensions/providers in different languages (go, rust, java, etc..)
3. Allow run-time installation, removal and updating of extensions (i.e. similar for instance to jenkins) from the UI
4. Allow developers to specify more clearly what versions of airflow are compatible with their extension versus mostly finding out when install fails / borks out.
5. Removes the need to sort out complex dependency issues when you have multiple packages installed alongside airflow
6. Insulate extension developers against major version updates, instead of them having to quickly refactor their code for changes in hooks or classes they need to sub-class, airflow can make the appropriate changes behind the interface to accommodate the changes (to an extent) and keep older extensions still working.

As a first sort of go around, taking the approach of creating a plugin itself that manages this stuff which hooks into airflow internals, might at least help to decide if it's feasible and the assumptions about benefits above are tangible ones.

Major concerns?

1. Performance impacts
2. Current and legacy extensions and providers

As you said, it might not be worth the effort, and it might not even align with the goals of the project overall, i'm going to pick around a bit more in airflows internals over the weekend and see if the approach of integrating a plugin itself to manage plugins is reasonable as a test.