You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
An <a href="https://github.com/argoproj/argo-workflows/blob/master/docs/executor_plugins.md">Executor Plugin</a> for <a href="https://argoproj.github.io/argo-workflows/">Argo Workflows</a> that runs WebAssembly modules! 🚀
9
+
Runs WebAssembly in your Argo Workflows! 🚀
10
10
<br />
11
11
<a href="https://github.com/Shark/wasm-workflows-plugin/#about-the-project"><strong>Find out why that's awesome »</strong></a>
12
12
<!--
@@ -47,108 +47,61 @@
47
47
48
48
## About The Project
49
49
50
-
This is a tool that allows you run WebAssembly modules instead of containers for your steps in [Argo Workflows](https://argoproj.github.io/argo-workflows/). You might rightfully ask yourself what problem this solves for you.
50
+
This is an <ahref="https://github.com/argoproj/argo-workflows/blob/master/docs/executor_plugins.md">Executor Plugin</a> for <ahref="https://argoproj.github.io/argo-workflows/">Argo Workflows</a> that runs WebAssembly modules!
51
51
52
-
The two most important aspects are security and performance:
52
+
These are the benefits of using Wasm instead of Docker containers in your workflows:
53
53
54
-
*:lock:**Security**
54
+
*:airplane:**Portability**
55
55
56
-
The [list of things to do](https://cheatsheetseries.owasp.org/cheatsheets/Docker_Security_Cheat_Sheet.html) when you want to run containers securely is long and the topic is more complex than even ambitious users care about. Containers are [vulnerable in many ways](https://ieeexplore.ieee.org/document/8693491) because of their denylist approach to security: they're allowed to do many things by default.
56
+
You must build Docker containers individually for every CPU architecture. Working on a Mac with Apple Silicon, but your Kubernetes nodes run on Intel CPUs? You'll cross-compile your container images all day.
57
57
58
-
WebAssembly's security model is the opposite. As with smartphone apps, they must be given permission for potentially infringing tasks. For example, you might want to give a module the permission to read and write files but not communicate over the internet.
58
+
Wasm modules are architecture-independent by design. Build once, run everywhere.
59
59
60
-
Container images from third parties you don't know are usually a security nightmare. With WebAssembly, you can run code you don't fully trust with more confidence. Say you have a workflow step that renders Markdown. When the author of your Markdown parser container image decides to deliver a crypto miner instead, most Kubernetes setups will happily run it. If you were using this project and a WebAssembly module: zero chance, since it's easy for you to know that the step doesn't need the network but only takes an input parameter and produces some output. [This example is not made up](https://www.trendmicro.com/vinfo/fr/security/news/virtualization-and-cloud/malicious-docker-hub-container-images-cryptocurrency-mining).
60
+
*:runner:**Performance**
61
61
62
-
<details>
63
-
<summary>More about the difference between containers and Wasm modules</summary>
<p>Linux processes use more than 300 system calls for any task that involves sharing data with outside of a process. Containers are a combination of different Linux Kernel technologies (namespaces, cgroups etc.) that segment one computer into many seemlingly independent containers. But this very much depends on a) the secure implementation of all syscalls not to leak anything and b) trust in the application inside the container to do what the user intends it to.</p>
66
-
<p>Wasm modules are very restricted by default. We use application-level capabilities to allow them to access external resources like the network, S3 object stores, or the filesystem. The modules are the capability consumers, the Wasm runtime is the capability provider. The capability provider translates the requests from the Wasm module and acts as a secure proxy to the outside.</p>
67
-
</details>
62
+
It takes a while for Kubernetes to spin up a container and run your code. The process has quite a few steps: pulling a container image, often 100s of megabytes in size, creating namespaces and virtual network interfaces. Starting the runtime for interpreted languages takes a while, too.
68
63
69
-
*:runner:**Performance**
64
+
Wasm does not emulate a complete operating system as containers do. They are a much simpler abstraction. This means that a module executes in a matter of milliseconds.
65
+
66
+
*:lock:**Security**
70
67
71
-
Containers have some overhead: for each workflow step, Argo creates new Kubernetes Pod. This Pod has several containers to enable all the Argo features, your code is just one of them. All the containers must execute, then results are gathered and sent back to Argo. This all takes time: container images are often towards 100s of megabytes, they may rely on interpreted languages like Python or have huge dependencies leading to a slow start time. You may know the [Cold Start issue](https://aws.amazon.com/blogs/compute/operating-lambda-performance-optimization-part-1/) with Function-as-a-Service. In Argo, every workflow step is a cold start.
68
+
Securing a container runtime [is a challenge](https://cheatsheetseries.owasp.org/cheatsheets/Docker_Security_Cheat_Sheet.html) because [containers are vulnerable in many ways](https://ieeexplore.ieee.org/document/8693491). Containers are powerful by design.
72
69
73
-
Because WebAssembly modules don't have to bring a whole operating system, they're much smaller. And there is less setup work to do, even for interpreted languages. This means that a module can be run in a matter of milliseconds rather than tens of seconds.
70
+
Wasm is a minimal runtime that is just powerful enough to run a program. Rather than allowing everything by default, its security works more like on a smartphone, where you give apps permissions explicitly.
74
71
75
-
WebAssembly is a new technology in the browser and even more so in the Cloud-Native ecosystem. But there are several ways to ease the transition:
72
+
[Read more about the benefits here.](doc/benefits.md)
76
73
77
-
* Containers and Wasm modules play together just fine in the same workflow. Existing container setups don't have to be migrated just because they could. Use Wasm for the tasks for which is a good fit and leave the rest to containers.
74
+
Even though Wasm is a new technology in Cloud Native, incorporating Wasm into your workflow is seamless:
78
75
79
-
*You can find ready-to-use templates for popular programming languages in the [`wasm-modules/templates/`](wasm-modules/templates/) folder.
76
+
*Containers and Wasm modules co-exist in the same workflow. You can pass artifacts and parameters between them.
80
77
81
-
* We will provide pre-made modules for popular use cases such as image and text processing, API connectors, etc.
78
+
* We have included [ready-to-use templates](wasm-modules/templates/), [examples](wasm-modules/examples/), and even [some useful modules for running off-the-shelf](wasm-modules/contrib/).
82
79
83
80
### Built with
84
81
85
-
Open Source software stands on the shoulders of giants. It wouldn't have been possible to build this tool with very little extra work without the authors of these lovely projects below.
82
+
Open Source software stands on the shoulders of giants. It wouldn't have been possible to build this tool without the authors of these projects:
86
83
87
84
*[Rust](https://rust-lang.org) is used to implement the Argo Executor Plugin API, pull and execute Wasm modules
88
85
*[Axum](https://github.com/tokio-rs/axum) is the Rust web framework to handle RPC calls
89
-
*[Wasmtime](https://github.com/bytecodealliance/wasmtime) is the WebAssembly Virtual Machine ([WASI](https://wasi.dev) is [supported](https://crates.io/crates/wasmtime-wasi), too)
90
-
*[wit-bindgen](https://github.com/bytecodealliance/wit-bindgen) provides the interface between this project as the Wasm host and the Wasm modules
86
+
*[Wasmtime](https://github.com/bytecodealliance/wasmtime) is the WebAssembly Virtual Machine with [WASI support](https://wasi.dev)
91
87
*[oci-distribution](https://crates.io/crates/oci-distribution) allows the tool to pull Wasm modules from OCI registries
You must install Argo Workflows (v3.3.0 or newer) and the [`argo` CLI](https://argoproj.github.io/argo-workflows/cli/). `kubectl` needs access to your cluster.
110
93
111
-
1. Build the plugin ConfigMap:
94
+
**Install the plugin:**
112
95
113
-
```shell
114
-
argo executor-plugin build .
115
-
```
96
+
Go to the [Releases page](https://github.com/Shark/wasm-workflows-plugin.git) and follow the descriptions for installing the plugin through the ConfigMap.
116
97
117
-
1. Register the plugin with Argo in your cluster:
98
+
**Submit your first Wasm workflow:**
118
99
119
-
Ensure to specify `--namespace` if you didn't install Argo in the default namespace.
100
+
Run `argo submit --watch https://raw.githubusercontent.com/Shark/wasm-workflows-plugin/main/wasm-modules/examples/ferris-says/workflow.yaml`.
The `wasm` template will produce an output parameter `text` with an awesome message:
104
+
The workflow produces an output parameter `text` with a cool message:
152
105
153
106
```
154
107
___________________
@@ -163,103 +116,27 @@ The `wasm` template will produce an output parameter `text` with an awesome mess
163
116
/ '-----' \
164
117
```
165
118
166
-
Input and output parameters between workflow steps work just like you'd expect. Other features like artifacts may still be on the [roadmap](#roadmap) though, which is advised to check for your use case.
167
-
168
119
### Module Development
169
120
170
-
Creating a new Wasm module is easy and works with every language.
You implement a [WASI](https://wasi.dev) module. WASI is a modular system interface for Wasm. The principle is easy: the module is given its input in a file at `/work/input.json`. It is expected to write its results to a file at `/work/result.json` and exit.
179
-
180
-
We created an easy-to-use wrapper for Rust. The wrapper abstracts all the file handling magic and lets you implement a function with a signature like this:
121
+
Creating a new Wasm module for use with Argo Workflows is described in the [Module Development Guide](doc/module-development.md).
Capabilities expand what modules can do. Out of the box, modules can take input parameters and artifacts and produce some output. Take a look at the [capabilities for wasmCloud](https://wasmcloud.dev/reference/host-runtime/capabilities/) for a more complete list of useful capabilities. The capabilities that this plugin offers will be extended in the future.
202
-
203
-
#### HTTP Capability
204
-
205
-
The HTTP capability provider allows you to make HTTP requests from your Wasm module. The capability is available in every module mode. Please refer to the [`wasi-experimental-http`](https://github.com/deislabs/wasi-experimental-http) repository for complete information of how to access the HTTP capability from your module. There you will find examples for both Rust and AssemblyScript.
206
-
207
-
When using the HTTP capability, you need to whitelist the hosts that the module is allowed to connect to. This illustrates the ease-of-use that WebAssembly's capability-oriented security model offers: for you, it's very easy to tell if a module should be able to connect outside – and now securing your code got easy.
208
-
209
-
You can find a full-featured module at [`wasm-modules/contrib/http-request`](wasm-modules/contrib/http-request/).
210
-
211
-
The module supports the following input parameters:
The plugin has two modes of how it can execute a Wasm module.
127
+
The plugin will run Wasm modules within the plugin process by default. This is the recommended mode because it's easy to set up and is powerful enough for most scenarios.
251
128
252
-
The `local` mode is the default and recommended mode. It will run Wasm modules within the plugin process. Argo will create one plugin container per workflow instance. This is fine for most use cases that don't need infinite scaling within a workflow. It's also very easy to use because there is nothing to configure: it just works.
129
+
The distributed mode creates pods for Wasm modules in a workflow task, much like Argo does for Docker containers.
253
130
254
-
The :test_tube: `distributed`mode is more advanced. It is provided as a technical prototype. This mode orchestrates Wasm modules in a Kubernetes cluster much like Argo itself. It creates a Pod for each workflow task. The Pod is executed by a virtual Kubernetes node that is provided by Krustlet. [Read more about :test_tube: `distributed` mode](doc/distributed-mode.md).
131
+
[Read more in the Distributed Execution Guide.](doc/distributed-mode.md)
255
132
256
133
## Roadmap
257
134
258
-
Our roadmap is managed on the [*Developing wasm-workflows-plugin* GitHub project board](https://github.com/users/Shark/projects/1/views/1).
135
+
We manage our roadmap on the [*Developing wasm-workflows-plugin* GitHub project board](https://github.com/users/Shark/projects/1/views/1).
259
136
260
137
## Contributing
261
138
262
-
Contributions are what make the open source community such an amazing place to learn, inspire, and create. Any contributions you make are **greatly appreciated**.
139
+
Contributions make the open source community such an amazing place to learn, inspire, and create. Any contributions you make are **greatly appreciated**.
263
140
264
141
If you have a suggestion that would make this better, please fork the repo and create a pull request. You can also simply open an issue with the tag "enhancement".
265
142
Don't forget to give the project a star! Thanks again!
The [list of things to do](https://cheatsheetseries.owasp.org/cheatsheets/Docker_Security_Cheat_Sheet.html) when you want to run containers securely is long and the topic is more complex than even ambitious users care about. Containers are [vulnerable in many ways](https://ieeexplore.ieee.org/document/8693491) because of their denylist approach to security: they're allowed to do many things by default.
6
+
7
+
WebAssembly's security model is the opposite. As with smartphone apps, they must be given permission for potentially infringing tasks. For example, you might want to give a module the permission to read and write files but not communicate over the internet.
8
+
9
+
Container images from third parties you don't know are usually a security nightmare. With WebAssembly, you can run code you don't fully trust with more confidence. Say you have a workflow step that renders Markdown. When the author of your Markdown parser container image decides to deliver a crypto miner instead, most Kubernetes setups will happily run it. If you were using this project and a WebAssembly module: zero chance, since it's easy for you to know that the step doesn't need the network but only takes an input parameter and produces some output. [This example is not made up](https://www.trendmicro.com/vinfo/fr/security/news/virtualization-and-cloud/malicious-docker-hub-container-images-cryptocurrency-mining).
10
+
11
+
<details>
12
+
<summary>More about the difference between containers and Wasm modules</summary>
<p>Linux processes use more than 300 system calls for any task that involves sharing data with outside of a process. Containers are a combination of different Linux Kernel technologies (namespaces, cgroups etc.) that segment one computer into many seemlingly independent containers. But this very much depends on a) the secure implementation of all syscalls not to leak anything and b) trust in the application inside the container to do what the user intends it to.</p>
15
+
<p>Wasm modules are very restricted by default. We use application-level capabilities to allow them to access external resources like the network, S3 object stores, or the filesystem. The modules are the capability consumers, the Wasm runtime is the capability provider. The capability provider translates the requests from the Wasm module and acts as a secure proxy to the outside.</p>
16
+
</details>
17
+
18
+
*:runner:**Performance**
19
+
20
+
Containers have some overhead: for each workflow step, Argo creates new Kubernetes Pod. This Pod has several containers to enable all the Argo features, your code is just one of them. All the containers must execute, then results are gathered and sent back to Argo. This all takes time: container images are often towards 100s of megabytes, they may rely on interpreted languages like Python or have huge dependencies leading to a slow start time. You may know the [Cold Start issue](https://aws.amazon.com/blogs/compute/operating-lambda-performance-optimization-part-1/) with Function-as-a-Service. In Argo, every workflow step is a cold start.
21
+
22
+
Because WebAssembly modules don't have to bring a whole operating system, they're much smaller. And there is less setup work to do, even for interpreted languages. This means that a module can be run in a matter of milliseconds rather than tens of seconds.
0 commit comments