Add Cloud Events support for #55 (#56) (#64)

* Ignore IDE files.

* Use the test file directory as a basis instead of cwd. Allows tests to be run from anywhere and enables IDE debugger

* Add support for Cloud Events. Rough draft.

I will squash a bunch of these interim commits before submitting the PR.

DO NOT SUBMIT

* Return the functions return value. Test Cloud Events SDK 0.3. Add some error handling.

Please see all the TODO questions before I finish off this PR.

DO NOT SUBMIT

* Minor cleanup. Split test code.

* Clean up unused paths, split large test files into two, ensure functions DO NOT return a custom value. General tidy-up. Support binary functions.

* Fix lint errors with black.

* Fix lint errors with black.

* Update setup.py

Co-authored-by: Dustin Ingram <[email protected]>

* Update tests/test_cloudevent_functions.py

Co-authored-by: Dustin Ingram <[email protected]>

* Update tests/test_cloudevent_functions.py

Co-authored-by: Dustin Ingram <[email protected]>

* Update src/functions_framework/__init__.py

Co-authored-by: Dustin Ingram <[email protected]>

* Update tests/test_functions/cloudevents/main.py

Co-authored-by: Dustin Ingram <[email protected]>

* Clearer imports.

* don't factor out routes.

* Add a TODO for testing the different combinations of events and signature types.

* Add cloudevent as a signature type in the argument list.

* Clarify import.

* Clarify import.

* A sample that shows how to use a CloudEvent.

* In the case of a sig type / event type mismatch throw a 400

* Update the docs to use CloudEvent sig type instead of Event sig type. Note that I wrote the "Event" type is deprecated. Not sure if this is accurate.

* Lint fixes.

* Tests for checking correct event type corresponds to correct function sig. Fixed abort import error.

* Sort imports.

* Remove old example.

* Readme to explain how to run the sample locally.

* Rename cloud_event to cloudevent

* For legacy docs, add a notice to the new docs.

* There is no 1.1 event type.

* use the term cloudevent rather than event everywhere where we are talking about a CloudEvent to disambiguate these signature types.

* Update examples/cloudevents/README.md

Co-authored-by: Dustin Ingram <[email protected]>

* Update examples/cloudevents/README.md

Co-authored-by: Dustin Ingram <[email protected]>

* Update examples/cloudevents/README.md

Co-authored-by: Dustin Ingram <[email protected]>

* Update examples/cloudevents/main.py

Co-authored-by: Dustin Ingram <[email protected]>

* Update tests/test_view_functions.py

Co-authored-by: Dustin Ingram <[email protected]>

* Add legacy event back to docs.

* Add legacy event back to docs.

* Use abort from flask for consistency and fix return in event test.

* update docs and error messages to better mirror the other runtimes.

* Minor fixes to docs w.r.t. naming.

* Update src/functions_framework/__init__.py

Co-authored-by: Dustin Ingram <[email protected]>

* Fix enum per reviewer suggestion.

* Rename text event => strucuture event.

Co-authored-by: Joel Gerard <[email protected]>
Co-authored-by: Dustin Ingram <[email protected]>

Co-authored-by: joelgerard <[email protected]>
Co-authored-by: Joel Gerard <[email protected]>

Author: 3 people

README.md

@@ -129,27 +129,57 @@ You can configure the Functions Framework using command-line flags or environmen
 | `--host`           | `HOST`                    | The host on which the Functions Framework listens for requests. Default: `0.0.0.0`                                                                                                               |
 | `--port`           | `PORT`                    | The port on which the Functions Framework listens for requests. Default: `8080`                                                                                                                  |
 | `--target`         | `FUNCTION_TARGET`         | The name of the exported function to be invoked in response to requests. Default: `function`                                                                                                     |
-| `--signature-type` | `FUNCTION_SIGNATURE_TYPE` | The signature used when writing your function. Controls unmarshalling rules and determines which arguments are used to invoke your function. Default: `http`; accepted values: `http` or `event` |
+| `--signature-type` | `FUNCTION_SIGNATURE_TYPE` | The signature used when writing your function. Controls unmarshalling rules and determines which arguments are used to invoke your function. Default: `http`; accepted values: `http` or `event` or `cloudevent` |
 | `--source`         | `FUNCTION_SOURCE`         | The path to the file containing your function. Default: `main.py` (in the current working directory)                                                                                             |
 | `--debug`          | `DEBUG`                   | A flag that allows to run functions-framework to run in debug mode, including live reloading. Default: `False`                                                                                   |
 
-# Enable CloudEvents
 
-The Functions Framework can unmarshall incoming [CloudEvents](http://cloudevents.io) payloads to `data` and `context` objects.  These will be passed as arguments to your function when it receives a request.  Note that your function must use the event-style function signature:
+# Enable Google Cloud Functions Events
+
+The Functions Framework can unmarshall incoming
+Google Cloud Functions [event](https://cloud.google.com/functions/docs/concepts/events-triggers#events) payloads to `data` and `context` objects.
+These will be passed as arguments to your function when it receives a request.
+Note that your function must use the `event`-style function signature:
+
 
 ```python
 def hello(data, context):
     print(data)
     print(context)
 ```
 
-To enable automatic unmarshalling, set the function signature type to `event` using the `--signature-type` command-line flag or the `FUNCTION_SIGNATURE_TYPE` environment variable. By default, the HTTP signature type will be used and automatic event unmarshalling will be disabled.
+To enable automatic unmarshalling, set the function signature type to `event`
+using a command-line flag or an environment variable. By default, the HTTP
+signature will be used and automatic event unmarshalling will be disabled.
+
+For more details on this signature type, check out the Google Cloud Functions
+documentation on
+[background functions](https://cloud.google.com/functions/docs/writing/background#cloud_pubsub_example).
+
+See the [running example](examples/cloud_run_event).
+
+# Enable CloudEvents
+
+The Functions Framework can unmarshall incoming
+[CloudEvent](http://cloudevents.io) payloads to a `cloudevent` object.
+It will be passed as an argument to your function when it receives a request.
+Note that your function must use the `cloudevent`-style function signature
+
+
+```python
+def hello(cloudevent):
+    print("Received event with ID: %s" % cloudevent.EventID())
+    return 200
+```
+
+To enable automatic unmarshalling, set the function signature type to `cloudevent` using the `--signature-type` command-line flag or the `FUNCTION_SIGNATURE_TYPE` environment variable. By default, the HTTP signature type will be used and automatic event unmarshalling will be disabled.
 
-For more details on this signature type, check out the Google Cloud Functions documentation on [background functions](https://cloud.google.com/functions/docs/writing/background#cloud_pubsub_example).
+See the [running example](examples/cloud_run_cloudevents).
 
 # Advanced Examples
 
-More advanced guides can be found in the [`examples/`](./examples/) directory.
+More advanced guides can be found in the [`examples/`](./examples/) directory. You can also find examples
+on using the CloudEvent Python SDK [here](https://github.com/cloudevents/sdk-python).
 
 # Contributing
 

examples/README.md

@@ -1,4 +1,5 @@
 # Python Functions Frameworks Examples
 
 * [`cloud_run_http`](./cloud_run_http/) - Deploying an HTTP function to [Cloud Run](http://cloud.google.com/run) with the Functions Framework
-* [`cloud_run_event`](./cloud_run_event/) - Deploying a CloudEvent function to [Cloud Run](http://cloud.google.com/run) with the Functions Framework
+* [`cloud_run_event`](./cloud_run_event/) - Deploying a [Google Cloud Functions Event](https://cloud.google.com/functions/docs/concepts/events-triggers#events) function to [Cloud Run](http://cloud.google.com/run) with the Functions Framework
+* [`cloud_run_cloudevents`](./cloud_run_cloudevents/) - Deploying a [CloudEvent](https://github.com/cloudevents/sdk-python) function to [Cloud Run](http://cloud.google.com/run) with the Functions Framework

examples/cloud_run_cloudevents/Dockerfile

@@ -0,0 +1,15 @@
+# Use the official Python image.
+# https://hub.docker.com/_/python
+FROM python:3.7-slim
+
+# Copy local code to the container image.
+ENV APP_HOME /app
+WORKDIR $APP_HOME
+COPY . .
+
+# Install production dependencies.
+RUN pip install gunicorn cloudevents functions-framework
+RUN pip install -r requirements.txt
+
+# Run the web service on container startup.
+CMD exec functions-framework --target=hello --signature-type=cloudevent

examples/cloud_run_cloudevents/README.md

@@ -0,0 +1,52 @@
+# Deploying a CloudEvent function to Cloud Run with the Functions Framework
+This sample uses the [Cloud Events SDK](https://github.com/cloudevents/sdk-python) to send and receive a CloudEvent on Cloud Run.
+
+## How to run this locally
+Build the Docker image:
+
+```commandline
+docker build --tag ff_example .
+```
+
+Run the image and bind the correct ports:
+
+```commandline
+docker run -p:8080:8080 ff_example
+```
+
+Send an event to the container:
+
+```python
+from cloudevents.sdk import converters
+from cloudevents.sdk import marshaller
+from cloudevents.sdk.converters import structured
+from cloudevents.sdk.event import v1
+import requests
+import json
+
+def run_structured(event, url):
+    http_marshaller = marshaller.NewDefaultHTTPMarshaller()
+    structured_headers, structured_data = http_marshaller.ToRequest(
+        event, converters.TypeStructured, json.dumps
+    )
+    print("structured CloudEvent")
+    print(structured_data.getvalue())
+
+    response = requests.post(url,
+                             headers=structured_headers,
+                             data=structured_data.getvalue())
+    response.raise_for_status()
+
+event = (
+    v1.Event()
+    .SetContentType("application/json")
+    .SetData('{"name":"john"}')
+    .SetEventID("my-id")
+    .SetSource("from-galaxy-far-far-away")
+    .SetEventTime("tomorrow")
+    .SetEventType("cloudevent.greet.you")
+)
+
+run_structured(event, "http://0.0.0.0:8080/")
+
+```

examples/cloud_run_cloudevents/main.py

@@ -0,0 +1,21 @@
+# Copyright 2020 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# This sample creates a function that accepts a Cloud Event per
+# https://github.com/cloudevents/sdk-python
+import sys
+
+
+def hello(cloudevent):
+    print("Received event with ID: %s" % cloudevent.EventID(), file=sys.stdout, flush=True)

examples/cloud_run_cloudevents/requirements.txt

@@ -0,0 +1 @@
+# Optionally include additional dependencies here

examples/cloud_run_event/Dockerfile

@@ -12,4 +12,4 @@ RUN pip install gunicorn functions-framework
 RUN pip install -r requirements.txt
 
 # Run the web service on container startup.
-CMD exec functions-framework --target=hello --signature-type=event
+CMD exec functions-framework --target=hello --signature_type=event

examples/cloud_run_event/README.md

@@ -0,0 +1,3 @@
+# Google Cloud Functions Events Example
+This example demonstrates how to write an event function. Note that you can also use [CloudEvents](https://github.com/cloudevents/sdk-python)
+([example](../cloud_run_cloudevents)), which is a different construct. 

setup.py

@@ -52,6 +52,7 @@
         "click>=7.0,<8.0",
         "watchdog>=0.10.0",
         "gunicorn>=19.2.0,<21.0; platform_system!='Windows'",
+        "cloudevents<1.0",
     ],
     entry_points={
         "console_scripts": [

src/functions_framework/__init__.py

@@ -12,13 +12,19 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-import functools
+import enum
 import importlib.util
+import io
+import json
 import os.path
 import pathlib
 import sys
 import types
 
+import cloudevents.sdk
+import cloudevents.sdk.event
+import cloudevents.sdk.event.v1
+import cloudevents.sdk.marshaller
 import flask
 import werkzeug
 
@@ -35,6 +41,12 @@
 DEFAULT_SIGNATURE_TYPE = "http"
 
 
+class _EventType(enum.Enum):
+    LEGACY = 1
+    CLOUDEVENT_BINARY = 2
+    CLOUDEVENT_STRUCTURED = 3
+
+
 class _Event(object):
     """Event passed to background functions."""
 
@@ -67,38 +79,83 @@ def view_func(path):
     return view_func
 
 
-def _is_binary_cloud_event(request):
-    return (
+def _get_cloudevent_version():
+    return cloudevents.sdk.event.v1.Event()
+
+
+def _run_legacy_event(function, request):
+    event_data = request.get_json()
+    if not event_data:
+        flask.abort(400)
+    event_object = _Event(**event_data)
+    data = event_object.data
+    context = Context(**event_object.context)
+    function(data, context)
+
+
+def _run_binary_cloudevent(function, request, cloudevent_def):
+    data = io.BytesIO(request.get_data())
+    http_marshaller = cloudevents.sdk.marshaller.NewDefaultHTTPMarshaller()
+    event = http_marshaller.FromRequest(
+        cloudevent_def, request.headers, data, json.load
+    )
+
+    function(event)
+
+
+def _run_structured_cloudevent(function, request, cloudevent_def):
+    data = io.StringIO(request.get_data(as_text=True))
+    m = cloudevents.sdk.marshaller.NewDefaultHTTPMarshaller()
+    event = m.FromRequest(cloudevent_def, request.headers, data, json.loads)
+    function(event)
+
+
+def _get_event_type(request):
+    if (
         request.headers.get("ce-type")
         and request.headers.get("ce-specversion")
         and request.headers.get("ce-source")
         and request.headers.get("ce-id")
-    )
+    ):
+        return _EventType.CLOUDEVENT_BINARY
+    elif request.headers.get("Content-Type") == "application/cloudevents+json":
+        return _EventType.CLOUDEVENT_STRUCTURED
+    else:
+        return _EventType.LEGACY
 
 
 def _event_view_func_wrapper(function, request):
     def view_func(path):
-        if _is_binary_cloud_event(request):
-            # Support CloudEvents in binary content mode, with data being the
-            # whole request body and context attributes retrieved from request
-            # headers.
-            data = request.get_data()
-            context = Context(
-                eventId=request.headers.get("ce-eventId"),
-                timestamp=request.headers.get("ce-timestamp"),
-                eventType=request.headers.get("ce-eventType"),
-                resource=request.headers.get("ce-resource"),
+        if _get_event_type(request) == _EventType.LEGACY:
+            _run_legacy_event(function, request)
+        else:
+            # here for defensive backwards compatibility in case we make a mistake in rollout.
+            flask.abort(
+                400,
+                description="The FUNCTION_SIGNATURE_TYPE for this function is set to event "
+                "but no Google Cloud Functions Event was given. If you are using CloudEvents set "
+                "FUNCTION_SIGNATURE_TYPE=cloudevent",
             )
-            function(data, context)
+
+        return "OK"
+
+    return view_func
+
+
+def _cloudevent_view_func_wrapper(function, request):
+    def view_func(path):
+        cloudevent_def = _get_cloudevent_version()
+        event_type = _get_event_type(request)
+        if event_type == _EventType.CLOUDEVENT_STRUCTURED:
+            _run_structured_cloudevent(function, request, cloudevent_def)
+        elif event_type == _EventType.CLOUDEVENT_BINARY:
+            _run_binary_cloudevent(function, request, cloudevent_def)
         else:
-            # This is a regular CloudEvent
-            event_data = request.get_json()
-            if not event_data:
-                flask.abort(400)
-            event_object = _Event(**event_data)
-            data = event_object.data
-            context = Context(**event_object.context)
-            function(data, context)
+            flask.abort(
+                400,
+                description="Function was defined with FUNCTION_SIGNATURE_TYPE=cloudevent "
+                " but it did not receive a cloudevent as a request.",
+            )
 
         return "OK"
 
@@ -193,19 +250,27 @@ def create_app(target=None, source=None, signature_type=None):
         app.url_map.add(werkzeug.routing.Rule("/<path:path>", endpoint="run"))
         app.view_functions["run"] = _http_view_func_wrapper(function, flask.request)
         app.view_functions["error"] = lambda: flask.abort(404, description="Not Found")
-    elif signature_type == "event":
+    elif signature_type == "event" or signature_type == "cloudevent":
         app.url_map.add(
             werkzeug.routing.Rule(
-                "/", defaults={"path": ""}, endpoint="run", methods=["POST"]
+                "/", defaults={"path": ""}, endpoint=signature_type, methods=["POST"]
             )
         )
         app.url_map.add(
-            werkzeug.routing.Rule("/<path:path>", endpoint="run", methods=["POST"])
+            werkzeug.routing.Rule(
+                "/<path:path>", endpoint=signature_type, methods=["POST"]
+            )
         )
-        app.view_functions["run"] = _event_view_func_wrapper(function, flask.request)
+
         # Add a dummy endpoint for GET /
         app.url_map.add(werkzeug.routing.Rule("/", endpoint="get", methods=["GET"]))
         app.view_functions["get"] = lambda: ""
+
+        # Add the view functions
+        app.view_functions["event"] = _event_view_func_wrapper(function, flask.request)
+        app.view_functions["cloudevent"] = _cloudevent_view_func_wrapper(
+            function, flask.request
+        )
     else:
         raise FunctionsFrameworkException(
             "Invalid signature type: {signature_type}".format(