Skip to content

Commit 68a6a62

Browse files
PigiusPigius
authored
feat: Add aws-ruby-step-functions-with-callback example (#640)
1 parent cccc2ba commit 68a6a62

19 files changed

+10207
-1
lines changed

‎README.md

+1-1
Original file line numberDiff line numberDiff line change
@@ -125,8 +125,8 @@ serverless install -u https://github.com/serverless/examples/tree/master/folder-
125125
| [Aws Ruby Line Bot](https://github.com/serverless/examples/tree/master/aws-ruby-line-bot) <br/> Example demonstrates how to setup a simple Line echo bot on AWS | ruby |
126126
| [Aws Ruby Simple Http Endpoint](https://github.com/serverless/examples/tree/master/aws-ruby-simple-http-endpoint) <br/> Example demonstrates how to setup a simple HTTP GET endpoint | ruby |
127127
| [Aws Ruby Sinatra Dynamodb Api](https://github.com/serverless/examples/tree/master/aws-ruby-sinatra-dynamodb-api) <br/> Example of a Ruby Sinatra API service backed by DynamoDB with traditional Serverless Framework | ruby |
128+
| [Aws Ruby Step Functions Callback](https://github.com/serverless/examples/tree/master/aws-ruby-step-functions-with-callback) <br/> Ruby example that make usage of AWS Step Functions with callback pattern, AWS Lambda, DynamoDB, Amazon Comprehend, API Gateway, and Step Functions flows | ruby |
128129
| [Serverless Ruby Sqs Dynamodb](https://github.com/serverless/examples/tree/master/aws-ruby-sqs-with-dynamodb) <br/> A serverless ruby example that creates DynamoDB records with the usage of SQS, API Gateway, and AWS Lambda functions. | ruby |
129-
| [Serverless Ruby Sqs Dynamodb](https://github.com/serverless/examples/tree/master/aws-ruby-sqs-with-dynamodb/src) <br/> A serverless ruby example that creates DynamoDB records with usage of SQS, API Gateway, and AWS Lambda functions | ruby |
130130
| [Aws Ruby Step Functions](https://github.com/serverless/examples/tree/master/aws-ruby-step-functions) <br/> Ruby example that make usage of AWS Step Functions with AWS Lambda, DynamoDB and Step Functions flows. | ruby |
131131
| [Aws Rust Simple Http Endpoint](https://github.com/serverless/examples/tree/master/aws-rust-simple-http-endpoint) <br/> Example demonstrates how to setup a simple HTTP GET endpoint with rust | nodeJS |
132132
| [Azure Nodejs](https://github.com/serverless/examples/tree/master/azure-node-line-bot) <br/> Azure Functions sample for the Serverless framework | nodeJS |

‎aws-ruby-step-functions-with-callback/Gemfile

+5
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
source 'https://rubygems.org'
2+
3+
gem 'aws-sdk-comprehend'
4+
gem 'aws-sdk-dynamodb'
5+
gem 'aws-sdk-states'

‎aws-ruby-step-functions-with-callback/README.md

+231
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,231 @@
1+
<!--
2+
title: 'Ruby AWS Step Functions with Callback pattern'
3+
description: 'Ruby example that make usage of AWS Step Functions with callback pattern, AWS Lambda, DynamoDB, Amazon Comprehend, API Gateway, and Step Functions flows.'
4+
layout: Doc
5+
framework: v2
6+
platform: AWS
7+
language: Ruby
8+
authorLink: 'https://github.com/pigius'
9+
authorName: 'Daniel Aniszkiewicz'
10+
authorAvatar: 'https://avatars1.githubusercontent.com/u/8863200?s=200&v=4'
11+
-->
12+
# Ruby AWS Step Functions with Callback pattern
13+
14+
This is an example of using `AWS Step Functions` with [callback pattern](https://docs.aws.amazon.com/step-functions/latest/dg/callback-task-sample-sqs.html). It uses `AWS Lambda`, `DynamoDB`, `API Gateway`, `Amazon Comprehend`, `flows` from `Step Functions`.
15+
16+
17+
## Diagram
18+
19+
![draw](./images/step-functions-with-callback.png)
20+
21+
![diagram](./images/stepfunctions-with-callback-graph.png)
22+
23+
24+
This Workflow is quite simple. Assuming we have a comment system in our application, we would like to check for PII data before adding the comment to the database.
25+
26+
To do this, we use Amazon Comprehend which, based on trained ML models, checks if a given unstructured text contains sensitive data and returns information on which positions in the sentence it is located.
27+
28+
When detecting sensitive data with Amazon Comprehend, we use the `taskToken` from AWS `step functions` to wait for the detection results and continue our workflow.
29+
30+
Keep in mind, that `taskToken` is generated by Step Functions automatically.
31+
32+
Then, depending on the detection results, we perform a redaction process, replacing the sensitive data with an asterisk (*)
33+
34+
At the very end, the redacted comment is saved into the database (DynamoDB)
35+
36+
37+
![detailed-diagram](./images/detailed-diagram-callback-sf.png)
38+
39+
## Setup
40+
41+
`npm install` to install all needed packages.
42+
43+
## Deployment
44+
45+
In order to deploy the service run:
46+
47+
```bash
48+
sls deploy
49+
```
50+
51+
for deploying with a specific `profile` (located in `~/.aws/credentials`) you can simply use the command:
52+
53+
```bash
54+
AWS_PROFILE=YOUR_PROFILE_NAME sls deploy
55+
```
56+
57+
for deploying to the specific stage, let's say `staging` do:
58+
59+
```bash
60+
sls deploy --stage staging
61+
```
62+
63+
The expected result should be similar to:
64+
65+
```bash
66+
Serverless: Running "serverless" installed locally (in service node_modules)
67+
Serverless: Packaging service...
68+
Serverless: Excluding development dependencies...
69+
Serverless: Clearing previous build ruby layer build
70+
[ '2.2' ]
71+
Serverless: Installing gem using local bundler
72+
Serverless: Zipping the gemfiles to /.serverless/ruby_layer/gemLayer.zip
73+
Serverless: Configuring Layer and GEM_PATH to the functions
74+
✓ State machine "WorkflowWithCallback" definition is valid
75+
Serverless: Uploading CloudFormation file to S3...
76+
Serverless: Uploading artifacts...
77+
Serverless: Uploading service aws-ruby-step-functions-with-callback.zip file to S3 (137.4 KB)...
78+
Serverless: Uploading service gemLayer.zip file to S3 (749.01 KB)...
79+
Serverless: Validating template...
80+
Serverless: Updating Stack...
81+
Serverless: Checking Stack update progress...
82+
.........................
83+
Serverless: Stack update finished...
84+
Service Information
85+
service: aws-ruby-step-functions-with-callback
86+
stage: dev
87+
region: us-east-1
88+
stack: aws-ruby-step-functions-with-callback-dev
89+
resources: 19
90+
api keys:
91+
None
92+
endpoints:
93+
functions:
94+
check-comment: aws-ruby-step-functions-with-callback-dev-check-comment
95+
redact-comment: aws-ruby-step-functions-with-callback-dev-redact-comment
96+
layers:
97+
gem: arn:aws:lambda:your-region:XXXXXXXXXXX:layer:aws-ruby-step-functions-with-callback-dev-ruby-bundle:49
98+
Serverless StepFunctions OutPuts
99+
endpoints:
100+
POST - https://XXXXXXXXXXXX.execute-api.your-region.amazonaws.com/dev/comments/add
101+
```
102+
103+
## Usage
104+
105+
There are two possible ways of invoking the example:
106+
107+
## Via Api Gateway
108+
109+
After the deployment, grab the POST endpoint for this service. You can make a API call either by cURL or some tools like Postman.
110+
111+
Use payload like:
112+
113+
```json
114+
{
115+
"comment": "My pin number is 1234",
116+
"author": "Daniel"
117+
}
118+
```
119+
120+
or something with more PII data included:
121+
122+
```json
123+
{
124+
"comment": "Good morning, everybody. My name is Van Bokhorst Serdar, and today I feel like sharing a whole lot of personal information with you. Let's start with my Email address [email protected]. My address is 2657 Koontz Lane, Los Angeles, CA. My phone number is 818-828-6231. My Social security number is 548-95-6370. My Bank account number is 940517528812 and routing number 195991012. My credit card number is 5534816011668430, Expiration Date 6/1/2022, my C V V code is 121, and my pin 123456. Well, I think that's it. You know a whole lot about me. And I hope that Amazon comprehend is doing a good job at identifying PII entities so you can redact my personal information away from this document. Let's check.",
125+
"author": "Daniel"
126+
}
127+
```
128+
As a response you will get:
129+
130+
```json
131+
{
132+
"executionArn": "arn:aws:states:your-region:XXXXXXXXXXXX:execution:workflow-with-callback-dev:79fa0214-8533-4506-b47f-513ca8721fe0",
133+
"startDate": 1.628965530156E9
134+
}
135+
```
136+
137+
![dynamo](./images/postman.png)
138+
139+
140+
After it, you can then check the Dynamo database to see if a record has been created.
141+
142+
![dynamo](./images/dynamodb-results.png)
143+
144+
145+
## Via AWS Dashboard
146+
147+
After the deployment, go to the AWS Dashboard, and enter Step Functions page. You will see a newly created state machine.
148+
149+
Open the state machine and click on `Start Execution`. You need to provide the input in the JSON schema.
150+
151+
Example:
152+
153+
```json
154+
{
155+
"comment": "My pin number is 1234",
156+
"author": "Daniel"
157+
}
158+
```
159+
160+
or something with more PII data included:
161+
162+
```json
163+
{
164+
"comment": "Good morning, everybody. My name is Van Bokhorst Serdar, and today I feel like sharing a whole lot of personal information with you. Let's start with my Email address [email protected]. My address is 2657 Koontz Lane, Los Angeles, CA. My phone number is 818-828-6231. My Social security number is 548-95-6370. My Bank account number is 940517528812 and routing number 195991012. My credit card number is 5534816011668430, Expiration Date 6/1/2022, my C V V code is 121, and my pin 123456. Well, I think that's it. You know a whole lot about me. And I hope that Amazon comprehend is doing a good job at identifying PII entities so you can redact my personal information away from this document. Let's check.",
165+
"author": "Daniel"
166+
}
167+
```
168+
169+
Later on, simply start the excecution.
170+
171+
After it, you can then check the Dynamo database to see if a record has been created.
172+
173+
As we use `Standard` workflow type, feel free to check the executions:
174+
175+
![executions](./images/output.png)
176+
177+
178+
If there are no detections, the redaction process will not be triggered, so flow will look like this:
179+
180+
![without-redaction](./images/data-without-pii.png)
181+
182+
183+
## Log retention
184+
185+
The log retention is setup for 30 days. To change it simply change the value of this attribute in `serverless.yml` file:
186+
187+
188+
``` bash
189+
logRetentionInDays: 30
190+
```
191+
192+
## Advanced configuration
193+
More options (like alerting in case of failed excecutions), could be found in the plugin [repository](https://github.com/serverless-operations/serverless-step-functions).
194+
195+
## Structure
196+
197+
| Path | Explanation |
198+
|-----------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------|
199+
| `./src` | All code for the project. |
200+
| `./src/handlers/send_token` | Handler for lambda. |
201+
| `./src/common/` | Space for common, reusable pieces of code. |
202+
| `./src/common/adapters/step_functions_adapter.rb` | Adapter for the AWS Step Functions with the usage of AWS SDK for Ruby. Only used for sending success and failure task_token. |
203+
| `./src/common/services/send_task_token_service.rb` | The service object pattern is widely used within ruby/rails developers. A class that is responsible for doing only one thing. In our case is handling task token. |
204+
| `./src/common/services/detection_service.rb` | Responsible for making detection with Amazon Comprehend to detect PII data. |
205+
| `./src/common/services/redaction_service.rb` | Responsible for redacting comment based on the Amazon Comprehend detection results. |
206+
207+
## Serverless plugins
208+
209+
For this example, there are two serverless plugins used:
210+
211+
| Plugin | Explanation |
212+
|-----------------------|------------------------------------------------------------------------------------------------|
213+
| [serverless-ruby-layer](https://www.npmjs.com/package/serverless-ruby-layer) | For bundling ruby gems from `Gemfile` and deploys them to the lambda layer. |
214+
| [serverless-step-functions](https://www.npmjs.com/package/serverless-step-functions) | Serverless Framework plugin for AWS Step Functions. |
215+
216+
## Ruby gems
217+
218+
| Gem | Explanation |
219+
|--------------------|--------------------------------------------------------------------------------------------------------------------------------|
220+
| `aws-sdk-comprehend` | It's a part of the AWS SDK for Ruby. Used for Amazon Comprehend, in the case of this example - the detection of the PII data. |
221+
| `aws-sdk-dynamodb` | It's a part of the AWS SDK for Ruby. Used for DynamoDB, in the case of this example - the creation of the new record. |
222+
| `aws-sdk-states` | It's a part of the AWS SDK for Ruby. Used for AWS Step Functions. |
223+
224+
## Remove service
225+
226+
To remove the service do:
227+
228+
```bash
229+
sls remove
230+
```
231+
And the stack will be removed from the AWS.

‎aws-ruby-step-functions-with-callback/images/data-without-pii.png

11.8 KB
Loading

‎aws-ruby-step-functions-with-callback/images/detailed-diagram-callback-sf.png

37.9 KB
Loading

‎aws-ruby-step-functions-with-callback/images/dynamodb-results.png

28.5 KB
Loading

‎aws-ruby-step-functions-with-callback/images/output.png

68 KB
Loading

‎aws-ruby-step-functions-with-callback/images/postman.png

40.8 KB
Loading

‎aws-ruby-step-functions-with-callback/images/step-functions-with-callback.png

34.5 KB
Loading

‎aws-ruby-step-functions-with-callback/images/stepfunctions-with-callback-graph.png

6.26 KB
Loading

0 commit comments

Comments
 (0)