Updated instructions for deployment

[vikhyat187]

List of features built

1. JWT Validation Using RDS Public Key

• All incoming requests are authenticated using JWTs (JSON Web Tokens).
• The JWTs are validated against RDS public key

2. Cross-origin requests are allowed only from trusted domains using this regex:

• ^https?://([a-zA-Z0-9-]+\.)*realdevsquad\.com$

3. Lambda Throttling Based on Request Count

• Lambda functions are automatically throttled when the total number of requests (both successful and failed) exceeds the allowed threshold.
• This prevents abuse and protects downstream systems from overload.

4. Reset limit lambda - We can trigger this manually

• Increase the request limit dynamically.
• Raise Lambda concurrency from zero to allow traffic again when needed.

5. Rate limit lambda - This gets trigger automatically when the pendingLimit in RequestLimit table gets less than 1

• Setting concurrency of all Lambda functions to zero, effectively disabling them.
• This is useful for enforcing throttling or stopping processing temporarily.

6. CORS lambda

• This lambda gets called for all the OPTIONS requests made to the feature flag service
• This will evaluate the origin header and return the dynamic header in the response headers.

7. CloudWatch Logging for All Lambda Functions

• Every Lambda function is configured to push logs to AWS CloudWatch.
• This provides visibility into function executions, errors, and performance.
• 

8. API Gateway Logging Setup

• Logs for API Gateway are not enabled by default and must be manually configured:
• Go to API Gateway → Select Stage → Logs and Tracing
• Enable Error logs by default
• Optionally enable INFO logs for debugging when needed.

9. Authentication for the lambdas

• All Lambda functions require JWT-based authentication, ensuring secure access.

10. Rate limiting logic

• On each API request (OPTIONS / other HTTP methods) we reduce the value of pending limit and once this value becomes less than 1 we make all the lambdas throttle (By setting their concurrency to 0)
• On this the lambdas return 429 as the response code

[vikhyat187]

Deployment steps

• Create DDB tables

Details

 ## Tables in DynamoDB
###  featureFlag
  Partition Key - Id (string)
  
###  featureFlagUserMapping
  Partition Key - userId (string)
  Sort Key - flagId (string)
  
###  requestLimit
  Partition Key - limitType (String)
  
  No sort key required here

• Create a pipeline using sam pipeline --bootstrap.

• Add Secret keys in Github Secrets.

  Details

  These are defined directly in the workflow or derived dynamically:
  
  
  🔒 Required Secrets (secrets)
  🧑‍💻 Authentication & Authorization
  These are used for assuming IAM roles and configuring AWS credentials:
  
  AWS_ACCESS_KEY_ID:  (Pipeline user access key) -> generated by `sam init pipeline` command
  The access key ID for the IAM user or role used in the pipeline.
  
  AWS_SECRET_ACCESS_KEY:  (Pipeline user secret key)
  The corresponding secret access key.
  
  PIPELINE_EXECUTION_ROLE: (Take from AWS -> Cloudformation select the feature flag backend -> Resources tab you find it there)
  The ARN of the IAM role to assume for packaging and deployment tasks.
  
  CLOUDFORMATION_EXECUTION_ROLE: (same as above in Cloudformation -> resources)
  The IAM role used by CloudFormation to execute the change set during deployment.
  
  📦 Artifact & Stack Configuration
  ARTIFACTS_BUCKET: (same as above in Cloudformation -> resources)
  The S3 bucket where the packaged artifacts (like Lambda code) are stored.
  
  REGION: 
  The AWS region where you’re deploying (e.g., us-east-1).
  
  STACK_NAME: 
  The name of the CloudFormation stack to be created or updated.
  

• Add Permissions listed below for the IAM Role add them manually in (AWS -> IAM Roles -> new role), this is being used by AWS lambdas to access DBs, logs, call other lambdas.

• Store RDS public key in AWS Systems manager - Parameter store (Add: RDS PROD backend Public key in AWS parameter store #143)

• Change the Physical ID of lambda's in template.yaml to the lambda that got created in your Cloudformation stack.

  • These are the ARNs of the lambda functions, these gets used in Rate limit lambda / Reset limit lambda function
  Details

  Replace the second value with your Lambda ARN (Select a lambda and you will find the ARN -> Amazon Resource number)
  

• Enabling Logging in cloudwatch (This has to be done explicitly)

  • - Go to API Gateway → Select Stage → Logs and Tracing

Updated policy as we have also added the auth and reading the public key from the Systems manager parameter store

IAM Role Policy for the CORS lambda (AWS -> IAM Roles -> new role)

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "dynamodb:PutItem",
                "lambda:InvokeFunction",
                "dynamodb:GetItem",
                "dynamodb:UpdateItem"
            ],
            "Resource": [
                "arn:aws:dynamodb:us-east-1:502287753636:table/requestLimit",
                "arn:aws:lambda:us-east-1:502287753636:function:feature-flag-RateLimiterFunction-PQZQRZzSKeD0"
            ]
        },
        {
            "Sid": "VisualEditor1",
            "Effect": "Allow",
            "Action": [
                "logs:CreateLogStream",
                "logs:PutLogEvents",
                "ssm:GetParameter"
            ],
            "Resource": "*"
        }
    ]
}

IAM Role Policy for the rate limit and reset limit lambda (AWS -> IAM Roles -> new role)

{
	"Version": "2012-10-17",
	"Statement": [
		{
			"Sid": "VisualEditor0",
			"Effect": "Allow",
			"Action": [
				"dynamodb:PutItem",
				"dynamodb:GetItem",
                                 "dynamodb:UpdateItem",
				"lambda:DeleteFunctionConcurrency",
				"ssm:GetParameter",
				"lambda:PutFunctionConcurrency"
			],
			"Resource": [
				"arn:aws:ssm:us-east-1:502287753636:parameter/*",
				"arn:aws:dynamodb:us-east-1:502287753636:table/requestLimit",
				"arn:aws:lambda:us-east-1:502287753636:function:*"
			]
		}
	]
}

Policy for the all other lambdas (AWS -> IAM Roles -> new role)

NOTE: We will have to change the AWS account Id in these permissions with the actual AWS account Id in staging / production.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "dynamodb:BatchGetItem",
                "logs:CreateLogStream",
                "dynamodb:BatchWriteItem",
                "dynamodb:PutItem",
                "lambda:InvokeFunction",
                "dynamodb:GetItem",
                "dynamodb:Scan",
                "dynamodb:Query",
                "dynamodb:UpdateItem",
                "ssm:GetParameter",
                "logs:PutLogEvents"
            ],
            "Resource": [
                "arn:aws:ssm:us-east-1:502287753636:parameter/*",
                "arn:aws:logs:us-east-1:502287753636:log-group:*:log-stream:*",
                "arn:aws:dynamodb:us-east-1:502287753636:table/*",
                "arn:aws:lambda:us-east-1:502287753636:function:feature-flag-RateLimiterFunction-PQZQRZzSKeD0"
            ]
        },
        {
            "Sid": "VisualEditor1",
            "Effect": "Allow",
            "Action": "logs:CreateLogGroup",
            "Resource": [
                "arn:aws:logs:us-east-1:181388287005:*",
                "arn:aws:logs:us-east-1:502287753636:log-group:*:log-stream:*"
            ]
        }
    ]
}

Steps to verify thing working

• We can create a feature flag, see if the entry got created in table

Details

curl --location 'localhost:3000/feature-flags/' \ --header 'Origin: https://dashboard.realdevsquad.com' \ --header 'Content-Type: application/json' \ --data '{ "name": "Test-flag-via-postman", "description": "Feature flag created via postman", "userId":"vikhyat" }'

• We can add a user to the feature flag, try fetching this entry

Details

curl --location 'localhost:3000/users/vikhyat/feature-flags/22d400ca-8736-4c11-8a48-c74ff7d66f09' \ --header 'Origin: https://dashboard.realdevsquad.com' \ --header 'Content-Type: application/json' \ --data '{ "status": "DISABLED", "userId": "vikhyat187" }'

• We can set the pending Limit value to 1 and on next time hitting the API verify if the lambda functions concurrency is set to 0 and they are throttled.

Details

This can be done by making the value in the `RequestLimit` table `pendingLimit` value to 1 and in next call the lambdas will get throttled + their `reserved concurrency` is being set to 0.

This will only work -> All permissions are listed above in IAM roles

• If lambdas have permissions to invoke other lambda
• Permission to update value in ddb

• We can call the resetLimit and verify if the reservedConcurrency is removed and lambdas are using the 20 unreserved concurrency.

Details

curl --location 'localhost:3000/Prod/reset-limit' \ --header 'Origin: https://staging-api.realdevsquad.com' \ --header 'Cookie: rds-session-staging=eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJ1c2VySWQiOiJTaEpvMzUyejFadUVORzFzb1RjRSIsImlhdCI6MTczNzYyNDIwMCwiZXhwIjoxNzY4NzI4MjAwfQ.DxzdfDdPz0ya7PeEBLMVbQMwU54h4N1DkloKFuOfX-OKA4R1l4FGvk3g4FRbnjL2I2yc9HNCEP7NsT5sccjflcg4mvFw6hK3ZVcXhn1AUMT95Pu_ocj-xtkXK8pZeEr9RrQ7viLOt00gFNNgl_ZxtqmA_GfamiUhdQ17whEQzGc' \ --header 'Content-Type: application/json' \ --data '{ "pendingLimit": 10 }'

• We can verify the logs coming in cloudwatch for both API gateway and lambda functions.

[vikhyat187]

Permissions to be granted on staging

1. Permission to write to DDB in createFeatureFlagFunction

Details

AccessDeniedException: User: arn:aws:sts::181388287005:assumed-role/feature-flag-staging-CreateFeatureFlagFunctionRole-Ktm4nSrCXhOl/feature-flag-staging-CreateFeatureFlagFunction-jtNqeZdpSPyX is not authorized to perform: dynamodb:PutItem on resource: arn:aws:dynamodb:us-east-1:181388287005:table/featureFlag because no identity-based policy allows the dynamodb:PutItem action |   -- | --

[vikhyat187]

Staging deployment proof

1. Get all feature flags function

Details

2. Get feature flags by Id

Details

3. Update feature flag
   < details>

4. Update user feature flag

5. CORS headers

Details

6. Get all user feature flags

7. Get specific feature flag for user

8. Update rate limit of lambdas

9. Reset limit