Manage Webhook Events in a MessageQ using AWS SQS

A message queue is a middleware solution that allows systems and applications to communicate with each other by storing data temporarily in a node before it's available to use by the other apps. A typical example of a message queue is Email, where a sender sends a message; it gets stored in an intermediate node until the system or the app is ready to advance it. At the receiving end, the message is stored in an email box until the receiver is prepared to read it.

In this guide, you'll learn how to implement a message queuing system using Contentstack Webhooks, AWS SQS (a message queue service from Amazon), and AWS Lambda to automate a series of actions (e.g., notifications, file sharing, etc.).

Overview of the process: When you publish/unpublish any entry or asset in Contentstack, a webhook will trigger. This webhook will invoke the AWS Lambda function which will add the data to the SQS queue. You can then use this data as per your requirement to perform automated tasks, after which the data dequeues automatically.

Steps for Execution

Here are the steps to automate the creation of an entry in Contentstack:

Create a message queue using AWS SQS
Create a “sender” AWS Lambda function
Attach Policies to Role
Set up API Gateway
Set up Webhooks in Contentstack
Create a “receiver” AWS Lambda function
Try out the Setup

Prerequisites

To proceed with the procedure, you will need the following:

Basic knowledge of AWS SQS
Working knowledge of AWS Lambda
Access to the AWS environment
Contentstack account

Create a message queue using AWS SQS
Create a queue in AWS to store the messages (incoming data from webhooks) by performing the following steps:
1. Login to your AWS Console, and select the Simple Queue Service from the Services list at the header.
2. On the Amazon SQS page, click the Create queue button.
3. On the Create queue page, select Type as Standard, provide a suitable name for the queue under Name.
4. Keep other options to their default values and click the Create queue button. This creates a new queue in the SQS console.
5. On your SQS page, make sure to copy and store the queue URL displayed in the Details section, as we will be using it in the following steps.

Create a “sender” AWS Lambda function

Now let’s create a lambda function that sends a message (received from Contentstack Webhooks) to the SQS queue by performing the following steps

Navigate to the Lambda service from the Services dropdown list.
On the Functions page, click on the Create function button to create a lambda function.
Then, on the Create function page, select the Author from Scratch option, enter the function name (for example, “Sender”), and choose Runtime as Node.js.
Under the Permissions section, click the Change default execution role link, select Create a new role from AWS policy templates, name the role, and choose Amazon SQS poller permissions as the “Policy templates.”
Click on the Create function button. This will create your lambda function and take you to the lambda function’s page.

AWS Lambda provides an inline code editor where you can write your lambda code. To do so, scroll down to the Code source section, and replace the existing code with the following code in the index.js file.

'use strict';
var AWS = require('aws-sdk');
var sqs = new AWS.SQS({
    region: '<your_region_code>' //for example, us-east-1
});

exports.handler = function(event, context, callback) {

//var accountId = context.invokedFunctionArn.split(":")[4];
    var queueUrl = '//paste your queue url here';

    // response and status of HTTP endpoint
    var responseBody = {
        message: '//text'
    };
    var responseCode = 200;

    // SQS message parameters
    var params = {
        MessageBody: event.body,
        QueueUrl: queueUrl
    };

sqs.sendMessage(params, function(err, data) {
        if (err) {
            console.log('error:', "failed to send message" + err);
            var responseCode = 500;
        } else {
            console.log('data:', data);
            responseBody.message = 'Sent to ' + queueUrl;
            responseBody.messageId = data.MessageId;
        }
        var response = {
            statusCode: responseCode,
            headers: {
                'Content-Type': 'application/json'
            },
            body: JSON.stringify(responseBody)
        };

        callback(null, response);
    });
}

Place the URL of the queue, generated in Step 1, in the "queueURL" variable.

Attach Policies to Role
You need to attach certain policies to the role you created in the above step. To do so, perform the following steps:
1. Go to the IAM service and then select Roles from the left navigation panel.
2. Go to the role you’ve created and click the Attach policies button.
3. On the Attach policies page, select these policies: AmazonSQSFullAccess and AWSLambdaSQSQueueExecutionRole policies.
4. Click the Attach policy button.
Set up API Gateway
For your Lambda to be accessible publicly, you need to configure the AWS API Gateway using your function as the backend for the API as shown below:
1. On your lambda function’s page, in the Function overview section, click the + Add trigger button.
2. From the Trigger configuration dropdown menu, select the API Gateway option.
3. Then, from the API dropdown menu, select Create an API option.
4. Select API type as REST API, keep Security as Open, and click Add.
  You’ll be redirected to the Triggers section.
Set up Webhooks in Contentstack
To create and set up a webhook log in to your Contentstack account and perform the following steps:
1. Hover over the “Settings” gear icon and click on Webhooks.
2. Click on + New Webhook
3. On the Create Webhook page, fill up the Name field (for example SQS). In the URL to notify field, enter the URL that you generated when you deployed your APIs, in the previous step.
4. Scroll down to the When section for creating a trigger for the webhook as shown below:
5. Once done, click on the Save button to save your settings.
With this, we have set up the webhooks for triggering messages.
Create a “receiver” AWS Lambda function
Next, let’s create another AWS lambda function to poll the messages from the queue after the messages are added to the queue by performing the following steps:
1. In your AWS Management Console, go to the Lambda service page and click on the Create function button.
2. On the Create function page, choose the Author from Scratch option, name your function, and select Node.js as Runtime.
3. Under the Permissions section, click the Change default execution role link, select Use an existing role, and choose the role created in Step 1.
4. Click the Create function button.
5. Navigate to the inline code editor, open the index.js file, and replace the existing code with the following code:
```
'use strict';

exports.handler = (event, context, callback) => {
  const response = {
    statusCode: 200,
    body: JSON.stringify({
      message: 'Received message successfully!',
      input: event,
    }),
  };

  var body = event.Records[0].body;
  console.log("Received message: ",body,event.Records[0]);

  callback(null, response);
};
```
  Next, let’s create an SQS trigger that will activate when a message is enqueued. This trigger will process the message and delete it from the queue after its successful processing.
6. Scroll up to theFunction overview section and click the +Add trigger button.
7. From the Trigger configuration dropdown menu, select SQS.
8. Enter the queue name, select the Enable trigger checkbox, and click on the Add button.
  Note: This will only log the data in the AWS Cloudwatch. You can write your business logic to process the received messages.
Try out the Setup
After completion of this setup, go to Contentstack, publish any entry (from which Webhook will trigger), and then check the logs in AWS Cloudwatch.