Manage Webhook Events in a MessageQ using AWS SQS

A message queue is a middleware solution that allows systems and applications to communicate with each other by storing data temporarily in a node before it's available to use by the other apps. A common example of a message queue is E-mail where a sender sends a message, it gets stored in an intermediate node until the system or the app is ready to advance it. At the receiving end, the message is stored in an email box until the receiver is ready to read it.

In this guide, we will learn how to implement a message queuing system using Contentstack Webhooks, AWS SQS (a message queue service from Amazon), and AWS Lambda to automate a series of actions (e.g., notifications, file sharing, etc).

Overview of the process: When an event occurs in Contentstack, a webhook triggers the Lambda function and puts the data into the AWS SQS queue. This data is then processed to perform some automated tasks and the data (message) dequeues automatically after processing.

Prerequisites

To proceed with the procedure, you will need the following:

Basic knowledge of AWS SQS
Working knowledge of AWS Lambda
Access to the AWS environment
Contentstack account

Set up Webhooks with AWS SQS

Follow the steps given below to set up webhooks with AWS SQS:

Create a message queue using SQS
Create a “sender” AWS Lambda function
Set up an API Gateway
Configure a webhook in Contentstack
Create a “receiver” AWS Lambda function
Set up SQS trigger

Create a message queue using SQS
Create a queue in AWS to store the messages (incoming data from webhooks).
1. Login to your AWS Console, and select the Simple Message Queue service from the Services list.
2. In the Create New Queue page, provide a suitable name for the queue under Queue Name, and select queue type as Standard Queue.
3. Click on the Quick-Create Queue button. This creates a new queue in the SQS console.
Note: Make sure to copy and store the queue URL displayed in the Details section, as we will be using it in the following steps.

Create a “sender” AWS Lambda function

Create a lambda function that sends a message (received from Contentstack Webhooks) to the SQS queue.

Navigate to the Lambda service in the “Services” list.
Click on the Create Function button to define a lambda function.
Select the Author from Scratch option, enter the function name (for example, Sender), and choose Runtime as Node.js 8.10.
Under the Permission section, select Create a new role from AWS policy templates, name the role, and choose Amazon SQS Poller permissions as the Policy Templates.
Note: Make sure to add AmazonSQSFullAccess and AWSLambdaSQSQueueExecutionRole policies by selecting the IAM service from the Services list and then selecting Roles > Attach Policies. In the window that pops up, search for the above two policies and add them.
Click on the Create function button.

AWS Lambda provides an inline code editor where you can write your JavaScript code. Replace the existing code with the following code in the index.js file.

'use strict';
var AWS = require('aws-sdk');
var sqs = new AWS.SQS({
    region: 'us-east-1'
});

exports.handler = function(event, context, callback) {

//var accountId = context.invokedFunctionArn.split(":")[4];
    var queueUrl = '//paste your queue url here';

    // response and status of HTTP endpoint
    var responseBody = {
        message: '//text'
    };
    var responseCode = 200;

    // SQS message parameters
    var params = {
        MessageBody: event.body,
        QueueUrl: queueUrl
    };

sqs.sendMessage(params, function(err, data) {
        if (err) {
            console.log('error:', "failed to send message" + err);
            var responseCode = 500;
        } else {
            console.log('data:', data);
            responseBody.message = 'Sent to ' + queueUrl;
            responseBody.messageId = data.MessageId;
        }
        var response = {
            statusCode: responseCode,
            headers: {
                'Content-Type': 'application/json'
            },
            body: JSON.stringify(responseBody)
        };

        callback(null, response);
    });
}

Place the URL of the queue, generated in Step 1, in the "queueURL" variable and Save the function.

Set up API Gateway
For your Lambda to be accessible publicly, you need to configure the AWS API Gateway using your function as the backend for the API as shown below:
1. Navigate to the Lambda service, and click on your Lambda function that you just created in the above step.
2. Click on + Add trigger.
3. In the Trigger configuration drop-down, select API Gateway
4. In the API drop-down, select Create a new API, and choose REST API template
5. Under the Security drop-down, select Open
6. Click Add to save your settings
After performing the above steps, you'll get an API URL that you can use while configuring webhooks in the next step.
Set up Webhooks in Contentstack
To create and set up a webhook log in to your Contentstack account and perform the following steps:
1. Hover over the “Settings” gear icon and click on Webhooks.
2. Click on + New Webhook
3. On the Create Webhook page, fill up the Name field (for example SQS). In the URL to notify field, enter the URL that you generated when you deployed your APIs, in the previous step.
4. Scroll down to the When section for creating a trigger for the webhook as shown below:
5. Once done, click on the Save button to save your settings.
With this, we have set up the webhooks for triggering messages.
Create a “receiver” AWS Lambda function
Create another AWS lambda function to poll the messages from the queue after the messages are added in the queue.
1. Login to your AWS Management Console and navigate to the Lambda service.
2. Create a new Lambda function by clicking the Create function button.
3. Choose the Author from Scratch option, name your function, and select Node.js 8.10 as runtime.
4. In the Permissions section, select Use an existing role, and choose the role created in Step 2.
5. Click the Create function button.
6. Navigate to the inline code editor, open the index.js file, and replace the existing code with the following code:
```
'use strict';

exports.handler = (event, context, callback) => {
  const response = {
    statusCode: 200,
    body: JSON.stringify({
      message: 'Received message successfully!',
      input: event,
    }),
  };

  var body = event.Records[0].body;
  console.log("Received message: ",body,event.Records[0]);

  callback(null, response);
};
```
7. Save the function.
Set up SQS Trigger
Create an SQS trigger that will activate when a message is enqueued. This trigger will process the message and delete it from the queue after its successful processing. To do this, follow the steps given below:
1. Go to the AWS Lambda function created in step 5 and click on the + Add trigger button.
2. In the Trigger configuration dropdown, select SQS.
3. Enter the queue name, select the Enable trigger checkbox, and click on the Add button.
Note: This will only log the data in the AWS Cloudwatch. You can write your business logic to process the received messages.
After completion of this setup, go to Contentstack, publish any entry (from which Webhook will trigger), and then check the logs in AWS Cloudwatch.