Manage Webhook Events in a MessageQ System using AWS SQS

A message queue is a middleware solution that allows systems and applications to communicate with each other by storing data temporarily in a node before it's available to use by the other apps. A typical example of a message queue is Email, where a sender sends a message; it gets stored in an intermediate node until the system or the app is ready to advance it. At the receiving end, the message is stored in an email box until the receiver is prepared to read it.

In this guide, you'll learn how to implement a message queuing system using Contentstack Webhooks, AWS SQS (a message queue service from Amazon), and AWS Lambda to automate a series of actions (e.g., notifications, file sharing, etc.).

Overview of the process: When you publish/unpublish any entry or asset in Contentstack, a webhook will trigger. This webhook will invoke the AWS Lambda function which will add the data to the SQS queue. You can then use this data as per your requirement to perform automated tasks, after which the data dequeues automatically.

Steps for Execution

Here are the steps to automate the creation of an entry in Contentstack:

  1. Create a message queue using AWS SQS
  2. Create a “sender” AWS Lambda function
  3. Attach Policies to Role
  4. Set up API Gateway
  5. Set up Webhooks in Contentstack
  6. Create a “receiver” AWS Lambda function
  7. Try out the Setup

    Prerequisites

    To proceed with the procedure, you will need the following:

    • Basic knowledge of AWS SQS
    • Working knowledge of AWS Lambda
    • Access to the AWS environment
    • Contentstack account
    1. Create a message queue using AWS SQS

      Create a queue in AWS to store the messages (incoming data from webhooks) by performing the following steps:

      1. Login to your AWS Console, and select the Simple Queue Service from the Services list at the header.
      2. On the Amazon SQS page, click the Create queue button.
      3. On the Create queue page, select Type as Standard, provide a suitable name for the queue under Name.
      4. Keep other options to their default values and click the Create queue button. This creates a new queue in the SQS console.
      5. On your SQS page, make sure to copy and store the queue URL displayed in the Details section, as we will be using it in the following steps.
        copy-url.png
    2. Create a “sender” AWS Lambda function

      Now let’s create a lambda function that sends a message (received from Contentstack Webhooks) to the SQS queue by performing the following steps

      1. Navigate to the Lambda service from the Services dropdown list.
      2. On the Functions page, click on the Create function button to create a lambda function.
      3. Then, on the Create function page, select the Author from Scratch option, enter the function name (for example, “Sender”), and choose Runtime as Node.js.
      4. Under the Permissions section, click the Change default execution role link, select Create a new role from AWS policy templates, name the role, and choose Amazon SQS poller permissions as the “Policy templates.”Permissions.PNG
      5. Click on the Create function button. This will create your lambda function and take you to the lambda function’s page. 

      6. AWS Lambda provides an inline code editor where you can write your lambda code. To do so, scroll down to the Code source section, and replace the existing code with the following code in the index.js file.
        'use strict';
        var AWS = require('aws-sdk');
        var sqs = new AWS.SQS({
            region: '<your_region_code>' //for example, us-east-1
        });
        
        exports.handler = function(event, context, callback) {
        
        //var accountId = context.invokedFunctionArn.split(":")[4];
            var queueUrl = '//paste your queue url here';
        
            // response and status of HTTP endpoint
            var responseBody = {
                message: '//text'
            };
            var responseCode = 200;
        
            // SQS message parameters
            var params = {
                MessageBody: event.body,
                QueueUrl: queueUrl
            };
        
        sqs.sendMessage(params, function(err, data) {
                if (err) {
                    console.log('error:', "failed to send message" + err);
                    var responseCode = 500;
                } else {
                    console.log('data:', data);
                    responseBody.message = 'Sent to ' + queueUrl;
                    responseBody.messageId = data.MessageId;
                }
                var response = {
                    statusCode: responseCode,
                    headers: {
                        'Content-Type': 'application/json'
                    },
                    body: JSON.stringify(responseBody)
                };
        
                callback(null, response);
            });
        }
      7. Place the URL of the queue, generated in Step 1, in the "queueURL" variable.
    3. Attach Policies to Role

      You need to attach certain policies to the role you created in the above step. To do so, perform the following steps:

      1. Go to the IAM service and then select Roles from the left navigation panel.
      2. Go to the role you’ve created and click the Attach policies button.
      3. On the Attach policies page, select these policies: AmazonSQSFullAccess and AWSLambdaSQSQueueExecutionRole policies.
      4. Click the Attach policy button.
    4. Set up API Gateway

      For your Lambda to be accessible publicly, you need to configure the AWS API Gateway using your function as the backend for the API as shown below:

      1. On your lambda function’s page, in the Function overview section, click the + Add trigger button.
      2. From the Trigger configuration dropdown menu, select the API Gateway option.
      3. Then, from the API dropdown menu, select Create an API option.
      4. Select API type as REST API, keep Security as Open, and click Add.
        You’ll be redirected to the Triggers section.
        Sender-API-gateway-details.png
    5. Set up Webhooks in Contentstack

      To create and set up a webhook log in to your Contentstack account and perform the following steps:

      1. Hover over the “Settings” gear icon and click on Webhooks.
      2. Click on + New Webhook
      3. On the Create Webhook page, fill up the Name field (for example SQS). In the URL to notify field, enter the URL that you generated when you deployed your APIs, in the previous step.
      4. Scroll down to the When section for creating a trigger for the webhook as shown below: Set up a webhook.PNG
      5. Once done, click on the Save button to save your settings.

      With this, we have set up the webhooks for triggering messages.

    6. Create a “receiver” AWS Lambda function

      Next, let’s create another AWS lambda function to poll the messages from the queue after the messages are added to the queue by performing the following steps:

      1. In your AWS Management Console, go to the Lambda service page and click on the Create function button.
      2. On the Create function page, choose the Author from Scratch option, name your function, and select Node.js as Runtime.
      3. Under the Permissions section, click the Change default execution role link, select Use an existing role, and choose the role created in Step 1.
      4. Click the Create function button.
      5. Navigate to the inline code editor, open the index.js file, and replace the existing code with the following code:
        'use strict';
        
        exports.handler = (event, context, callback) => {
          const response = {
            statusCode: 200,
            body: JSON.stringify({
              message: 'Received message successfully!',
              input: event,
            }),
          };
        
          var body = event.Records[0].body;
          console.log("Received message: ",body,event.Records[0]);
        
          callback(null, response);
        };
        
        Next, let’s create an SQS trigger that will activate when a message is enqueued. This trigger will process the message and delete it from the queue after its successful processing.
      6. Scroll up to theFunction overview section and click the +Add trigger button.
      7. From the Trigger configuration dropdown menu, select SQS.
      8. Enter the queue name, select the Enable trigger checkbox, and click on the Add button.Add Trigger.PNG

        Note: This will only log the data in the AWS Cloudwatch. You can write your business logic to process the received messages.

    7. Try out the Setup

      After completion of this setup, go to Contentstack, publish any entry (from which Webhook will trigger), and then check the logs in AWS Cloudwatch.

    Additional Resource: To use webhooks for managing translation, you can refer to our guide on Setting up Translation System with Contentstack Webhooks, Memsource, and AWS Lambda for more details.

    Next Steps

    Now that you know how to use AWS SQS, you can try to set up a notification System with Contentstack Webhooks, AWS, Lambda, and AWS SNS.

    Was this article helpful?

    Thanks for your feedbackSmile-icon

    On This Page

    ^