Add Elasticsearch to Contentstack-powered websites

Elasticsearch is a popular document-oriented database. It's designed to store, manage, and retrieve document-oriented or semi-structured data. The data in Elasticsearch is stored in the JSON document format. You can then create queries to retrieve the data stored in Elasticsearch.

Additional Resource: We also have a guide that elaborates on how you can integrate Swiftype Search with your Contentstack-powered website.

In this document, we will discuss the steps required to use Elasticsearch with your Contentstack-powered websites.

Overview

We will set up webhooks such that whenever any content is published, unpublished, or deleted in Contentstack, a webhook will be triggered. This will invoke the specified application to perform CRUD operations in Elasticsearch.

Prerequisites

Basic understanding of Nodejs and Elasticsearch
Contentstack account

Setting up the Search

To use Elasticsearch with Contentstack, you need to follow the steps given below:

Set up Webhooks in Contentstack
Install and set up Elasticsearch
Configure your application

Set up Webhooks in Contentstack
Log in to your Contentstack account and perform the following steps:
1. Navigate to the Settings gear icon and click on Webhooks.
2. On the Webhooks page, click on + New Webhook.
3. On the Create Webhook page, fill up the Name field for now and add a dummy URL in the URL field (we will fill this up later).
4. Scroll down to the When section for creating a trigger for the webhook.
5. Enter the content types, entries, and assets that you’d like to be notified on they are updated.
  
  Ideally, setting up webhooks for the following should work fine:
  
  All Entries: publish, unpublish, and delete
  All Assets: publish, unpublish, and delete
  Content type: delete
  1. Once done, click on Save to save your settings.
With this, you have set up the Webhook for triggering notifications. Let's move on to setting up Elasticsearch.
Install and set up Elasticsearch
Before moving to set up Elasticsearch, we recommend you to read through the Getting started with Elasticsearch article. Follow the steps given there to set up the base with Elasticsearch.
Note: In this example, we have used a local version of Elasticsearch server running on the system.
Make sure you create the right index for your data before moving forward. Here’s a sample of the Elasticsearch index used for our example:
```
{
  "settings" : {
    "number_of_shards" : 1
  },
  "mappings" : {
    "properties" : {
      "title" : { 
        "type" : "text"
      },
      "url": {
        "type": "text"
      },
      "data": {
        "type": "text"
      },
      "locale": {
        "type": "keyword"
      },
      "uid": {
        "type": "keyword"
      },
      "content_type_uid": {
        "type": "keyword"
      }
    }
  }
}
```
We'll be storing 6 properties for our content.
- The “title” property would be the entry’s title field which can also be used for searching.
- The “url” property is the page’s URL field which helps our website to show links to the page where the searched phrase exists.
- The “data” property is the field where the contents we’d want to search on.
- The “uid” property will be a combination of the entry’s UID and LOCALE. The locale property is also prefixed, in case you have a multilingual website and the entry is published or unpublished in only one of the multiple languages.
- The “locale” property could be used to identify the language in which the content gets published.
- The “content_type_uid” property is used to remove all the documents that are part of it, in case the content type is deleted.
Now, once the indexes have been created, we can move on to create the application.
Note: For interactive use, or to play around you can use Elastic - Kibana to visualize the data you are working with.

Configure your Application

Here, we will use the expressjs framework for creating a sample, you can choose any framework of your choice.

Install and get started with expressjs
Create all routes and place them inside the “routes” folder
Create an “index.js” file inside the routes folder.
Create a “routes/contentstack-webhook.js” file that will handle the webhook triggered through Contentstack.

Note: Make sure to handle the entry's and asset’s publish, unpublish, and delete, accordingly. In case of content type deletion, you’d need to remove all the documents that are part of it.

Here’s a sample snippet to get you started.

Note: We are using @elastic/elasticsearch v7.3.0 in our examples. Also, we are saving the “_id” of each document as the combination of the entry's UID, followed by its locale.

We are doing this as it will help us to remove the entry from a particular locale when unpublished.

For content upserts

 const type = req.body.module
   const data = req.body.data
   const event = req.body.event
   const elasticData = {
     title: data.entry.title || 'Undefined',
     uid: data.entry.uid,
     id: `${data.entry.uid}_${data.locale}`,
     locale: data.locale,
     data: data.entry.data,
     url: data.entry.url,
     content_type_uid: data.content_type.uid,
   }
   return client.update({
     index: indexName,
     id: elasticData.id,
     body: {
       scripted_upsert: true,
       doc: elasticData,
       upsert: elasticData
     }
   }).then(({ body }) => {
     res.status(200).send('Indexed to elastic search successfully!')
   }).catch((error) => {
     res.status(422).send('Unprocessible Entity!')
   })

For content unpublishing

 return client.delete({
     index: indexName,
     id: `${data.entry.uid}_${data.locale}`,
   }).then(({body}) => {
     res.status(200).send('Document unpublished from elasticsearch successfully!')
   }).catch((error) => {
     res.status(422).send('Unprocessible Entity!')
   })

For entry and asset deletion

return client.deleteByQuery({
     index: indexName,
     body: {
       query: {
         match: {
           uid: data.entry.uid
         }
       }
     }
   }).then(({body}) => {
     res.send('Document deleted from elasticsearch successfully!')
   }).catch((error) => {
     res.status(422).send('Unprocessible Entity!')
   })

For content type deletion

   return client.deleteByQuery({
     index: indexName,
     body: {
       query: {
         match: {
           content_type_uid: data.content_type.uid
         }
       }
     }
   }).then(({body}) => {
     res.status(200).send(`Document of content type ${data.content_type.uid} removed from elasticsearch successfully!`)
   }).catch((error) => {
     res.status(422).send('Unprocessible Entity!')
   })

Once you have your application set up, you’d need to add a public URL to it. To sample this on your local system, you can use ngrok.

After the installation, you’ll get a public IP, which you can copy and paste it in Contentstack webhooks (in the URL to Notify field). So, any time the content you are working on is updated/published, the webhook will be triggered on the URL which in turn will notify the application to update the data.

Recommendations

By default, only entries having URL would get indexed, according to the above sample. Thus, you won’t be able to index entries that are part of content blocks OR references.
If you like to index “pages” instead of entries, we recommend that you make a request to the “Webpage” on your site as soon as an entry is published. Scrape its contents (NPM libraries like cheerio is a useful tool) and then save the content in Elasticsearch.

Add Elasticsearch to Contentstack-powered websites

Overview

Prerequisites

Setting up the Search

Set up Webhooks in Contentstack

Install and set up Elasticsearch

Configure your Application

Recommendations

More articles in "Working with Search Platforms"

On This Page