New! We are thrilled to be launching our CommunityNew! Contentstack CommunityJoin today

Headless CMS blog

Read everything on headless CMS technology, tips, best practices, and how-tos

Jul 14, 2022

What You Need to Know About E2E Testing with Playwright

Contentstack recently launched Marketplace, a one-stop destination that allows users to find, create and publish apps, connect with third-party apps and more. It aims to amplify the customer experience by enabling them to streamline operations. Marketplace now has a few Contentstack-developed apps and we will introduce more in the future.Initially, we tried to test these apps manually but found this too time-consuming and not scalable. The alternative was to use an end-to-end (E2E) testing tool (Playwright in our case), which helped us streamline and accelerate the process of publishing the apps.Playwright is a testing and automation framework that enables E2E testing for web apps. We chose Playwright because of its classic, reliable and fast approach. Besides, its key features such as one-time login, web-first approach, codegen and auto-wait make Playwright suitable for the task at hand.This article will walk you through the processes we used and the learnings we gathered.Our Testing ProcessesIn this section, we detail the processes we followed to test the Marketplace apps using Playwright.Set-up and Tear-down of Test DataPlaywright permits setting up (prerequisites) and tearing down (post-processing) of test data on the go, which helped us accelerate our testing.There are additional options available for this:global set-upglobal tear-downbeforeAll & afterAll hooksbeforEach & afterEach hooksIdeally, a test establishes prerequisites automatically, thereby saving time. Playwright helped us do that easily. Once the test was concluded, we deleted the app, content type, entry or the other data we initially set up.Playwright helped us achieve the following on the go:Auto-create app in the dev centerAuto-create content type and entryTest Directory StructureWe added all the test-related files and data to the test's repository. The following example explains the process:For the illustration app (see image below), we added the E2E test inside the 'test/e2e' folder.Next, we included the 'page-objects/pages' (diverse classes) for multiple web pages and tests. The Page Object Model is a popular pattern that allows abstractions on web pages, simplifying the interactions among various tests.We then placed the different tests (spec.js) under the test folders and the utility operations under /utilsAll the dependencies of E2E tests were put in the same .json package but under dev dependencies.We attached .env(env. sample) with correct comments to add the environment variables correctly.After that, we added support for basic auth on staging/dev.In the next stage, we added the Readme.md details about the project.We used the global-setup for login management to avoid multiple logins.Next, we used the global-tear-down option to break the test data produced during the global-setup stage.Finally, we used beforeAll/afterAll hooks to set-up/breakdown test data for discrete tests.How to Use Playwright Config Options & Test HooksGlobal-setup & Global tear-down:Both global-setup and global tear-down can be configured in the Playwright config file.Use global-setup to avoid multiple logins (or any other task later required during the test execution) before the tests start:Global set-up easily evades repetitive steps like basic auth login and organization selection.That way, when the tests are conducted, the basic requirements are already in place.Below is the example of a sample code snippet for a global set-up:Use global-tear-down to break down any test data created during the global-setup file.The test data generated using global-setup can be eliminated in global-teardown.While global-setup/global-teardown are the config option/s for an entire test suite, before/after tests hooks are for the individual tests.Test Hooks Available in PlaywrightPlaywright hooks improve the efficiency of testing solutions. Here is a list of test hooks available in Playwright:test.beforeAll & test.afterAllThe test.beforeAll hook sets test data shared between test execution like entries, creating content types and establishing a new stack. The test.afterAll hook is used to break or tear the test data. This option helps eliminate any trace of data created for test purposes.test.beforeEach&test.afterEachThis hook is leveraged to set up and break down test data for individual tests. However, the individual text execution and the concurring data might vary. Users can set up the data according to their needs.Tips & Tricks for Using PlaywrightWhile using Playwright, we learned a few valuable lessons and tips that could be useful to you:Using the codegen feature to create tests by recording your actions is a time-saving approach.You can configure Retires in the playwright config file. It helps in case of a test failure. You can re-run the test to come up with relevant results.The Trace Viewer allows you to investigate a test failure. This feature includes test execution screencast, action explorer, test source, live DOM snapshots and more.Use the timeout feature for execution and assertion during testing.By setting up a logger on Playwright, you can visualize the test execution and breakpoints.Using the test data attributes during a feature development navigates the test through multiple elements, allowing you to identify any element on the DOM quickly.Recommended Best PracticesWhile using Playwright for E2E testing of our marketplace apps, we identified a few best practices that might come in handy for other use cases.Parallelism:Test files run by default on Playwright, allowing multiple worker processes to run simultaneously.Tests can be conducted in a single file using the same worker process.It's possible to disable the test/file execution and run it parallelly to reduce workers in the config file.The execution time increases with the number of tests; parallelly running tests are independent.Isolation:Each browser context is a separate incognito instance, and it's advisable to run each test in individual browsers to avoid any clash.In isolation, each browser can emulate multi-page scenarios.It's possible to set up multiple project requirements in playwright config as per the test environment similar to baseURL, devices and browserName.Speed of Execution:Parallel test execution, assigning worker processes and isolation expedite the running of test results.Elements like test data, global tear-down and set-up affect the execution speed regardless of the number of worker processes.Double Quotes Usage:Use double quotes if you come across multiple elements on the exact partial string.Help establish case sensitivity. For instance, awaitpage.locator('text=Checkout') can return both elements if it finds a "Checkout" button and another "Check out this new shoe."The double usage quotes can also help return the button on its own, like await page.locator('text="Checkout"'). For details, check out the Playwright text selectors.Prioritizing User-facing Attributes:It's advisable to use user-facing elements like text context, accessibility tiles and labels whenever possible. Avoid using "id" or "class" to identify elements. For example, use await page.locator('text=Login') instead of await page.locator('#login-button') is recommended.A real user will not find the id but the button by the text content.Use Locators Instead of Selectors:Locators will reduce flakiness or breakage when your web page changes. You may not notice breakages when using standard selectors.Example:se await page.locator('text=Login').click() instead of await page.click('text=Login').Playwright makes it easy to choose selectors, ensuring proper and non-flaky testing.Wrapping UpIn a world dominated by Continuous Integration and Delivery, E2E testing is the need of the hour. Though it's a tedious task, following the practices above will save you time and improve your product.

Apr 12, 2022

Zero-cost Disaster Recovery Plan for Applications Running on AWS

Statistics show that over 40% of businesses will not survive a major data loss event without adequate preparation and data protection. Though disasters don’t occur often, the effects can be devastating when they do.A Disaster Recovery Plan (DRP) specifies the measures to minimize the damage of a major data loss event so businesses can respond quickly and resume operations as soon as possible. A well-designed DRP is imperative to ensure business continuity for any organization. If you are running an application, you must have a Disaster Recovery Plan in place, as it allows for sufficient IT recovery and the prevention of data loss. While there are traditional disaster recovery solutions, there has been a shift to the cloud because of its affordability, stability, and scalability. AWS gives the ability to configure multiple Availability Zones to launch an application infrastructure. In an AWS Region, Availability Zones are clusters of discrete data centers with redundant power, networking, and connectivity. If downtime occurs in a single availability zone, AWS will immediately shift the resources to a different availability zone and launch services there. Of course, downtimes do occur occasionally. To better handle them, you should configure the Auto Scaling Groups (ASGs), Load Balancers, Database Clusters, and NAT Gateways in at least three Availability Zones, to withstand (n-1) failures; that is, failure of two availability zones (as depicted in the diagram below). Disaster Management within an AWS RegionRegional Disaster Recovery Plan Options A regional disaster recovery plan is the precursor for a successful business continuity plan and addresses questions our customers often ask, such as:What will be the recovery plan if the entire production AWS region goes down? Do you have a provision to restore the application and database in any other region?What is the recovery time of a regional disaster?What is the anticipated data loss if a regional disaster occurs?The regional disaster recovery plan options available with AWS range from low-cost and low-complexity (of making backups) to more complex (using multiple active AWS Regions). Depending on your budget and the uptime SLA, there are three options available:Zero-cost optionModerate-cost optionHigh-cost optionWhile preparing for the regional disaster recovery plan, you need to define two important factors:RTO (Recovery Time Objective) i.e. the time to recover in case of disasterRPO (Recovery Point Objective ) i.e. the maximum amount of data loss expected during the disaster Zero-cost Option: In this approach, you begin with database and configuration backups in the recovery region. The next step involves writing the automation script to facilitate the infrastructure launch within a minimum time in the recovery region. In case of a disaster, the production environment is restored using the existing automation scripts and backups. Though this option increases the RTO, there is no need to launch any infrastructure for disaster recovery.Moderate-cost Option: This approach keeps a minimum infrastructure in sync in the recovery region, i.e. the database and configuration servers. This arrangement reduces the DB backup restoration time, significantly lowering the RTO. High-cost option: This is a resource-heavy approach that involves installing load balancers in the production environment across multiple regions. Though it's an expensive arrangement, with proper implementation and planning the application is successfully recovered with little downtime for a single region disaster. Zero-cost Option: The Steps The zero-cost option does not require the advance launch of additional resources in the recovery region; the only cost incurred is for practicing the DR drills. Step 1: Configure Backups At this stage, reducing data loss is the top priority. The first step is configuring the cross-region backups in the recovery region. With a proper backup configuration, you can reduce RPO. It's essential to configure the cross-region backups of: S3 bucketsDatabase backupsDNS zone file backupsConfiguration (chef/puppet) server configurationCICD (Jenkins/GoCD/ArgoCD) server configurationApplication configurationsAnsible playbooksBash scripts for deployments and cronjobsAny other application dependencies required for restoring the applicationStep 2: Write Infrastructure-as-a-Code (IaaC) Templates - Process Automation Using IaaC to launch the AWS infrastructure and configure the application will reduce the RTO significantly, and automating the process will lessen the likelihood of human errors. Many automation tools are widely available. Terraform code to launch application infrastructure in AWSAnsible playbooks to configure Application AMI, Chef server, CICD servers, MongoDB Replica Sets Clusters, and other standalone serversScripts to bootstrap the EKS clusterStep 3: Prepare for a DR Drill The preparation for a DR drill should be done in advance through a specified process. The following is a sample method to get ready for a DR drill: Select an environment similar to the productionPrepare a plan to launch complete production infrastructure in the recovery regionIdentify all the application dependencies in the recovery regionConfigure the cross-region backup of all the databases & configurationsGet ready with automation scripts with the help of Terraform, Ansible, and Shell-ScriptsIdentify the team members for DR Drill and make their responsibilities known Test your automation scripts and backup restoration in the recovery regionNote the time taken for each task to get a rough estimate of the drill timeStep 4: Execute the DR Drill The objective of the DR drill is to test the automation scripts and obtain the exact RTO. Once the plan is set, decide a date and time to execute your DR drill. Regular practice is advisable to perfect your restoration capabilities. Benefits of DR DrillsPracticing DR Drill boosts confidence that the production environment can be restored within a decided timeline.Drills help identify gaps and provide exact RTO and RPO timelines. They provide your customers with research-backed evidence of your disaster readiness. Conclusion Though AWS regions are very reliable, preparing for a disaster is a business-critical SaaS Application requirement. Multi-region or Multi-cloud deployments are complex, expensive architectures, and deciding the appropriate DR option depends on your budget and uptime SLA to recover during such disasters.

Mar 14, 2022

5 Things You Need to Know About Adopting Kubernetes

Kubernetes has become the go-to container orchestration platform for enterprises. Since the COVID-19 pandemic, organizations have increased their usage by over 68% as tech architects look to Kubernetes to handle the increased delivery demands of the post-pandemic age. Why now? It offers faster deployment, better portability and helps keep pace with the modern software development requirements. Contentstack has adopted Kubernetes as a platform to build the next generation of microservices that power our product, and it offers incredible benefits. But adopting a transformational technology like Kubernetes is never straightforward. We faced our own set of challenges while implementing it, and we learned quite a few lessons in the process. In this piece, we share what we learned to help you avoid common pitfalls and position yourself for success.Kubernetes is Not Just an Ops Concern With the rise of DevOps, more companies are moving toward the “you build it, you run it” approach. The operations team is no longer the only team responsible for maintaining the app. Developers get more operational responsibilities, bringing them closer to the customers and improving issue resolution time. However, not all developers have exposure to Kubernetes. This may lead to developers being unable to optimize their applications. Some of their struggles include:Inability to implement “Cloud Native” patternsDifficulty in communicating and implementing scaling requirements for servicesInflexible configuration management for servicesInadvertent security loopholes Developers also face debugging issues in production, which can often lead to catastrophic outcomes for organizations that are targeting strict availability SLAs. We realized this when we started using Kubernetes at Contentstack. We manage this gap by investing in upskilling developers in Kubernetes. It has improved our SDLC (Software Development Life Cycle) lead time. We see improved communication between developers and operators and a clear shift to “Cloud Native” solutions.Pay Close Attention to Microservice Delivery “Microservice delivery” refers to the testing, packaging, integrating and deploying microservices to production. Streamlining delivery is an important aspect of managing microservices. For example, simply moving microservice deployments to Kubernetes will not give you immediate benefits if those deployments are not automated. Following are some of the first steps for setting up an efficient delivery pipeline:Package your microservices: While containers allow you to bundle the application code, you still need an abstraction for managing the application’s Kubernetes configuration. Kubernetes configuration is required for defining the microservice image, ports, scaling parameters, monitoring, etc. At Contentstack, we use Helm as the package manager.Implement a CI/CD pipeline to automate delivery: A CI/CD (Continuous Integration/Continuous Delivery) pipeline automates the entire process from building and testing apps to packaging and deploying them. Read more about Contentstack’s GitOps-based approach to CI/CD.Automate tests for your application: An ideal test suite gives you fast feedback while being reliable at the same time. You can achieve this by structuring your tests as a pyramid. These automated tests need to be integrated into the CI/CD pipeline so no defective code makes its way into production.Create a strategy for potential rollbacks: Failures are part of software development. However, your strategy for dealing with failures determines the reliability of your services. ‘Rollbacks’ is one such strategy. It involves re-deploying the previous working version when a build fails. A battle-tested strategy for implementing rollbacks using the CI/CD pipeline needs to be in place so you can handle deployment failures gracefully.Secure Your Workloads It’s no secret that containerization and microservices bring agility. However, security shouldn’t take a backseat. A recent survey found that about 94% of respondents experienced at least one security incident in their Kubernetes environments in the last 12 months. Most of these security issues were due to misconfiguration or manual errors. Security is one of the top challenges for the adoption of Kubernetes in the enterprise. We took several steps to ensure that our applications on Kubernetes clusters are secure:Scan container images: Docker images are atomic packages of the microservice. These images are a combination of the application code and the runtime environment. These runtimes should be scanned regularly for Common Vulnerabilities and Exposures (CVEs). We use Amazon ECR’s Image Scanning feature to enable this.Secure intra-cluster communication: Microservices running in a cluster need to talk to each other. These microservices may run across multiple nodes. The communication between them must be encrypted and they must be able to communicate only with authorized services. mTLS (mutual-TLS) is a great standard that helps to encrypt and authenticate clients. At Contentstack, we use istio, a service mesh tool, to automate the provisioning and rotation of mTLS certificates.Manage secrets and credential injection: Injecting credentials into microservices is required for the microservices to connect to databases and other external services. You must manage these credentials carefully. There are several techniques and tools to do this, including using version-controlled sealed-secrets and Hashicorp Vault. This also helps improve reliability of your deployments using automation.Invest in Effective Monitoring for Your Services According to the “Kubernetes and Cloud Native Operations Report 2021,” about 64% of respondents said maintenance, monitoring and automation are the most important goals for their team to become cloud-native. Monitoring is an often overlooked aspect of operations, but it is crucial, especially when moving to platforms like Kubernetes. While Kubernetes may make it very easy to run your services, it may trick you into believing everything will keep working as expected. The fact is, microservices may fail for a variety of reasons. Without effective monitoring in place, your customers may alert you of degraded performance, instead of you catching it first. Contentstack has several measures in place to monitor our services:Use centralized logging tools: When using microservices, having a centralized logging tool is invaluable. It helps developers and operations teams debug and trace issues across several microservices. Without access to centralized logging, you will have to spend a lot of time manually co-relating and tracking logs.Create monitors and alerts: For operations, there are several SLAs (Service Level Agreements) and SLOs (Service Level Objectives) that are monitored. Getting alerts (on messaging tools like Slack) on degraded performance will help you take timely action. It will also help you predict and prevent potentially catastrophic issues.Create monitoring dashboards: Comprehensive monitoring dashboards give you a birds-eye view of the health of the microservices. Dashboards are a perfect starting point for daily monitoring. At Contentstack, each team that manages a fleet of microservices has its own dashboard. Team members routinely check these dashboards for potential issues. Since both developers and operations teams rely on these dashboards, we can co-relate application information from both application logs and infrastructure monitors on the same dashboard.Take Advantage of Being ‘Cloud Native’ Kubernetes is an all-encompassing platform that offers many abstractions for solving common infrastructure problems. While the solutions address infrastructure problems, they can also solve application problems. Being “Cloud Native” combines using certain patterns and techniques with cloud-native tools. Here are some examples:Sidecar pattern: The sidecar pattern is the foundation of most service mesh technologies. It involves having a companion application (injected automatically) along with the main application container. In service meshes, it is used for routing and filtering all traffic to the main application container. At Contentstack, we have leveraged this pattern for distributed authorization across the cluster. Each application communicates with an authorization sidecar to validate incoming requests.Kubernetes jobs: Your application may have to process some one-off tasks that are not in sync with the request-response cycle, such as batch processing jobs. In the past, we depended on a separate service that kept running in the background looking for new tasks to process. Kubernetes comes out of the box with “Jobs,” which allows running such tasks as a pod. At Contentstack, we use Jobs for running database migrations before releasing a new version of an application on the cluster.Health probes: Kubernetes has a good health check system in place for your services. This means it will notify you if the health of any service is not as expected. Apart from notification, it also supports automatically restarting the service. Read more about how Contentstack configures health probes for its services. At Contentstack, we strive to continuously learn and adopt new practices and technology to stay ahead of the curve, and we are glad to share what we learn with the rest of the world. Adopting Kubernetes is a slow but rewarding journey that allows you to take advantage of the latest best practices for creating resource-efficient, performant and reliable infrastructure.

Mar 10, 2022

How to Get Your Technical Debt Under Control

Unless you live in la-la land with a dream team of developers who write perfect code, it is impossible to avoid technical debt. It is a natural by-product of fast-paced development of any SaaS product. As developers, we always have to choose between delivering on time and delivering with perfect code. It's a trade-off. In most cases, we choose to deliver on time, with a promise to deal with the byproduct later. Technical debt also occurs because of reasons such as:Change in technologyDevelopment of annual frameworks and librariesNon-maintenance of codebase or librariesIntroduction of new product featuresAddition of a new workforce to an existing project to ship fasterNeed for frequent releases and pressure of timeline And so, technical debt accumulates. We can either ignore it until it snowballs into something massive or fix some of it to reduce its impact on current and future development. I have seen teams try to figure out everything from the beginning so there is no minimum tech debt. But in an environment where delivery time matters, slowing down development could cause companies to lose opportunities or customers to competitors. It makes more sense to come to peace with the fact that there will always be some technical debt. Acknowledging this and then defining some best practices to manage technical debt effectively can reduce its impact on your product.10 Ways to Manage Your Technical Debt1. Opt for Modular and Extensible Architecture An excellent way to start a new project is by adopting an extensible architecture. This involves accounting for all our current requirements while extending the project to add new features in the future without any rewriting. Start with identifying the various modules based on their functionalities. Develop each module so it is independent and does not affect the working of other modules. However, all the modules should be loosely coupled. By doing this, you can easily break the project into multiple microservices and scale them individually if the need arises.2. Develop Only What is Required To build a product or its modules with more flexibility (or to make the next release easy), developers may load it with extra functionalities. This is only effective if you are sure about the future requirements, which is never the case. These added functionalities build up over time, increasing your technical debt.3. Plan Your Trade-Offs Carefully Pick what matters the most. For long-term projects that have a high return on investment, carefully consider the design and implementation and minimize technical debt as much as possible. If delivery is the top priority and efforts to build are low, you can knowingly create some debt you can fix later.4. Never send POC to production Before developing a new feature or a product, developers build a proof of concept (POC) to check its feasibility or to convey the idea. If the POC is accepted, the feature is included in the road map. At times, especially when it is working as expected, it is very tempting to put this POC into production. However, that is a bad idea. A POC is just what it says: a concept, not a complete solution. While developing a POC, we rarely think of all use cases or scenarios because the focus is on writing the code to get desired results quickly. That code is never meant for production, and writing tests around it usually doesn't work. Most of the time, POCs don’t have proper structures, error handling, data validation or extensibility. Use it as a reference and only deploy a complete production solution.5. Write Code With Proper Documentation Good code with proper documentation offers multiple benefits, including quick handoff to others, increased reusability and reduced time to build more on top of your code. There are two types of documentation: within your code and about your code. Both are equally important. Some examples of documentation within your code are:Function signatureInstructions for usersDescriptions explaining confusing or complex pieces of codeTo-dos for future reconsiderationNotes for yourselfComments about recent changes Examples of documentation about your code include:Readme filesAPIs referenceHow-to guidesFAQs6. Request Early Code Freezes Frequent releases (often with limited resources) are major contributors to the accumulation of technical debt. To meet tight deadlines, developers add hard-coded values or resort to quick fixes in codes. And there are always last-minute requirements or requests for change in functionalities. Over time, these quick fixes lead to performance, stability and functional issues. A good way to deal with this is to introduce a process for early code freeze, so developers can focus on improving code quality and stability instead of adding code until the last minute.7. Add Technical Debt in Sprint Eliminating technical debt all at once is not feasible. Product managers are not likely to commit extensive resources since it adds no direct or immediate value to customers. At the same time, developers want to spend more time developing something concrete, from scratch, instead of reworking on the same code. But there is a practical way out: Create a healthy backlog of debt items and a road map (with timeline) to replace the code's hard-coded, unoptimized and quick-fix scenarios. Add stories to every sprint so you can fix these debt items gradually and in a more organized way.8. Version Your Code Your code will likely change as you add new features to your application. One of the best ways to track and maintain these changes is to version your code. You can create incremental code through versioning, leave behind some old code, introduce breaking changes through newer versions, give users access to multiple versions and stop support for older versions. All of this is in your control. Versioning is probably the best process through which your code evolves. It is also the preferred way in the SaaS world to shape (or reshape) your product over time without piling up a lot of baggage from the legacy code.9. Refactoring is Your Friend Refactoring is the ultimate weapon in your fight against tech debt. How often you need to refactor the code is up to you, but doing this regularly is essential. Make sure you version your changes while refactoring. This keeps your code organized.10. Balance is Key Technical debt is okay as long as it's intentional and strategic and not the result of poor codes and design. As developers, our aim should always be to balance it and cushion its blow to the product.

Nov 09, 2021

What is Terraform and How to Use It with S3

With Infrastructure as Code (IaC), developers can automate an entire data center using programming scripts. It is a method of building, managing, and provisioning IT infrastructures combining both physical and virtual machines. IaC has multiple benefits; who would not want to use code to create and provision IT infrastructure rather than do it through a grueling manual process? IaC does suffer from a few significant hurdles:One has to learn the codeThe change impact is often undeterminedChanges are untrackableIt cannot automate a resource A simple solution to these challenges is Terraform. That brings us to one of the most trending questions in the tech industry today.What Is Terraform? Developed by HashiCorp, Terraform is a popular open-source IaC tool. Developers use it to create, define, and provision infrastructures using easy-to-follow declarative languages. Using Terraform, you can set up a cloud infrastructure as codes. It is similar to CloudFormation used to automate AWS, with the only difference being Terraform can be used in combination with other cloud platforms. How Does Terraform Operate? For a complete understanding of Terraform, it’s beneficial to know how its life cycle operates and its four primary stages. Init - Plan- Apply- Destroy Init: In this stage, Terraform initializes the dictionary, which contains all the configuration files. Terraform downloads the required provider’s configuration in the current working directory. Plan: Once a developer creates a desired state of infrastructure, creating an execution plan helps to reach that state. Changes made in the existing configuration files achieve that state. Apply: As the name suggests, the Apply stage executes the plans developed in the planning stage. By the end of this stage, the infrastructure’s state mirrors the desired one. Destroy: This stage lets you delete old infrastructure files and resources rendered inapplicable in the Apply phase. After making the changes, your desired infrastructure is ready. All you have to do is store your codes. Storing code is where Terraform’s remote state comes into play. Terraform Remote State Terraform state is simply a screengrab of the infrastructure from when you last ran the Apply command. You might have done it on your S3 bucket, DNS record, or even the Github repository. It’s simply a JSON file that includes the details of each configuration characteristic since you last applied the Terraform scripts. Virtually, these states are essential as it is through these that Terraform determines what’s new and what’s redundant. As a rule of thumb, Terraform stores the states locally, which works as long as there are not many developers working on it. As soon as more than two developers work on it, the simple magic of Terraform gets complicated. Besides, each user will require the latest state to take the work further, and when one is using the Terraform, others will not be able to run it. Terraform’s remote state presents a simple solution. Using backends informs Terraform to move snapshots of the infrastructure from the terraform.tfstate file on the local filesystem to a remote location, such as on S3 buckets, Azure Blob Storage, Alibaba Cloud OSS, or something else. Cloud engineers dread the idea of losing Terraform state files, a problem cleverly avoided by using Terraform’s remote state. The following explains other benefits that make the Terraform remote state popular.Benefits of Terraform Remote State A centralized repository: This is one of the primary benefits of Terraform’s remote state, a centralized location of facts. A single source of truth becomes crucial if a team is working on a specific project and using more than one machine. After issuing the Terraform apply command, the state of the infrastructure following each change is uploaded to the backend. That way, any person or machine wanting to make further changes will be aware of the changes already made, thereby not removing or undoing them. You can check out a few Terraform remote state examples to get a better understanding of the process. An extra durability wall: When you store remotely, you can take durability a notch up. In simpler words, Terraform remote state makes it a lot harder to delete any .tfstate file stored remotely accidentally. State locking: In case multiple engineers are working on a single environment using a terraform code, the remote state is locked if any changes are being made to the infrastructure by any engineer. This feature further prevents data corruption that might occur if two engineers are applying different changes simultaneously. Terraform keeps other tasks on hold until applying all changes. Intermodule referencing: Storing state files remotely helps when retrieving resource details created outside the current module. Example: The AWS Security group module requires an association with a VPC ID. In a scenario where VPC and Security groups are written separately, the state file of the VPC module is fetched in the security groups’ code. To store the VPC module’s state file in an AWS S3 bucket, you use a remote backend configuration like this: terraform { backend "s3" { bucket = "s3_bucket_name_to_store_state_file" key = "vpc/terraform.tfstate" region = "aws_s3_bucket_region" } } The VPC ID needs to be exported as output to be referred by other modules like this: output "vpc_id" { value = aws_vpc.vpc.id } In the Security group module, VPC ID stored in terraform state file needs to be fetched with a data resource as shown below: Data resource: data "terraform_remote_state" "vpc" { backend = "s3" config = { bucket = "s3_bucket_name_for_state_file" key = "vpc/terraform.tfstate" region = "aws_s3_bucket_region" } } The above data resource fetches the VPC state file. Here the output file plays a key role while reading the values from the backend. To get VPC ID from the output of the VPC module, we can use the following: data.terraform_remote_state.vpc.outputs.vpc_id Terraform Workspace Once you know what Terraform is and have worked with Terraform workflow using the init, plan, apply, and destroy stages, you already have a working knowledge of Terraform workspace as its a default feature. Terraform workspace allows you to create a diverse independent state using the same configuration. Moreover, since these are compatible with the remote backend, all teams share the workspaces. An associated backend is assigned to each Terraform configuration highlighting the operation progress and the specific location of a given Terraform state. This persistent data that is stored in the backend belongs to the workspace. Though initially there is only one “default” workspace, specific backends support multiple workspaces using a single configuration. The default workspace, however, cannot be deleted. The list contains S3, Manta, AzureRM, Local, and Postgres, among a few others. Of course, there are multiple advantages of using Terraform workspace, two of which are:Chances of improved features coming upReduced code usage One of the primary advantages of Terraform workspace is the simplicity it brings to the entire coding process, as seen in the following use case. Example: Two or more development servers require similar Terraform codes in the initial development phase. However, Terraform anticipates further infrastructure changes. Now let’s consider storing the backend files to AWS S3 and add them in the following manner: In this scenario, the Terraform workspace is effective. But before applying the Terraform code, it’s advisable to create workspaces, for instance, Dev1 and Dev2, using the following command: terraform workspace new Dev1 && terraform workspace new Dev2 Note: Environment is workspace_key_prefix in the above use case. terraform { backend "s3" { bucket = "s3_bucket_name_to_store_state_file" workspace_key_prefix = "environment" key = "vpc/terraform.tfstate" region = "aws_s3_bucket_region" } } S3 and Terraform, and Deployment with Jenkins While other storage facilities are compatible with Terraform, S3 is considered the safest for the following reasons. Simple usage: The S3 bucket is straightforward to use. It’s a unique platform designed to simplify functions as much as possible. It also features mobile and web app management consoles. Besides, developers can use APIs to integrate S3 with other technologies. Overall, it’s cost-effective and transfers data to third-party networks. Enhanced security: Not only does S3 allow data transfer on SSL, but it also enables automatic data encryption upon uploading. You can also use it to configure policies that grant permission to objects. Initiating the AWS identity, S3 makes way for controlled access to crucial information. It’s possible to monitor who is accessing the data, location, and devices used. S3 also features VPC endpoints which makes way for secured connection creation. Reliability and durability: S3 offers a robust infrastructure for storing essential data. Additionally, it provides 98.99% durability of objects. Since S3 stores data across multiple devices, it improves data accessibility. Being available in different regions, S3 allows geographical redundancy per region. Integration: It’s simple to integrate S3 with other services available on AWS. It can be embedded with other security platforms like KNS or IAM and even with computing platforms like Lambda. For these reasons and more, the S3 bucket is quite popular among developers using Terraform. However, the deployment of S3 with Jenkins gets tricky. At Contentstack, we use Jenkins for Terraform code deployment. We include two bash scripts to run the codes, requiring two inputs, i.e., environment and resource names. The following tasks are required to execute a bash script using Jenkins: Download the code from GitHub. You can opt for a simple git clone too Validate the Terraform syntax: terraform validate Copy the configuration of the specific environment: tfvars file from the S3 bucketTerraform Plan stage: Execute the plan command and get the outputs displayed in the Jenkins consoleTerraform Apply stage: Launches the resources Note: At Contentstack, we use separate Jenkins jobs for the Terraform Apply command in the test environments. Contentstack and Terraform At Contentstack, we use Terraform for two purposes: infrastructure management and disaster recovery. The following sections share examples of how Contentstack uses Terraform for infrastructure management and disaster recovery. Infrastructure Management Using Terraform Opting for traditional click-ops can take days to deploy or even months; besides, there is no guarantee that they will be error-free. However, with Terraform, complete deployment and migration happen in a matter of minutes. Thus, Contentstack is currently using Terraforms to migrate all its services from EC2 to dockers, specifically Kubernetes. Contentstack currently runs on hybrid infrastructure, combining EC2 and dockers. All the code is written using Terraform to ensure zero human error and repetition. We chose Kubernetes for managing the infrastructure, and the infrastructure was launched using AWS. You can use code from AWS resources like EC2, EKS, SNS, S3, SQS, WAF, and load balancers. While the migration is still in progress, the existing Terraform templates ensure effective disaster recovery. Disaster Recovery Terraform allows Contentstack to be flexible and dynamic from the disaster recovery perspective. The cloud infrastructure enables us to set the infrastructure on demand which is easier to do using Terraform. We provide an RTO (Recovery Time Objective) of 12-15 hours, i.e., we can launch and restore the complete infrastructure within a 15-hour timeline, which, if done manually, will take days altogether. In the worst-case scenario, if AWS goes down regionally, Contentstack can restore data in a short time, thanks to Terraform. The Recovery Point Objective or RPO provided is till the last minute preventing any form of data loss. For a closer look, view our source code.Hacker’s Corner! Terraform hack: Selecting a single resource from the Jenkins choice box is not feasible in a scenario where you want to launch multiple resources. In this case, using a loop for launching multiple resources rather than selecting one resource at a time is time-saving. Example:ENVIRONMENT=$1 //Input from Jenkins parameterJenkins Shell command: For RESOURCES in vpc sg ec2 sns sqs eks; do bash terraform-deploy.sh $ENVIRONMENT $RESOURCES; doneSummary From improving multi-cloud infrastructure deployment to automating infrastructure management, Terraform is a powerful tool. This infrastructure as code software tool is why more and more developers are using it to help manage their infrastructures. Terraform is indeed the future as it helps enterprises improve and automate their DevOps methodologies.

Sep 02, 2021

GitOps: A New Way of DevOps Delivery

In 2017, seeing the lack of buzzwords in the tech industry, Alex Richardson from Weaveworks coined a new one — GitOps. Humor aside, what started as a blog describing their operational tooling has snowballed into something bigger. Descriptions of GitOps include “the next big thing,” “versioned CI/CD,” and even “placebo.”Declarative Systems To understand GitOps, you need to understand what a declarative system is. A declarative system is one where the code describes the final state of the system. An example of such a system would be the Infrastructure as Code (IaC) system, where YAML defines the configuration. We define the end state with declarative code, and then automation reconciles the system to the desired state.What is GitOps? Shapeshifting - the ability to physically transform oneself through an inherent ability, intervention, or manipulation. GitOps is the management of declarative systems using Git. The core idea of GitOps is to use a Git repository as the source of truth. It holds the desired state of the system and acts as a contract between various components. Here’s what a simple GitOps pattern looks like:The system state is defined in the Git repositoryThe GitOps Operator detects changes in GitTrigger a convergence mechanism to sync desired and observed stateReconcile the system whenever there is driftNotify when the system state changes GitOps tools empower the system to shapeshift based on the state defined in the Git repository. But which declarative system can effectively leverage this pattern? The answer is Kubernetes.What is Kubernetes? κυβερνητες - pronounced Kie-ver-nee-tees means helmsman or pilot in Greek. Over the last few years, containers have made it easy to run cloud-native applications. Containers are lightweight compared to virtual machines (VMs) and make more efficient use of the infrastructure. While container runtimes are great at managing individual containers on one host, they are not suited for managing applications running in multiple containers across multiple hosts. This type of management is where container orchestrators come into play. Kubernetes is an open-source orchestration system that automates the deployment and management of applications running in Linux containers. The analogy of an orchestra conductor is common for describing Kubernetes. What does Kubernetes do?It provides an abstraction layer over infrastructure. Details of underlying computers, networks, and other components are hidden from users and applications, making it easier to develop and configure.It makes an application portable as the underlying infrastructure no longer affects it. It allows deploying the same manifest on local computers and any cloud provider.It uses a declarative model to configure applications. You describe the application in manifest and Kubernetes converts it into a running application.After deploying an application, Kubernetes takes over its management. This management includes keeping applications running, recovering from failures, and scaling in case of changes in demand. Since Kubernetes manages tasks declaratively, we can employ GitOps practices and tools for cluster management and application delivery.GitOps and Kubernetes An anti-pattern is a common response to a recurring problem that is usually ineffective and can end up being counterproductive. The following lists results of a common anti-pattern that occurs with Kubernetes deployment automation when the CI system tests, builds a change, and then pushes the change directly to the Kubernetes cluster. The CI system needs access to the Kubernetes cluster API to support this push-based deployment model. This access poses a security risk.In a namespaced cluster, the deployment pipeline would require cluster-scoped credentials. It significantly increases risk.Another drawback is that the deployment pipeline triggers only when the Git repository changes. It cannot detect deviations from the desired state. A GitOps pipeline uses the same concepts as a push-based pipeline but differs in how the deployment works. Git, the CI system, and the GitOps operator make up the troika of a GitOps pipeline. Unsurprisingly, Git forms the fulcrum around which it revolves. The CI system feeds changes to the Git repository at one end, and the GitOps operator consumes at the other. The pull-based strategy eliminates the need to expose Kubernetes cluster API and credentials.It separates CI and CD systems, thus improving security posture as the two pipelines can only communicate through Git.It provides the flexibility to use any CI and Git tool.The GitOps operator corrects any divergence from the desired state.How Do We Do It? At Contentstack, we use the GitOps pipeline to manage application delivery on Kubernetes. Here’s a glimpse at our pipelines.Developers merge application code changes to the trunk and trigger the CI pipeline. It tests the change, builds a new container image, and pushes it to the container registry in Cloud.We use Helm to package applications for Kubernetes. Any change in the Helm configuration triggers the Helm CI pipeline, which tests the package, creates a chart, and uploads it to the Helm repository.Running either of the CI pipelines invokes the deployment pipeline. It combines artifacts from CI pipelines and writes a new state to the environment repository. We use Git branches to manage multiple environments within the environment repository. Change is propagated through environments using the CI system.The GitOps operator running in a Kubernetes cluster detects a new commit in the environment repository and marks the observed state out of sync. This status change kicks in the convergence mechanism. Kubernetes pulls the latest changes and starts deployment updates to match the state in Git.Upon completion, the GitOps operator notifies the deployment state by setting commit status in the Git repository. The CI system uses this status to determine if the change is successfully deployed and run any post-deployment actions.Why Should You Adopt GitOps?Faster Deployments By adopting GitOps best practices, developers can deploy updates and features to Kubernetes more rapidly. With GitOps, you don’t have to switch tools for deploying your application. Everything happens in the version control system used for developing the application anyways.Easier Compliance and Auditing A change in an application environment requires updating the configuration in the corresponding Git repository. This change creates a complete history of desired state changes, including a record of who made the change and why. Since changes are tracked and recorded in Git, auditing is made easier. Faster Recovery from Failure As you have a complete history of changes in Git over time, recovering from failure is much easier. By reverting some commits, we can step back to a state that used to work previously.Secure Deployments As changes go through the Git repository, users and CI systems don’t need to access the cluster anymore. Separation of responsibility between packaging software and releasing it to an environment embodies the security principle of least privilege, reducing the impact of compromise and providing a smaller attack surface.Backup of the Environment In case of catastrophic failure, it’s possible to restore the environment quickly as the state of the system is stored in Git. Environment repository acts as a natural backup of the cluster.What are the Challenges with GitOps? “Any change, even a change for the better, is always accompanied by drawbacks and discomforts.” ― Arnold BennettSecret Management There is no standard practice on how to manage secrets. To store secrets in Git, we need to encrypt them. This encryption step introduces a separate deployment process for secrets. If they are not stored in Git, then the idea of having your cluster state the same as Git is not valid anymore.Multi-environment Deployments GitOps does not define how to propagate a change to multiple environments. The most popular way of handling different environments is by using different Git branches. To do this, you need to use the CI system.Complete Application Lifecycle GitOps only focuses on application deployment. Other parts of the development lifecycle such as building, testing, packaging are not addressed by GitOps. You need to use another CI system and the GitOps tool to automate the complete development lifecycle. Change of System State One of the assumptions of GitOps is that the cluster state is the same as what’s defined in Git. However, your system changes. It could be due to auto-scaling or system tools updating the state of your resources, so this assumption might not be correct.Conclusion GitOps shows an opinionated way to implement deployment patterns with specific tools using Kubernetes. It does not provide radical new ideas; instead, it builds on established concepts of continuous delivery. As of today, GitOps is the best way to do continuous delivery using Kubernetes. The prevalence of Kubernetes will lead to more and more organizations adopting the GitOps pattern for their deployment needs. Fueled by increasing the user base and maturity of tools, the GitOps pattern will keep improving Kubernetes deployments

Aug 13, 2021

How We Improved Our EMR Performance and Reduced Data Processing Cost By 50%

Businesses across industries leverage big data to make data-driven decisions, identify new opportunities, and improve overall performance. However, this involves processing petabytes of data. Scaling your servers dynamically to accommodate the variable load can be challenging. Services like Amazon’s Elastic MapReduce (EMR), a managed cluster platform that provides the infrastructure to run and scale big data frameworks such as Apache Spark, Hadoop, Hive, and others, can reduce these operational challenges. Having this kind of infrastructure certainly puts you in an excellent position to make the most out of the data. While the default configuration and setup is a great start, you can get significant performance gains and optimized results if you fine-tune specific parameters and resources. At Contentstack, we have been using Apache Spark on AWS EMR for big data analytics. While it does the job well at a reasonable cost, we realized that we could further reduce the processing time, improve the performance, and cut down the associated costs by making a few changes. The article is for the reader who has a basic understanding of Spark and EMR. Let’s look at the steps we performed to cut down our EMR cost by over 50% by achieving 2x performance improvements: Update EMR to the latest versionAvoid Spark from reading a large GZIP fileRemove Ignite instance Optimize code for MongoDB data storageSwitch to the optimum AWS instance type Reduce the number of instances by load testingUpdate EMR to the Latest Version Until some time ago, we ran EMR version 5.23.0, while the latest version was 6.2.0. However, version 6.2.0 was not supported in the m6gd.16xlarge instance. So, we upgraded to the latest possible version, i.e., 5.33.0. As a result of this change, we moved from Spark version 2.4.0 to 2.4.7, which offered several optimizations made to the Spark engine over time, such as a memory leak fix in 2.4.5, a bug fix in Kafka in 2.4.3, and more.Avoid Spark From Reading a Large GZIP File This one step alone has helped us reduce the processing time and improve the performance considerably. We receive a log of the last 24 hours in a single file compressed in GZIP format every day in Amazon S3. However, you cannot split a GZIP compressed file in Spark. Consequently, a single driver node takes up the file for reading and processing. There is no way to prevent the node from reading the file completely. In the meantime, the rest of the nodes lay idle. To understand how Spark processes this file, we look at the directed acyclic graph (DAG) to dig deeper into the problem. As you can see in the above diagram, Stage 2 reads and processes the data, all in a single task because it’s a single GZIP file. This process was taking considerable time. Hence, we decided to make specific changes to the flow. We decided to split the GZIP file into multiple chunks of data (around 250 MB each) to parallel process multiple tasks. We added a new EC2 c5.xlarge instance which decompresses the GZIP file, splits it into several .log files of about 250MB each, and stores them in S3 with a Time to Live (TTL) of 24 hours. This helped us cut down the waiting time for nodes. Also, instead of six r3.xlarge instances, it’s now just one c5.xlarge that waits for the processed files, so we could also cut down on instance cost. As expected, the results are quite encouraging. The new design now splits the one large GZIP into 20 small files, and all of these are read and processed simultaneously. As you can see in the diagram below, the 20 small files are taken as 20 tasks in Stage 4, thereby reducing the overall processing time. Remove Ignite Instance We used an Ignite instance to store the aggregated data from five different log files and later add them to create analytics data. And we used one r3.8xlarge instance for this. Consequently, you need to process all the log records in sequence. If any job failed due to a corrupted log entry or another reason, we had to reprocess all the jobs and get the aggregated data again. This reprocessing made us rethink our strategy, and after contemplating some options, we concluded that we might not need an ignite instance. We realized that we could restructure the data stored in the database to run this log file job independently. In the image below, the left-hand side represents how we stored data in Ignite for each log (k1) file, and then the aggregated data is saved in MongoDB. The right-hand side data shows the newly restructured JSON that we now store independently after each log processing.(k1, k2, and k3 are the types of services we use) Restructuring the JSON has helped us in the following two ways:Run the log processing job independentlyRemove unwanted Ignite instance (r3.8xlarge)This is a classic example of the KISS (keep it simple, stupid) design principle in action. Optimize Code for MongoDB Data Storage Earlier, we used to run a for-loop on the aggregated data from the Ignite cursor to create JSON data and store it in MongoDB. We do this to keep the JSON data size under 16MB so that we don’t hit the max size limit for the MongoDB document. However, the for-loop function can be run on a single thread at the time in Spark, which practically rendered our 180 parallelisms useless. So, despite having very high parallel-processing capacity, the for-loop was limiting us to using just one thread at a time. We added some new code, where we used Spark’s inbuilt collect_list to create a JSON array. We then used UDF to split this array into multiple arrays and used explode to create another array of JSON. These few lines of code helped us eliminate the usage of for-loop and take full advantage of parallelism in Spark.Switch to the Optimum AWS Instance Type Until recently, we were using a r3.8xlarge instance type, which provided us 32 vCPUs and 244.0 GiB at $2.66/hour. We looked at the various instance types that AWS provides and found that m6gd.16xlarge was optimal for our use case, which provided 64 vCPUs and 256.0 GiB at $2.89/hour. By just paying $0.23 more per hour, we doubled our vCPUs. This change helped us to optimize our Spark configurations. The old config is on the left-hand side in the image below, and you can see the new, updated config on the right-hand side. An essential learning from this exercise is to keep instances updated with optimal core or memory as per the use case.Reducing the Number of Instances by Load Testing With the new ETL architecture and changes in instances, it was time to do a load test and determine if we were using the resources optimally. We started load testing by generating artificial data of a size similar to that of our production. In the process, we found that we could remove two more instances, apart from the Ignite Instance (1 r3.8xlarge) that we had removed earlier, bringing the total down to three instances (1 master and 2 nodes) on production from six earlier. And the sample data set, which took about 2 hours 3 minutes earlier, now took just about 1 hour 42 minutes.The Outcome of These Optimization Strategies We optimized our EMR performance by 2x, thereby reducing the data processing costs by over 50%. And we learned some valuable insights while doing these. Here is a bulleted list of our learnings:To optimize the process, understand how the current engine/system/framework works internally, how it processes data, and with what it is compatible (and not compatible) Keep updating your software and systems to the latest versionsTry to evaluate and optimize your code at regular intervalsNever use Spark for any task that does not let you use its parallelism feature Use DAG to understand how Spark processes your dataRemove tools or instances if they are not adding any real valuePerform load testing to assess performance and optimize accordingly

Jul 19, 2021

Introducing the New JSON-powered, Block-Style Rich Text Editor

Among all the components of a modern content editor platform, the rich text editor remains an integral part of the content creation process. And for developers, it continues to be a huge pain. As newer channels of consumption emerge, rendering rich-text content across all digital platforms is becoming harder. With the introduction of the new JSON Rich Text Editor (RTE), we intend to make life easier for content managers and developers. Editors no longer have to undergo the hassle of handling HTML source code while editing rich text content. And since the editor returns the saved content in JSON format, developers can easily serialize the data in any human-readable format, paving the way for a truly omnichannel experience for customers. Note: The JSON RTE is available only in the new Contentstack interface.A Delightful, Focused Writing Experience for Content Editors The new JSON RTE field offers a beautiful editor for content managers with all the formatting options and functionalities of our existing HTML-based RTE, minus the HTML code view. This editor frees the content managers from the hassle of handling HTML source code, so they can focus on creating quality content and nothing else. Each element, such as paragraphs, images, lists, etc., is treated as a separate block, so editors can simply drag and drop them within the body text instead of cutting and pasting content. To apply custom styles, editors can choose from the many “properties” that the developer can add to the RTE. The rest of the components are the same as our HTML-based RTE, visually.We Love Our HTML Rich-Text Editor, but... Our HTML-based RTE field is incredible. There is no doubt about it. It provides a single content-editable element for adding HTML markup, allowing web browsers to consume and render content easily. It serves its purpose well, and it works like a charm… until you want to cater to a variety of platforms; or you want to reuse content for a separate frontend framework; or you want to save the data for what it is, without coupling it with the way it should be presented. The content of an HTML-based RTE is stored in raw HTML format, which means it stores content data along with markups for content appearance. This may have specific limitations, especially for cases that require quick and easy backend processing, rendering content on different channels, and evolving faster. Data in this format makes it challenging to write code that can transform rich text quickly or display it in different channels. The New JSON RTE Helps Developers Become Future-ReadyStructured Data for Easy Parsing The new block-style JSON editor stores and returns data in a structured JSON format. Each element (such as a paragraph) is a block; each block is an array; each array has multiple children, storing the actual text and the different nodes associated with the text. This format gives developers more control over the information, ease of performing any backend processing, and the ability to serialize the data in any human-readable format. Data in this format enables you to use any frontend of your choice for a true omnichannel experience: HTML for web browsers, render natively for mobile apps, audio versions for voice assistants, and so on, making your rich-text content truly reusable and future proof, especially for cases such as moving to a new presentation layer or redesigning an app. Let’s look at an example to understand how the new JSON RTE stores and returns rich-text data. If the JSON RTE contains the following text: The field output in the API response will be as follows: { "entry": { "title": "Entry 1", "url": "/entry-1", "tags": [ "first entry" ], "locale": "en-us", "uid": "bltb6ea3a0ab9699748", "created_by": "blt702565fb0d35107f", "updated_by": "blt702565fb0d35107f", "created_at": "2020-11-13T14:15:50.389Z", "updated_at": "2021-07-09T11:41:05.883Z", "ACL": {}, "_version": 2, "_in_progress": false, "json_rte": [ { "uid": "fb1006018d73482aad018eadc8e6818f", "_version": 2, "attrs": {}, "children": [ { "type": "p", "attrs": {}, "uid": "d0efa7610f06413499d09c8f61389d4c", "children": [ { "text": "This RTE is " }, { "text": "amazing", "bold": true }, { "text": "." } ] } ], "type": "doc" } ] } } While you can convert this data back to HTML, we have a serializer package that can help you do that quickly. Excited? Migrate Content From HTML RTE to JSON RTE There are certainly a ton of benefits of moving your content to the new JSON RTE. You can use our CLI to migrate the content of your existing HTML RTE fields to the new JSON RTE fields for all entries of a content type in just a few minutes. Refer to our migration guide to perform the steps, understand the different flags you can use, and migrate the content safely without any data loss.Helpful Resources This blog only scratches the surface of the endless possibilities available using our new JSON RTE field. Read our documentation to understand how to use, configure, and customize this new field to do more with your content. The new JSON RTE field Migrating content from the HTML-based RTE field to the JSON RTE field using CLI

Jul 16, 2021

Kubernetes — Achieving High Availability Through Health Probes and Readiness Gates

Kubernetes, or K8s, is a powerful tool for automating the deployment and management of containers. Due to the incredible scalability, velocity, and ease of management that it brings to your cloud deployments, it’s not hard to see why businesses are adopting it at a rapid pace. Apart from the host of features it offers for your deployments, you can configure it to work as a high-availability tool for your applications. This blog post looks at a couple of features that you can use to ensure that your application is never down and is always available. Container Health Checks Through Probes Probes are periodic health checks that you can perform to determine if the running containers are in a healthy state. There are two different types of probes you can use: a liveness probe and a readiness probe.Liveness Probe In Kubernetes, a pod is the smallest deployable unit of computing that you can create and manage. A liveness probe determines if a pod is alive or not. If it’s alive, Kubernetes does not perform any action. If dead, it will restart the pod. For example, if you have running microservices, and one of the microservices fails to function (maybe, due to a bug), this probe can help bring the pod back to life by restarting it using the default restart policy.Readiness Probe The readiness probe determines if the pod is ready to receive traffic. If not, Kubernetes will not send any traffic to this pod until it is ready. Such a probe is performed throughout the lifecycle of the pod. If the pod needs to be made unavailable for some reason, such as scheduled maintenance or to perform some background tasks, it can be configured to respond to probes with different values. These probes are performed on containers in a pod. Let’s take a look at an example to understand this better. Say you have a pod with a container for each nginx application and a GraphQL application. In this case, nginx will have its own liveness and readiness configuration and GraphQL will have its own configurations. For your nginx app, the config could be as follows: lifecycle: preStop: exec: command: - sh - -c - "sleep 30 && /usr/local/openresty/bin/openresty -s quit" ports: - name: http containerPort: 80 protocol: TCP livenessProbe: tcpSocket: port: 80 initialDelaySeconds: 5 periodSeconds: 1 failureThreshold: 1 readinessProbe: httpGet: path: /health port: 80 initialDelaySeconds: 5 periodSeconds: 1 failureThreshold: 1 We have tweaked the default values of some parameters. Let’s look at the meaning of some of these: tcpSocket: port: 80 If tcpSocket is found open at port 80, it is considered a success for livenessProbe. initialDelaySeconds: 5 The probe executes 5 seconds after the container starts. periodSeconds: 1 The probe executes after every one second. failureThreshold: 1 If the probe fails one time, it will mark the container as unhealthy or not ready. httpGet: path: /health port: 80 The pod IP connects to /health at port 80 for readinessProbe. For GraphQL, you can use: lifecycle: preStop: exec: command: - sh - -c - "sleep 20 && kill 1" The preStop hook above is a container hook called immediately before terminating a container. It works in conjunction with a parameter called terminationGracePeriodSeconds. NOTE: The terminationGracePeriodSeconds parameter applies to a pod. So, after executing this parameter, the pod will start terminating. You need to set this value keeping in mind the container executes preStop successfully. To add some context, in most applications, a “probe” is an HTTP endpoint. If the endpoint returns a status code from 200 to 399, the probe is successful. Anything else is considered a failure.Readiness Gates to Handle Unexpected Issues The above probes help determine if your pods are healthy and can receive traffic. However, it is likely that the infrastructure responsible for delivering traffic to your pod is not ready. It may be due to reasons such as network policies or load balancers taking more time than expected. Let’s look at an example to understand how this might happen. Suppose you have a GraphQL application running on two pods, and you want to restart the deployment. Load balancers expose these pods to the outside world, so they need to be registered in the load balancer target group. When you execute #kubectl rollout restart deploy/graphql to restart, the new pods will start the cycle until they are in a running state. When this happens, Kubernetes will start terminating old pods, irrespective of whether the new pods are registered in the load balancers and are ready to send and receive the traffic. An effective strategy to bridge such a gap is to use Pod Readiness Gates. It is a parameter to define if the new pods are registered with the load balancers and are healthy to receive traffic. It gets feedback from the load balancers and informs the upgrade manager that the new pods are registered in the target group and are ready to go. You can use helm to install and configure the GraphQL application for the above example, writing readinessGates in values.yaml.Pod Configuration for readinessGates: To implement the above, add readiness gate to your pod as follows: conditionType: target-health.alb.ingress.k8s.aws/<ingress name>_<service name>_<service port>. podAnnotations: {} podSecurityContext: {} # fsGroup: 2000 securityContext: {} # capabilities: # drop: # - ALL # readOnlyRootFilesystem: true # runAsNonRoot: true region: "" readinessgates: target-health.alb.ingress.k8s.aws/cs-ingress_dev8-graphql_80 command: args1: "node" args2: "server.js" ports: #node containerport containerport: 9000 env1: name: NODE_ENV value: "development" service: type: NodePort port: 80 targetport: 80 Once done, verify the status of readinessGates with the following command: kubectl get pod -o wide Using health probes on containers enhances the application’s performance and reliability, while readiness gates on pods ensure that it is ready before accepting any traffic. Having health probes and readiness gates configured can ensure that your app is always available without downtime.

Jul 02, 2021

Principles of Effective RESTful API Design

The RESTful style of API has been around for more than 20 years. It is one of the most common types of services that allow clients — including browser applications, mobile and IoT devices — to talk to the servers. If you are in the process of developing an app and have reached a stage where you are ready to create public APIs, it is worth pausing to ensure that you are on the right track. It’s difficult to make drastic changes to your APIs once they are out. So it makes sense to get as much right as possible from the very beginning. And since these APIs would form the core of your application, they should be:Secure, fast, and flexible for easy scalingBuilt using common and widely accepted standardsEasily understandable and consumable, enabling quick integrationSupported with good documentation that explains semantics and syntax Contentstack serves billions of API requests every month without any glitches, and it’s flexible enough to scale 10x the current volume in a flash. This blog post shares some of the best practices that we follow while developing our API. These tips should be helpful to anyone starting with building REST APIs. Make Your APIs Secure When developing APIs, security should come first. This is a must. Since you can execute an API call from the Internet, requests can come from anywhere. When it comes to API security, there is no reason not to use encryption. Use SSL/TLS, always. SSL security is a very straightforward and least expensive method to make the request and response encrypted. TLS is a cryptographic protocol designed to provide secure, encrypted communication over a computer network. It encrypts data between an API client and an API server, preventing data from being read if intercepted between point A and point B. TLS ensures encryption of data in transit. Another potential security threat could be long-lived authentication or authorization tokens for APIs. A best practice is to make them short-lived. You can do this by having custom API key management using low-overhead implementation protocols like OAuth or JWT. Short-lived API tokens are much easier to use and significantly more secure.Define Requests Clearly It all starts with defining your requests in a way that makes your API easy to use, reduces ambiguity, and brings some consistency. Let’s look at some of the ways that can help you achieve that. Follow request path conventionsMake use of resource names Your request path should have the name of the resource with which the API is going to interact. For example, if your app provides “products,” use “/products” in the API request as a noun. Avoid using verbs with resource names in the request (e.g., /api/create-products) since the request’s type should define the verb, as explained below.Use HTTP methods Most developers are familiar with the typical HTTP methods such as GET, POST, PUT, and DELETE. These are the type of common API requests developers make on the resources. So, something like “GET /products” indicates that the request is to fetch products.Use plural forms You can use plural forms for all resource names. For example, “/products” can be used to fetch all products, whereas “/products/20” is used to fetch a single product with an ID of “20.” Plural forms make your requests more consistent and intuitive.Use nested hierarchy You can provide a structured hierarchy for nested or related resources, so it’s easier for a larger group to work on a specific item or sub-item. But keep the depth level to a minimum. For example, for products that have reviews and ratings associated with them, you can define the relationship as follows:GET /products/:product_id/reviewsPOST /products/:product_id/reviewsPUT /products/:product_id/reviews/:review_id Most of the cases satisfy the resource mapping with a path. Still, certain exceptions are relevant to functionality and do not have any resources associated with it directly, such as /search or /bulk-actions. We can define such a request with its associated actions.Use Standard Exchange Format REST can use many exchange formats such as plain text, XML, CSV, etc. But go with JSON. JSON is a lightweight data format, allowing for faster encoding and decoding on the server-side. It can be easily consumed by different channels such as browsers, mobile devices, IoT devices, etc., is available in many technologies and is now a standard for most developers. To ensure that your API uses JSON, use “Content-type”: “application/json” in request and response. For the request body fields, use consistent casing, such as lowercase (recommended). For other non-textual formats, use “multipart/form-data,” which you can use for sending files over HTTP. While it also allows sending textual or numerical data, restrict its usage to sending files; use JSON for textual data. For this format, you need to use “Content-type”: “multipart/form-data” in the request header. Response header may vary based on the type of file it receives (e.g., images, application, PDFs, and documents).Provide Standard Headers Requests headers are used to transfer additional information from clients. Follow the standard headers to share information like content-type, basic authentication, content-length, accept, accept-encoding, and user-agent. We recommend transferring the security and authentication parameters, such as a user token or secret tokens, into the headers.Version Your APIs As you start introducing more features and update existing ones, you may have to make minor or major changes to your APIs. Changes are inevitable as you grow. The best way to maintain these changes is through versioning. Versioning your APIs ensures that you continue to support older APIs and release newer APIs systematically. It also prevents users from hitting invalid URLs. Each change should be versioned, and your APIs should be backward compatible. While doing this, you must ensure users have a way to migrate to the latest version. For significant changes, the version should be in the request URL. Examples: GET /v1/products - Version 1 of productsGET /v2/products - Version 2 of products For minor changes or fixes, pass the versions in the header, where you can use “date” to track the changes, as shown in the following example: GET /v1/products Header: Accept-Version: “2019-11-01”Offer Ways to Filter, Paginate, Sort, and SearchFiltering: Design your APIs to filter or query the stored data via specific parameters. For example, users may want to filter products by tags, categories, or price range. You can provide filters like this: /v1/products?query={tags: ["tag1", "tag2"]}.Pagination: Provide the ability to paginate responses so that users can request just what is required. For example, a mobile user may want the first five entries to show on the homepage, whereas web users may want 25 entries in a single request. There are various ways to provide pagination. The more flexibility you offer, the better it will be for users. The “skip,” “limit,” and “per_page” are some of the most common parameters that provide enough flexibility for paginations. It is, however, also important to define certain restrictions—for example, set a max limit on items that can be fetched “per page”—so the users do not exploit your services. /v1/products?skip=100&limit=10&per_page=10Sorting: It provides the ability to get the list of items in the desired order. You can provide options to sort in ascending and descending order based on values of certain fields, such as updated_at, price, etc. The users can then combine these options for more flexibility. For example /v1/products?sort=-price,updated_at.Searching: Searching is different from filtering or passing basic queries. It should give the ability to perform a full-text search, helping users find relevant results faster. So, for example, when the user runs this /v1/products?text={{search}}, your API should check for relevant terms in the values of all the fields.Field Selections Your APIs should provide your users the flexibility to fetch only the required details instead of everything. No over-fetching or under-fetching. Field selections are helpful for low-computing devices (such as mobile or IoT) as it allows them to save on bandwidth and improve the overall network transmission speed. You can do this by offering ways within the request to include or exclude data of specific fields. For example: /v1/products?skip_fields[]=created_at,updated_at and /v1/products?select_fields[]=title,price,color Also, you can extend this further by offering ways to include details of other related items in the single request to save on the network traffic. For example: /v1/products?include=categories,colors,brands&select_fields[categories]=title,price,colorStructure Responses the Right Way Your API requests need to have an appropriate response. You can achieve this by following certain standard practices. Here are some of the important ones. Response Status Codes Use the standard HTTP status codes in the API responses to inform the users about success or failure. While there are broad categories for these codes, each code has a meaning and should indicate the exact definition. Here are the categories of codes that you can use: Informational responses (100–199)Successful responses (200–299)Redirects (300–399)Client errors (400–499)Server errors (500–599) You can find more details on status codes here: https://developer.mozilla.org/en-US/docs/Web/HTTP/Status. Response Body Request errors should deliver proper error messages so that the user is informed about the exact issue. Attach each resource with its system-provided metadata, such as created_at, updated_at, created_by, and updated_by. Each successful response should be wrapped via its proper wrapper according to the path, to include the additional meta details about the response. Example: GET /v1/users?limit=10 { “users”: [{ first_name: “Amar”, last_name: “Akbar”, … }, { first_name: “Alpha”, last_name: “Beta”, … }], “skip”: 0, “limit”: 10, }Headers Response headers play an essential role in providing information about the server’s behavior to respond to the request. Make sure your headers support the following before you make your APIs public: RequestID This response header helps debug the specific request instead of asking the customer different questions such as the request type, time, etc. It can trace down the entire request lifecycle on the server. You can use the “X-Request-Id: {{TRACE_ID}}” header in the response.CORS Your APIs should have support for CORS if they are available for public consumption. CORS (cross-origin resource sharing) is an HTTP-header-based mechanism that allows a server to indicate any other origins than its own from which a browser should permit the loading of resources. If your APIs are not consumable publicly or are restricted, it is best to keep them closed by providing restricted CORS values. You can find more information about CORS here: https://developer.mozilla.org/en-US/docs/Web/HTTP/CORSRate Limit To ensure your system is secure from unwanted attacks and abuse, add rate limiting to your APIs. Rate limiting restricts users from making more than the defined requests in a given time period. While implementing this, a good practice is to provide a way for users to know about the time limits through the following headers.X-RateLimit: Max requests allowed in a given time periodX-RateLimit-Remaining: Requests remaining in a given time periodX-RateLimit-Reset: Time remaining in the current periodCompression Your API should support all types of compression on the response. This compression saves a lot of time for the network transmission of the data. Some of the popular compression types are gzip and brotli.Cache Headers Cache headers are mostly provided on GET calls to give information about the behavior of the request, more specifically to determine if the request was served from the cache or the origin server. You can use the following header for this: X-Cache: “HIT/MISS.” Let’s look at some of the other headers you can use:Cache-control This header allows the client to keep the resource stored on their end and not make requests until the cache is cleared or expired.Etags When generating a response, include an HTTP header Etag containing a hash or checksum of the representation. If an inbound HTTP request contains an If-None-Match header with a matching Etag value, the API should return a 304 Not Modified status code instead of the output representation of the resource.Have Adequate Documentation and Tools in Place Most developers would prefer reading the documentation before trying out your APIs or attempting any integration. It is, therefore, crucial to make your docs publicly available and easy to understand. There are tools like Swagger that help you generate the documentation API during the development phase. Ensure that your docs record change logs, version details, and deprecation notices whenever required. You can provide examples of requests and responses so users know what to expect. You can also provide Postman collections that allow users to configure the environment and try out the API requests through their accounts to make things easier. Since APIs are used by developers, it is important to follow good design practices to deliver a well-designed and effective developer experience.

Jun 21, 2021

Micro-Frontend Approach for Enterprise-Grade React Applications

Microservices have revolutionized the way we develop modern apps. And that’s primarily because of the huge advantage they bring along: Innovate faster by breaking down large products or projects into smaller, manageable pieces that you can move, update, or replace quickly. We have built our award-winning products based on this approach, and it has helped us scale incredibly fast. That’s the story of our backend. Our frontend, however, until some time ago, was still monolithic. We wanted to take a similar approach for our frontend to get the benefits of “microservices” there. This approach is called a “micro-frontend.” In the last few months, we tried and tested a few methods, and finally, we successfully implemented a micro-frontend for our application. We learned quite a few crucial lessons in the process. This blog is about these lessons. It’s not a theoretical introduction to micro-frontend nor a definitive guide to implementing it. It’s more about our journey of building micro-fronted for our applications, some lessons we learned, and sample code to help you build yours. Why We Chose Micro-Frontend A single, unified frontend works well until your app is catering to only a handful of customers. As you grow and start adding more capabilities, the inflexibility of a single, large frontend starts to throttle the pace of your delivery. We have broken down our product into several sub-domains, with each one being developed and owned by separate teams. When the different codebases from all these teams came together into a single application, there were natural consequences:All the teams had to sync often on deployment and testingReleases needed to be coordinated across the different teams and their schedulesMerge conflicts occurred frequently It was not an efficient approach and certainly not scalable. Dividing the frontend into smaller apps was the need of the hour. The Approach There are several ways to build micro-frontends. We did some initial research and found the following two ways to be more suitable for our requirements: IframesApproach as defined in https://micro-frontends.org/ We tried the “iframes” approach initially. It was, however, not a very effective approach, as it has issues related to security, usability, and SEO. However, it helped us bootstrap a repository and get the micro-frontend on the screen to start development. The second approach, as suggested by micro-frontends.org, was interesting because it leverages custom DOM elements. While the approach is well thought out, we realized that it could be overkill for us since we don’t use multiple frameworks — we use ReactJS across projects — and we don’t have many use cases for out-of-the-frame rendered components. Later, we came across the blog: 5 Steps to Turn a Random React Application Into a Micro Front-End. It suggests a few methods to help implement a micro-frontend the way we wanted, and they eventually became the base for our implementation. Let’s look at the methods in detail. The Implementation As defined in the blog, the basic implementation includes using a dynamic script tag to load the micro-frontend JavaScript on the page. Once loaded, the container calls a method on the window object. The micro-frontend provides this method to start the rendering process. The micro-frontend renders to a div as exposed by the container. Eventually, you get a setup that looks like the illustration below. The container app renders sections in green, and the section in blue is the micro-frontend.Challenges with rendering outside the micro-frontend — and the solution While the initial success using the above approach was quite encouraging, it was short-lived. We faced a significant challenge, and we quickly realized that our implementation needed to be slightly different: As you can see in the illustration above, the micro-frontend also needed to render a few components outside its DOM hierarchy. The left navigation bar and the top header need to render certain buttons, icons, and other components sourced from the micro-frontend. After some basic troubleshooting, we realized that we could use the React Portals API to solve this. This API allows you to render components outside the DOM tree where the micro-frontend is rendered. Using some well-known div IDs, we were able to expose an API where the micro-frontend can control these sections in the UI. Another challenge was to have a seamless routing experience when the micro-frontend wanted to change routes. Managing this became a little easier since we are creating a single-page application (SPA) with HTML 5 pushState API. When initializing the micro-frontend, we could pass the history object, which the micro-frontend can then use. Also, we used relative routes, so it worked without a hitch.Challenges With Rendering Outside the Micro-Frontend — and the Solution One unfortunate side effect to having independent micro-frontends is the relative size of each micro-frontend JavaScript bundle. Nowadays, loading an average ReactJS app and all the standard libraries consumes several megabytes of bandwidth. By implementing micro-frontends, we were going to multiply this with the number of micro-frontends in use. Some of the libraries that we use (such as React, Redux, Redux-Saga) are common among all the micro-frontends we have built. So, it did not make sense to download these common libraries for each micro-frontend. Instead, a better approach would be to bundle them together so you could download them only once. We achieved this using dll-plugin, a library for Webpack. It allows us to bundle the common libraries into a “DLL” (Dynamically-Linked-Library) that you can dynamically load within a webpack project. By using the DLL plugin, we achieved a significant 60% reduction in our bundle size.Summary Our application’s new user interface and some of our new incubation projects now have micro-frontends implemented, based on the approach given above. And all these apps have been working smoothly. We can roll out more changes quickly with less dependency on other teams, so deployment and scaling become more manageable. If you want the details of this approach, check out our micro-frontend example (with code) on GitHub.

Dec 23, 2020

5 Tips to Ensure Uptime During the Holidays

The most wonderful time of the year is here again! It’s the holidays, a time when most people typically spend days with family and friends, do things they love the most, and shop a lot. With the Covid-19 pandemic, people are shopping online more than ever. Salesforce reports that Global digital sales grew 45% year over year. This rise translates into high traffic for ecommerce websites. As a CMS admin, you can define systems and processes that can help you quickly manage traffic spikes without impacting your website or app’s availability, thereby providing a seamless, enjoyable experience to your clients. At Contentstack, we have been doing this for years, and we’re glad to share the five most important tips to achieve this. Tip 1: Forecast Expected Traffic and Load It is impossible to determine the exact traffic that your clients’ apps are likely to receive and the load your servers may need to handle. But it helps to get an estimate, which helps in planning and allocating resources. Before the holiday season, talk to all your customers to understand their plans and expected traffic. Parallelly, check historical data, and analyze patterns. These two sources can give you an idea of what to expect for the holidays. Now, add some buffer to this estimation to define an upper limit. For example, if you determine that you are likely to get 2.5x requests per second, set the upper limit to 4x or 5x. This data can serve as a baseline for your resource and capacity planning. Tip 2: Plan Capacity and Resources With a tentative idea of the expected traffic, it becomes easier to determine the additional resources and infrastructure you’ll need. Once you have a baseline, add the required infrastructure accordingly. This baseline also helps define the processes and devise the plan to serve the expected load. Additionally, to help you prepare for the increased traffic, you can set a mechanism that automatically scales the infrastructure when the traffic reaches an upper threshold (say, 70% of your expected baseline). This ensures that you are ready for dramatic increases. Tip 3: Run Performance and Load Tests Now that you have a baseline for the expected requests and the required infrastructure in place, it’s time to test if you can carry the load. Try to simulate the actual holiday season traffic by making requests close to or over your expected baseline. This simulation helps in checking how the infrastructure performs and how it auto-scales. It will also help identify gaps in your current setup so you can address them before the actual traffic comes in. Rigorous testing aims to ensure that you operate smoothly even at 4x or 5x the expected traffic and still have a buffer for more. Tip 4: Monitor Services 24/7 Monitoring how your resources are consumed helps identify patterns or unexpected behaviors in realtime. You can set up an internal, custom dashboard that monitors all your critical systems and services’ real-time usage. This performance dashboard should be available to all the internal teams responsible for managing uptime and communication (such as the API developers, testers, and customer support engineers). Make sure that a set of team members are continuously monitoring the performance. Additionally, you can configure alerts that notify you when anything needs attention so that no unexpected behavior goes unnoticed. Here at Contentstack, we monitor our services before the holiday season and after the season ends. Tip 5: Conduct Mock Drills With adequate capacity planning, rigorous load testing, and constant monitoring, you are well prepared for the holidays. But unforeseen things can happen, and you would want to be ready for those issues as well. There are always chances of network degradation, dependent service’s unavailability, or some random system failure. To make sure you are prepared to handle any surprise contingencies, conduct mock drills to create situations that simulate network or service failures. These drills can help you identify, analyze, and fix issues in the shortest time possible. And with every exercise, you get better and faster. Bonus Tip: Set up “War Rooms” A “war room” is a virtual or physical command room or control room where key stakeholders can get together instantly in case of any emergency. In the past, you would set up a war room in your office. Today, with many of us in lockdown, you can use a common collaboration platform (such as Slack or Microsoft Teams) to talk to anyone irrespective of our team members’ location or time zone. And invite people to join only in war-like situations. War rooms ensure that you don’t waste time on email or rely on phone communication and focus on instant action. Since people play an essential role in achieving all the above steps, make sure you have people working in all shifts, dedicated to monitoring and testing throughout the holiday season, so there is never a time when you don’t have the required resources. These tips will ensure that you maintain 100% uptime, so your clients can make the holiday season a pleasant and memorable experience for their customers.

Nov 25, 2020

Build a Technical Moat with JAMstack

A “moat” is a pit or trench surrounding a castle, usually filled with water. Historically, moats gave the castle residents a tactical advantage. A moat helps secure the premises acting as a line of defense to become less vulnerable to external attacks, especially during war times. Today’s consumer-driven, competitive world is nothing less than a war, and technology is the new battlefield. Consumers are accessing digital content like never before, and they demand a superior, seamless experience for every interaction. In today’s market, any enterprise without a strong technical “moat” to ensure it has a competitive advantage over its competitors runs the risk of losing customers. While there are several ways to develop a moat, one of the quickest ways is to have a reliable, robust web app or website. Yes, you read that right. Your website can be your technical “moat” if implemented the right way: the “JAMstack” way, using a headless CMS, such as Contentstack, and a static-site generator, such as Gatsby. We will talk about JAMstack in a while, but first, a bit on why your website architecture matters and how it can be your technical “moat.” A business’ website serves as a gateway to new customers. However, even simple website issues, such as a long load time, could cost companies a lot. For example, the BBC loses about 10% of users for every additional second the site takes to load. About 53% of your mobile users exit your site if it takes more than three seconds to load. That’s a lot if your sales funnel relies heavily on incoming website traffic. And it’s not just the site speed but the overall experience that matters. Consumers expect websites to deliver personalized content, straightforward translation, smooth video streaming, and a similar, seamless experience over all devices. Any gap in meeting these expectations translates into direct revenue loss. Poor online experiences end up costing companies to lose about 24% of online revenue. Your website matters a lot. And just by making it right, you can build a strong moat. But how is JAMstack the answer to all this? And what’s JAMstack anyway? Understanding JAMstack The “JAM” in JAMstack represents JavaScript, API, and Markup. It is a modern architecture for building websites and web apps. It helps build apps that are fast, secure, and scalable (at a low cost), making lives a little easier for developers. It makes this possible by handling some of the functionalities on the client-side and letting APIs lift the heavy burden of other functionalities, allowing developers to plug in services of their choice with ease. What Should Be Your Stack? Let’s look at the best tools that you can pick up to create your own JAMstack. For JavaScript, you can use vanilla JavaScript or any front-end framework or library of your choice. However, what you choose for API and Markup makes all the difference.API - Contentstack Headless CMS API (A) is the most critical part of JAMstack. It’s easy to plug in multiple, best-in-class third-party apps via APIs to manage different functions of your apps. But the flip side is that you can end up bundling up several services, thereby increasing your app’s cost and maintenance. Instead, it makes more sense to use an API-based content management system, such as Contentstack, that offers several other functionalities that you may need —apart from content delivery and content management —and then include other services for the other additional services required. Contentstack offers content delivery and GraphQL APIs for delivering content to any device or platform of your choice. It also has a built-in CDN for Content Management APIs for managing content efficiently. It also has Image Delivery APIs you can use to retrieve and manipulate images on the go. Contentstack allows you to set up webhooks on specific events, triggering actions in dependent services, such as Netlify, Gatsby-cloud, or build CI/CD systems easily. It also provides other features such as multilingual and multi-environment support, users and roles management, content workflows, and extensions, and more. Lastly, the headless nature of Contentstack allows you to create content once and reuse it across multiple channels, including web apps, IoT devices, wearables, and any other channel to which content can be delivered using APIs. It future proofs your technology and content. Markup - Gatsby Static-site Generator In JAMstack, Markup is generally applied as a templating engine for building pages. It is essential to connect markup languages with templating that should be pre-built during deployment. For example, static site generators include a build process to put JavaScript, APIs, and Markup together. Gatsby is a free, React-based, open-source framework that helps developers build blazing-fast websites using JAMstack. You get all the React framework benefits, such as faster DOM rendering, a supportive community, ease of learning, and so on. Gatsby has a wide range of data source plugins that you can choose from to build your website from different source nodes. You can get data from any third-party APIs/SaaS service, headless CMS, database, or markdown files into your pages using GraphQL API, which can act as a query language for your data. Sites built with Gatsby are easily scalable as there are no complicated deployments with databases and servers. This eventually reduces the time, complexity, and cost spent in maintaining the apps. Builds generated by Gatsby can be served by any CDN service, allowing you to scale at a minimal cost. Gatsby uses the latest web technologies, such as Webpack, React.js, CSS, modern JavaScript, and more to keep your site up-to-date with the most modern technologies. It is also possible to develop PWA (Progressive Web Apps) with Gatsby, which is the future of the web on mobiles and other devices.How Does This Translate Into a Technical Moat? While JAMstack offers many advantages, the JAMstack model combining JavaScript, Contentstack, and Gatsby is the one that helps you build the moat. This model allows you to build blazing fast web apps and deliver superior performance and scalability at a low cost. The decoupled architecture provides enough flexibility to remove or replace any component without impacting other components. Contentstack provides all the APIs you need for superior content management and an effortless content editor experience. Gatsby ensures that your apps are fast, using the latest technologies, paving the way to accommodate future needs.Next Steps We have made it easier for you to build a technical moat. Check out this starter app that uses Contentstack and Gatsby, enabling you to build static sites in minutes.

Oct 19, 2020

The Benefits of Contentstack’s CDN

For most web properties, delivering content fast is of the utmost importance. That’s why Content Delivery Networks (CDNs) exist. A CDN is a network of caching servers scattered across the globe, caching content regularly and delivering cached content to nearby requests. This improves the content delivery and page load time dramatically since the request does not need to travel to the origin server. Read more about CDN. Most website owners consider setting up a CDN. But with Contentstack as your CMS, you don’t need to worry about setting up any caching mechanism since Contenstack comes with a CDN. Lightning-fast content delivery comes by default. Let’s look at some of the other CDN benefits that you get with Contentstack.Leverage our CDN to serve high traffic We have partnered with a modern CDN provider with more than 60 cache servers worldwide, capable of delivering cached content in a fraction of a second. Additionally, there is no maximum limit to the number of GET requests per second you can make to the CDN. This means you do not need any caching mechanism or CDN, even if your website gets incredibly high traffic per day. In cases when Contentstack is down, which doesn’t happen, the CDN servers will continue to deliver cached content to the visitors.Keep your content fresh with Cache Purging Purging refers to the removal of cached content from the CDN servers. Your cached content remains in the Contentstack CDN servers for up to one year, after which it is cleared automatically. However, whenever any content is published, unpublished, or deleted, it purges the changed content (and some related content such as referred entries and assets) from the CDN servers instantly. Read about how cache purging works with Contentstack. After the cache is purged, when making subsequent page requests, the CDN fetches new content from the origin server, delivers to the requester, and saves the updated cache. This process ensures that your website visitors always get updated, fresh content. If an asset or image is updated, the cache is not purged. The updated asset or image gets a new URL, and the old image or asset would be available at the previous URL for one year. Near Realtime Cache Purging Cache purging happens in realtime. When any new content is published, unpublished, or deleted, the cache is purged instantly. So, subsequent requests are updated from the origin server. Cache Optimized for Fewer Requests to the Origin Server Contentstack’s intelligent caching mechanism purges the cache of only the content that has been changed (published, unpublished, or deleted) and other related content (e.g., referenced entries and assets) from the CDN servers. Also, the cache of only that specific locale and environments are purged. Purging happens for only the cached items that are changed. The cache of other unchanged content remains intact. This translates into a lower number of requests to the origin server. You can refer to our Cache Purging Scenarios doc for more details. Maintenance-free and Economical Having a CDN within your CMS means you don’t have to worry about choosing a CDN from the hundreds of available options, setting it up, and maintaining it forever. Nor do you have to worry about the CDN being compatible with the CMS or defining cache purging rules, etc. It’s all set up and ready to use. Additionally, this is much more cost-effective than another third-party CDN, even if you use and pay for over-usage with Contentstack. From the above points, it is undoubtedly clear that you do not need a separate CDN for your web properties if you use Contentstack as your CMS. However, if you still want to set up a separate CDN (or if you already have one), there are certain things to keep in mind when using Contentstack. Using other CDNs is a topic that we will cover in our follow-up CDN blog post.

Oct 06, 2020

Elasticsearch: Working With Dynamic Schemas the Right Way

Elasticsearch is an incredibly powerful search engine. However, to fully utilize its strength, it’s important to get the mapping of documents right. In Elasticsearch, mapping refers to the process of defining how the documents, along with their fields, are stored and indexed. This article dives into the two types of schemas (strict and dynamic) that you usually encounter when dealing with different types of documents. Additionally, we look at some common but useful best practices for working with the dynamic schema so that you get accurate results for even the most complex queries. If you are new to Elasticsearch, we recommend reading and understanding the related terms and concepts before starting. Schema Types, Their Mapping, and Best Practices Depending on the type of application that you are using Elasticsearch for, the documents could have a strict schema or a dynamic schema. Let’s look at the definition and examples of each, and learn more about their mapping. Strict Schema - The Simple Way A strict schema is where the schema follows a rigid format, with a predefined set of fields and their respective data types. For example, systems like logs, analytics, application performance systems (APMs), etc. have strict schema formats. With such schemas, you know that all the index documents have a known data structure, which makes it easier to load the data in Elasticsearch and get accurate results for queries. Let’s look at an example to understand it better. The following snippet shows the data of a log entry within Nginx. { "date": "2019-01-01T12:10:30Z", "method": "POST", "user_agent": "Postman", "status": 201, "client_ip": "0.0.0.0", "url": "/api/users" } All the log entries within Nginx use the same data structure. The fields and data types are known so it becomes easy to add these specific fields to Elasticsearch, as shown below. { "mappings": { "properties": { "date": { "type": "date" }, "method": { "type": "keyword" }, "user_agent": { "type": "text" }, "status": { "type": "long" }, "client_ip": { "type": "IP" }, "url": { "type": "text" } } } } Defining the fields, as shown above, makes it easy for Elasticsearch to get the relevant results for any query. Non-Strict Schema Challenges and How to Overcome Them There are several applications where the schema of the documents is not fixed and varies a lot. An apt example would be the various structures that you define in a content management system (CMS). Different types of pages (for example navigation, home page, products) may have different fields and data types. In such cases, if you don’t provide any mapping specifications, Elasticsearch has the ability to identify new fields and generate mapping dynamically. While this, in general, is a great ability, it may often lead to unexpected results. Here’s why: When documents have a nested JSON schema, Elasticsearch’s dynamic mapping does not identify inner objects. It flattens the hierarchical objects into a single list of field and value pairs. So, for example, if the document has the following data: { "group" : "participants", "user" : [ { "first" : "John", "last" : "Doe" }, { "first" : "Rosy", "last" : "Woods" } ] } In such a case, the relation between “Rosy” and “Woods” is lost. And for a query that requests for “Rosy AND Woods,” it will actually throw a result, which, in reality, does not exist. So, What’s the Solution to This? The best way to avoid such flat storage and inaccurate query results is to use nested data type for fields. The nested type is a specialised version of the object data type that allows arrays of objects to be indexed in a way that they can be queried independently of each other. This makes sure that the relation between the objects, if any, is maintained, and the query would return accurate results. The following example shows how you can add a generic schema for all pages of a CMS application. { "mappings": { "properties": { "doc_type": { "type": "keyword" }, "doc_id": { "type": "long" }, "fields": { "type": "nested", // important data type "properties": { "field_uid": { "type": "keyword" }, "value": { "type": "text", “fields”: { “raw”: { “type”: “keyword” } } } } } } } } Now let’s look at a couple of examples where different types of input objects can be ingested into a single type of index. Example data 1: { "first_name": "ABC", "last_name": "BCD", "city": "XYZ", "address": "Flat no 1, Dummy Apartment, Nearest landmark", "country": "India" } You can convert this data into Elasticsearch mapping, as shown below: { "doc_type": "user", "doc_id": 500001, "fields": [{ "field_uid": "first_name", "value": "ABC" },{ "field_uid": "last_name", "value": "BCD" },{ "field_uid": "city", "value": "XYZ" },{ "field_uid": "address", "value": "Flat no 1, Dummy Apartment, Nearest landmark" },{ "field_uid": "country", "value": "India" }] } Example data 2: { "title": "ABC Product", "product_code": "PRODUC_001", "description": "Above product description colors, sizes and prices", "SKU": "123123123123", "colors": ["a", "b", "c"], "category": "travel" } { "doc_type": "product", "doc_id": 100001, "fields": [{ "field_uid": "title", "value": "ABC Product" },{ "field_uid": "product_code", "value": "PRODUC_001" },{ "field_uid": "description", "value": "Above product description colors, sizes and prices" },{ "field_uid": "SKU", "value": "123123123123" },{ "field_uid": "colors", "value": ["a", "b", "c"] },{ "field_uid": "category", "value": "travel" }] } This type of mapping makes it easier to perform a search on multiple types of documents within an index. For example, let’s try to search for users where "country" is set to "India" AND for products where "category" is set to "travel." GET /{{INDEX_NAME}}/search { "query": { "nested": { "path": "fields", "query": { "bool": { "should": [ { "bool": { "must": [ { "match": { "fields.field_uid": "country" } }, { "match": { "fields.value": "India" } } ] } }, { "bool": { "must": [ { "match": { "fields.field_uid": "category" } }, { "match": { "fields.value": "travel" } } ] } } ] } } } } } In Conclusion If you are certain that your documents follow a strict schema, you don’t need to structure your data in a nested data type format. Follow the pattern shown in the “Strict Schema” section to input your data in Elasticsearch. However, suppose your documents are not likely to follow a strict schema. In that case, we highly recommended that you store the data in a nested format, which helps you consolidate all types of documents under a single index roof with uniform mapping.

Sep 10, 2020

Augmented Reality Frameworks for an Enterprise Web-Based AR Application

How do you create augmented reality? In the process of building an Augmented Reality proof of concept in under 4 weeks (see details here), the team at Valtech evaluated a series of AR frameworks and software development kits (SDKs) that would enable them to rapidly pull in data from a headless CMS (Contentstack) and display it in an Augmented Reality interface on a phone or tablet web browser. Here is their quick research report. For total beginners to AR (like me), an AR framework is the SDK to merge the digital world on-screen with the physical world in real-life. AR frameworks generally work with a graphics library, bundling a few different technologies under the hood — a vision library that tracks markers, images, or objects in the camera; a lot of math to make points in the camera registered to 3D space — and then hooks to a graphics library to render things on top of the camera view. Which software is best for our web-based Augmented Reality use case? The key considerations for the research were: Speed. The goal was to create a working prototype as fast as possible. Once we were successfully displaying content and had completed an MVP, we could continue testing more advanced methods of object detection and trackingTraining custom modelsIdentifying and distinguishing objects without explicit markersPotentially using OCR as a way to identify product namesMore of a wow-factor The team was agnostic on whether to work with marker or image-tracking -- willing to use whichever was most feasible for our use case. Object tracking - Since the team was not trying to place objects on a real-world plane (like a floor), they realized they may not need all the features of a native iOS or Android AR library (aside from marker tracking) Content display. That said, the framework needed to allow for content to be displayed in a cool and engaging way, even if we didn’t achieve fancy detection methods in 3 weeksSomething more dynamic than just billboarded text on videoMaybe some subtle animation touches to emphasize the 3D experience (e.g. very light Perlin movement in z plane) Platform. The preference was for a web-based build (not requiring an app installation) Comparing the available AR Frameworks: Marker tracking, object tracking, and platform-readiness Here's an overview of our AR / ML library research notes: AR.js Uses Vuforia*Cross-browser & lightweightProbably the least-effort way to get startedOffers both marker & image tracking. Image tracking uses NFT markers.Platforms: Web (works with Three.js or A-Frame.js) Zappar WebAR Has SDK for Three.js.SDK seems free; content creation tools are paidImage tracking onlyPlatforms: Web (Three.js / A-Frame / vanilla JS); Unity; C++ ARKit Not web-basedImage tracking is straightforward, but can’t distinguish between two similar labels with different textOffers both marker & image trackingPlatforms: iOS Argon.js Uses Vuforia Has a complex absolute coordinate system that must be translated into graphics coordinates. No Github updates since 2017. Offers both marker & image trackingPlatforms: Works in Argon4 browser Web XR Primarily for interacting with specialized AR/VR hardware (headsets, etc.) XR.plus Primarily an AR content publishing tool to create 3D scenes Google MediaPipe (KNIFT) Uses template images to match objects in different orientations (allows for perspective distortion.) You can learn more here.Marker and image tracking: Yes, sort of...even better. KNIFT is an advanced machine learning model that does NFT (Natural Feature Tracking), or image tracking -- the same as AR.js does, but much better and faster. It doesn't have explicit fiducial markers tracking, but markers are high-contrast simplified images, so it would handle them well, too. Platforms: Just Android so far, doesn't seem to have been ported to iOS or Web yet Google Vision API - product search Create a set of product images, match a reference image to find the closest match in the set. Cloud-based. May or may not work sufficiently in real-time?Image classificationPlatforms: Mobile / web Google AutoML (Also option for video-based object tracking) Train your own models to classify images according to custom labelsImage classificationPlatforms: Any Ml5.js Friendly ML library for the web. Experimented with some samples that used pre-trained models for object detection. Was able to identify “bottles” and track their position.Object detectionPlatforms: Web p5xr AR add-on for p5. Uses WebXR.Platforms: Seems geared towards VR / Cardboard * Vuforia is an API that is popular among a lot of AR apps for image / object tracking. Their tracking technology is widely used in apps and games, but is rivaled by modern computer vision APIs - from Google, for example Graphics Library Research Under the hood, browsers usually use WebGL to render 3D to a <canvas> element, but there are several popular graphics libraries that make writing WebGL code easier. Here's what we found in our graphics library research: Three.js WebGL framework in Javascript. Full control over creating graphics objects, etc., but requires more manual work. Examples: Github Repo A-Frame.js HTML wrapper for Three.js that integrates an entity-component system for composability, as well as a visual 3D inspector. Built on HTML / the DOMEasy to create custom components with actions that happen in a lifecycle (on component attach, on every frame, etc.)Examples: Github Repo PlayCanvas WebGL framework with Unity-like editorCould be convenient for quickly throwing together complex scenes. You can link out a scene to be displayed on top of a marker, or manually program a scene. Potentially less obvious to visualize / edit / collaborate / see what’s going on in code if you use an editor and publish a scene.Slightly unclear how easy it is to dynamically generate scenes based on incoming data / how to instantiate a scene with parametersExamples: Github Repo Recommendations for this project Here is what we decided to go with for our AR demo. Start with AR.js (another option was Zappar) + A-Frame.js for a basic working prototypeIn the longer term, explore options for advanced object recognition and tracking Read more about determining the best way to do marker tracking; narrowing down the use case and developing the interaction design; and content modeling for AR in our full coverage of week one of development.