In 2017, seeing the lack of buzzwords in the tech industry, Alex Richardson from Weaveworks coined a new one — GitOps. Humor aside, what started as a blog describing their operational tooling has snowballed into something bigger. Descriptions of GitOps include “the next big thing,” “versioned CI/CD,” and even “placebo.”

Declarative Systems

To understand GitOps, you need to understand what a declarative system is. A declarative system is one where the code describes the final state of the system. An example of such a system would be the Infrastructure as Code (IaC) system, where YAML defines the configuration. We define the end state with declarative code, and then automation reconciles the system to the desired state.

What is GitOps?

Shapeshifting - the ability to physically transform oneself through an inherent ability, intervention, or manipulation.

GitOps is the management of declarative systems using Git. The core idea of GitOps is to use a Git repository as the source of truth. It holds the desired state of the system and acts as a contract between various components.

Here’s what a simple GitOps pattern looks like:

The system state is defined in the Git repository
The GitOps Operator detects changes in Git
Trigger a convergence mechanism to sync desired and observed state
Reconcile the system whenever there is drift
Notify when the system state changes

GitOps tools empower the system to shapeshift based on the state defined in the Git repository. But which declarative system can effectively leverage this pattern? The answer is Kubernetes.

What is Kubernetes?

κυβερνητες - pronounced Kie-ver-nee-tees means helmsman or pilot in Greek.

Over the last few years, containers have made it easy to run cloud-native applications. Containers are lightweight compared to virtual machines (VMs) and make more efficient use of the infrastructure. While container runtimes are great at managing individual containers on one host, they are not suited for managing applications running in multiple containers across multiple hosts. This type of management is where container orchestrators come into play.

Kubernetes is an open-source orchestration system that automates the deployment and management of applications running in Linux containers. The analogy of an orchestra conductor is common for describing Kubernetes.

What does Kubernetes do?

It provides an abstraction layer over infrastructure. Details of underlying computers, networks, and other components are hidden from users and applications, making it easier to develop and configure.
It makes an application portable as the underlying infrastructure no longer affects it. It allows deploying the same manifest on local computers and any cloud provider.
It uses a declarative model to configure applications. You describe the application in manifest and Kubernetes converts it into a running application.
After deploying an application, Kubernetes takes over its management. This management includes keeping applications running, recovering from failures, and scaling in case of changes in demand.

Since Kubernetes manages tasks declaratively, we can employ GitOps practices and tools for cluster management and application delivery.

GitOps and Kubernetes

An anti-pattern is a common response to a recurring problem that is usually ineffective and can end up being counterproductive. The following lists results of a common anti-pattern that occurs with Kubernetes deployment automation when the CI system tests, builds a change, and then pushes the change directly to the Kubernetes cluster.

The CI system needs access to the Kubernetes cluster API to support this push-based deployment model. This access poses a security risk.
In a namespaced cluster, the deployment pipeline would require cluster-scoped credentials. It significantly increases risk.
Another drawback is that the deployment pipeline triggers only when the Git repository changes. It cannot detect deviations from the desired state.

A GitOps pipeline uses the same concepts as a push-based pipeline but differs in how the deployment works. Git, the CI system, and the GitOps operator make up the troika of a GitOps pipeline. Unsurprisingly, Git forms the fulcrum around which it revolves. The CI system feeds changes to the Git repository at one end, and the GitOps operator consumes at the other.

The pull-based strategy eliminates the need to expose Kubernetes cluster API and credentials.
It separates CI and CD systems, thus improving security posture as the two pipelines can only communicate through Git.
It provides the flexibility to use any CI and Git tool.
The GitOps operator corrects any divergence from the desired state.

How Do We Do It?

At Contentstack, we use the GitOps pipeline to manage application delivery on Kubernetes. Here’s a glimpse at our pipelines.

Developers merge application code changes to the trunk and trigger the CI pipeline. It tests the change, builds a new container image, and pushes it to the container registry in Cloud.
We use Helm to package applications for Kubernetes. Any change in the Helm configuration triggers the Helm CI pipeline, which tests the package, creates a chart, and uploads it to the Helm repository.
Running either of the CI pipelines invokes the deployment pipeline. It combines artifacts from CI pipelines and writes a new state to the environment repository. We use Git branches to manage multiple environments within the environment repository. Change is propagated through environments using the CI system.
The GitOps operator running in a Kubernetes cluster detects a new commit in the environment repository and marks the observed state out of sync. This status change kicks in the convergence mechanism. Kubernetes pulls the latest changes and starts deployment updates to match the state in Git.
Upon completion, the GitOps operator notifies the deployment state by setting commit status in the Git repository. The CI system uses this status to determine if the change is successfully deployed and run any post-deployment actions.

Why Should You Adopt GitOps?

Faster Deployments

By adopting GitOps best practices, developers can deploy updates and features to Kubernetes more rapidly. With GitOps, you don’t have to switch tools for deploying your application. Everything happens in the version control system used for developing the application anyways.

Easier Compliance and Auditing

A change in an application environment requires updating the configuration in the corresponding Git repository. This change creates a complete history of desired state changes, including a record of who made the change and why. Since changes are tracked and recorded in Git, auditing is made easier.

Faster Recovery from Failure

As you have a complete history of changes in Git over time, recovering from failure is much easier. By reverting some commits, we can step back to a state that used to work previously.

Secure Deployments

As changes go through the Git repository, users and CI systems don’t need to access the cluster anymore. Separation of responsibility between packaging software and releasing it to an environment embodies the security principle of least privilege, reducing the impact of compromise and providing a smaller attack surface.

Backup of the Environment

In case of catastrophic failure, it’s possible to restore the environment quickly as the state of the system is stored in Git. Environment repository acts as a natural backup of the cluster.

What are the Challenges with GitOps?

“Any change, even a change for the better, is always accompanied by drawbacks and discomforts.” ― Arnold Bennett

Secret Management

There is no standard practice on how to manage secrets. To store secrets in Git, we need to encrypt them. This encryption step introduces a separate deployment process for secrets. If they are not stored in Git, then the idea of having your cluster state the same as Git is not valid anymore.

Multi-environment Deployments

GitOps does not define how to propagate a change to multiple environments. The most popular way of handling different environments is by using different Git branches. To do this, you need to use the CI system.

Complete Application Lifecycle

GitOps only focuses on application deployment. Other parts of the development lifecycle such as building, testing, packaging are not addressed by GitOps. You need to use another CI system and the GitOps tool to automate the complete development lifecycle.

Change of System State

One of the assumptions of GitOps is that the cluster state is the same as what’s defined in Git. However, your system changes. It could be due to auto-scaling or system tools updating the state of your resources, so this assumption might not be correct.

Conclusion

GitOps shows an opinionated way to implement deployment patterns with specific tools using Kubernetes. It does not provide radical new ideas; instead, it builds on established concepts of continuous delivery. As of today, GitOps is the best way to do continuous delivery using Kubernetes. The prevalence of Kubernetes will lead to more and more organizations adopting the GitOps pattern for their deployment needs. Fueled by increasing the user base and maturity of tools, the GitOps pattern will keep improving Kubernetes deployments

Nilesh Naik

Ready to reimagine possible?