etcd in Kubernetes: What is it and Why is it Important?

In this article, I will walk you through:

• etcd in Kubernetes – context setting
  • What is etcd and what are its key features?
  • The types of Kubernetes data stored in etcd
  • What are Kubernetes’ requirements of a backing store?
  • What makes etcd an ideal fit for Kubernetes?
• An outline of etcd’s sub-systems
  • Raft consensus algorithm
  • gRPC for cluster communication
  • Write-Ahead Logging (WAL) & Snapshotting
  • BoltDB
• High-level overviews of potential issues in etcd
• Key Takeaways for Site Reliability Engineers (SREs)

What is etcd?

Let’s kick things off with a description that is a bit more digestible than the official etcd definition. I’ve highlighted the essential features of etcd.

The key phrase here is “strong consistency guarantees”, which is exactly what Kubernetes needs to store critical cluster data.

The Benefits of etcd – Unpacking etcd’s Benefits

There is a lot going on in the above definition. Let’s unpack it into individual parts and focus on the benefits as they pertains to Kubernetes.

etcd has a robust architecture tailored to meet the demands of distributed systems such as Kubernetes. Let’s look at each piece of functionality and their benefits in greater detail.

We’ve explored the key functional features of etcd. Let’s now turn our focus to how Kubernetes utilizes etcd to organize and manage cluster data.

3 Types of Kubernetes Data Persisted in etcd

As a container orchestrator, Kubernetes needs to persist a variety of data types. Specifically, we will delve into the three essential types of data stored within etcd — state data, configuration data and metadata.

All of these are stored in key-value format. Let’s unpack each of these categories.

Below is a typical YAML representation that outlines various types of Kubernetes objects which are typically stored in etcd.

To give you a sense of the wide variety of data that’s stored in etcd, I’ve outlined the core purpose of each Kubernetes object along with the way etcd interacts with API Server.

Each object type serves a unique role, ranging from operational configurations to security measures, and plays a critical part in the orchestrated management of containerized applications in a Kubernetes environment.

As a key-value store, etcd is an ideal fit for Kubernetes to store these varied types of objects. At its core etcd’s responsibility is quite simple:

• Create and delete key/value pairs
• Watch keys (more on this in a later section).

Now that we have an understanding of the types of Kubernetes objects that need to be persisted, let’s take a look at Kubernetes architecture and where etcd fits into it.

What does Kubernetes expect of a data store?

Requirement #1: Strong consistency. Kubernetes needs to exactly know how many pods to scale and cannot afford to rely on outdated data. etcd prioritizes consistency over latency and availability, ensuring accurate and reliable state information to manage service scales effectively.

Requirement #2: “Watch” and change notification. Kubernetes requires a “watch” feature to monitor cluster changes like scaling, configuration updates, or failure recovery. Polling the data store frequently would be inefficient. When a change is detected in etcd—such as a new Pod being scheduled, a deployment being updated, or a node changing its status—etcd sends a real-time update to Kubernetes. Kubernetes can get notified of key-value store modifications, enabling prompt responses to events like pod scheduling or node status updates.

Requirement #3: Availability. etcd ensures that the Kubernetes control plane remains operational even in the event of control plane failure or node failure, ensuring that existing workloads on worker nodes continue to serve requests.
etcd can be configured in a multi-node setup where data is replicated across several nodes. This setup not only helps in achieving high availability but also in preserving data integrity and consistency. If one etcd node goes down, the others continue to serve the cluster’s data needs without loss of information.

Requirement #4: High volume reads and writes. etcd allows reads from any cluster member, not just the leader, optimizing read performance. There is also the ability to cache reads at the API Server level. From a write perspective, Kubernetes needs the ability to manage write operations to prevent slowdowns during critical updates, maintaining cluster stability. While writes are slower due to Raft driven consensus processes, etcd ensures they are reliable.

Where Does etcd Fit into the Kubernetes Architecture?

Kubernetes is a container orchestrator. Let’s start by looking at a high-level conceptual view of Kubernetes. Later, we’ll expand it into a full architectural view.

At a very high level, you can divide Kubernetes into 3 categories:

• Hardware Infrastructure: Comprises of the physical and virtual resources necessary for the cluster, including compute servers, network infrastructure, storage systems, and a container registry.
• Kubernetes Worker Nodes: Serve as the fundamental computational units within a Kubernetes cluster, executing applications and managing workloads.
• Kubernetes Control Plane (aka Kubernetes Master): Acts as the command center for the cluster, encompassing the API server, scheduler, controller manager, and various control loops that regulate state and manage resources.

The master node acts as the control plane, managing the state of the cluster. The worker nodes host the actual applications in the form of containers, managed by the master.

The control plane itself has 4 key sub-systems within a Kubernetes cluster:

• Within the worker and control plane nodes:
  • Kubelet: This is a node-specific process or an agent running on each node. Kubelet looks at the desired configuration and sets up pods, attaches containers to the network and volume mounts.
• Exclusively within the control plane nodes :
  • Scheduler: Responsible for allocating pods to nodes based on resource availability and other scheduling constraints.
  • Controller Managers (KCM and CCM): The Kubernetes Controller Manager (KCM) oversees a collection of controllers. These controllers are background processes that handle routine tasks in the cluster, ensuring that the current state of the system matches the desired state specified by the user. The Cloud Controller Manager (CCM) manages cloud provider-dependent logic, such as managing services and routes within a Kubernetes cluster.
  • API Server: Serves as the frontend to the cluster’s shared state, allowing users and internal components to interact and configure Kubernetes resources. The API server that acts as the “glue” or “hub”, facilitating communication between control plane processes and nodes. It connects the internal controllers to the kubelet. API Server is the only component that talks to etcd as you will see below.

With the above in mind, we can now look at a more technical Kubernetes logical architecture diagram.

As you can see, the API server is the only component on the control plane that communicates with etcd. As an SRE, you must monitor the health, availability and performance of both API server and etcd.

Kubernetes needs a strongly consistent and reliable data store to manage distributed containerized workloads. These tasks can be visualized as follows.

etcd acts as a centralized repository for all cluster information. This includes tracking the current and desired states, resource configurations, and runtime data. For each task, etcd holds vital information and via the API server, enables Kubernetes to achieve these tasks.

Kubernetes and etcd: Operational Scenarios for SREs

etcd maintains the desired state and configuration of the entire cluster. Any changes or updates to the cluster state, whether initiated by administrators or automated controllers, are first recorded in etcd. This ensures that the actual state of the cluster can be reconciled with the desired state stored in etcd.

Let’s explore some real-world scenarios where etcd’s functionalities are pivotal within a Kubernetes context.

Scenario	Description	Real-world Example
Pod Creation	When an SRE creates a new pod using the Kubernetes API, the request is received by the Kube API Server, which validates the request and stores the desired state in etcd.	An SRE deploys a new microservice by creating a pod manifest. The Kubernetes API Server stores the pod specification in etcd, and the cluster’s scheduler uses this data to schedule the pod onto a suitable node.
Scaling Deployments	When an SRE scales a deployment, such as increasing the number of replicas, the Deployment controller updates the desired state in etcd, ensuring that the actual state converges to match the desired state.	In response to increased traffic, an SRE scales up a web application deployment from three replicas to five replicas. The Deployment controller updates the desired state in etcd, and the cluster automatically creates additional pods to meet the new desired state.
Cluster Configuration	All configuration data for the Kubernetes cluster, including information about nodes, pods, services, and namespaces, is stored in etcd. Any changes to the cluster configuration are reflected in etcd.	An SRE updates the network policy for a specific namespace using kubectl. The Kube API Server validates the request and updates the corresponding configuration in etcd, ensuring that the new network policy is enforced across the cluster.
Cluster State Recovery	In the event of a cluster failure or restart, Kubernetes relies on etcd to reconstruct the cluster state. All cluster data stored in etcd is used to bring the cluster back to its last consistent state.	Following a sudden power outage, the Kubernetes cluster restarts. The Kube API Server retrieves the cluster state from etcd, allowing the cluster to resume its operations from where it left off before the outage.

4 Key Sub-systems of etcd

etcd meets Kubernetes’ requirements through its robust architecture, which is built around four key sub-systems.

Each of these sub-systems plays a vital role in ensuring that etcd is not just a data store, but a dynamic, reliable, and integral part of the Kubernetes ecosystem. In this section, we’ll delve into these four sub-systems, exploring how they contribute to the performance, reliability, and consistency of etcd, and by extension, the entire Kubernetes cluster.

Sub-system #1: Raft Consensus Algorithm

TL;DR: Data consistency across the cluster.

Raft is a consensus algorithm designed for high availability and fault tolerance in distributed systems such as etcd. etcd uses Raft to ensure data consistency across multiple nodes. Raft manages leadership elections and replicates entries across all nodes to maintain a highly available system.

Key problem that Raft solves: Nodes and network failures happen all the time in distributed systems. Loss of node machines or network should not result in loss of critical service configurations and state data. Raft ensures that a consistent view of the etcd data is available across the cluster, even in the presence of node failures or network partitions.

How Raft Solves This Problem: Raft handles this through leader elections and log replication processes. etcd has the concept of a leader and one or many followers. Ideally, all changes to the database are made by a single leader within a cluster. When a node fails, Raft triggers a new election to choose a leader which then manages the data replication process to ensure all active nodes are synchronized and up to date.

Raft has the concept of a “proposal” This is a request that is sent to the Raft cluster for processing. The cluster receives proposals from members, commits them, and applies them to the local store.

Sub-system #2: Write-Ahead Log (WAL)

TL;DR: Data integrity and aids in recovery. Every change is logged before execution to support etcd’s reliability in Kubernetes environments.

We saw the concept of how Raft processes proposals. These proposals need to be stored somewhere for processing and tracking.

The Write-Ahead Log (WAL) in etcd serves as a Raft persistent storage mechanism for storing proposals. When a leader receives a proposal, it first appends this proposal to its own log in the WAL. Simultaneously, the leader replicates this proposal to its followers using the Raft consensus protocol. Each follower then independently appends the proposal to its own WAL before acknowledging it back to the leader. This concurrent replication process ensures that the proposal is safely stored in the WAL of the leader and its followers.

Key problem that WAL solves: When unexpected failures (like crashes or power outages), etcd should not lose the critical Kubernetes state data. Ideally, you need a snapshot of transactions in the etcd cluster that can be used to rebuild the etcd state if needed. The key idea is that state changes should be logged prior to any heavy update to permanent storage.

How WAL solves the problem: WAL ensures durability, consistency and recovery of Raft changes. WAL records every change before it is committed to etcd’s datastore. This log plays a critical role in recovery by providing a replay of events to bring etcd back to its last known good state before the failure, thus preventing data loss.

Sub-system #3: BoltDB (BBolt)

TL;DR: Store data on disk efficiently for fast, durable access. Crucial for Kubernetes’ rapid data retrieval needs.

BoltDB is the embedded key-value storage engine used by etcd for storing data on disk. It provides high performance, efficiency, and ACID (Atomicity, Consistency, Isolation, Durability) transactions, making it suitable for etcd’s needs. You get a high-performance key-value storage engine with BoltDB.

Internally BoltDB uses MVCC – Multi-Version Concurrency Control. MVCC is a mechanism used by etcd to manage concurrent access to data. It maintains multiple versions of data to ensure consistency during transactions. This allows for a key-value store to be viewed from past revisions since historical key revisions are retained.

This model is particularly useful in etcd as it ensures that the key-value store can be safely accessed and modified by multiple clients without conflicts or data loss.

Key problem that BoltDB solves: Efficient and reliable storage is needed for the key-value pairs that etcd manages, accommodating high throughput and low latency access to data.
High performance and efficient data access are critical for the responsiveness and stability of applications that rely on Kubernetes. Slow or inefficient data operations can lead to increased latency and reduced application performance.

How BoltDB solves the problem: BoltDB uses a B+ tree indexing structure which enables efficient read/write operations and helps in compact data storage. This structure is particularly effective in handling large datasets with minimal performance overhead, making it suitable for the high-performance requirements of etcd.

Sub-system #4: gRPC

TL;DR: efficient and scalable client-server communication between nodes and between API server and etcd.

gRPC is utilized for client-server communication within etcd, facilitating efficient, high-performance remote procedure calls (RPCs) that support the cluster’s distributed nature.

Key Problem Solved: Effective communication between distributed etcd nodes and between clients and etcd servers must be maintained to handle high volumes of requests efficiently.

Why It Is Important to Solve: Robust and scalable communication is essential for distributed systems to ensure that data updates and requests are processed reliably and without significant delays, which are critical for maintaining the operational integrity of Kubernetes clusters.

How It Solves This Problem: gRPC facilitates this communication through its support for advanced features like streaming and multiplexing. This enables etcd to manage a high volume of network traffic and data distribution demands efficiently, ensuring that communication between nodes and clients remains fast and reliable.

A client request to write data would involve communication from the gRPC server to the etcd server. The etcd server uses the Raft consensus algorithm to replicate the write across the cluster, and finally ensures the write is logged in the WAL before being applied to the store.

High-level Walkthrough of Potential Issues in etcd

We’ve now covered the foundational components that form etcd’s robust architecture. However, each of the sub-systems can undergo malfunction that could disrupt etcd’s performance. Understanding these potential issues is key for SREs to put in place preemptive measures and effective troubleshooting.

While a comprehensive exploration of these issues would be best suited for a dedicated blog post, I’ll provide a high-level walkthrough of common issues in etcd.

Key Takeaways on etcd in Kubernetes

Perfect fit: We’ve seen that etcd is a perfect fit for Kubernetes. It is the single-source-of-truth for Kubernetes’ clusters state – configuration data, state information, and metadata. etcd offers strong consistency, high availability through the Raft consensus algorithm, and the ability to stream changes to clients efficiently. All of these make etcd an ideal match for Kubernetes’ distributed system architecture.

etcd’s moving parts can trip you up: However, etcd has many moving parts which can trip up SREs. Performance issues can originate from any of etcd’s sub-systems. These varied issues could range from leader election inconsistencies, high latencies between nodes, network partitions, Fsync issues to the etcd database going out of space. Each of these could put the Kubernetes cluster stability at risk. No new changes can be made, and new pods cannot be scheduled.

Important to monitor etcd health: SREs need to build a strong foundational knowledge of etcd’s inner sub-systems. This can help you better understand where issues originate from and help detect and address problems before they impact the Kubernetes cluster.

Next Steps – Key Observability Signals to Monitor etcd

As an SRE, your next step would be to get a more in-depth view of etcd’s internals and the issues that arise from each of etcd’s sub-systems.

In our forthcoming blog series, we’ll delve deeper into common etcd problems, explore key performance metrics, and discuss strategies for effective monitoring and troubleshooting.

• Kubernetes Monitoring & Alerting Best Practices | eG (eginnovations.com)
• Kubernetes Monitoring & OpenShift Monitoring Metrics | eG (eginnovations.com)
• What is etcd? | eG Innovations