Swap space, or swap, is an essential part of the memory management system in Linux and other operating systems.
Understanding swap is simple; It provides additional memory capacity for a Linux system when it runs out of physical RAM. Swap works by moving less-frequently accessed data from the volatile memory (RAM) to a dedicated swap partition or file on the hard disk.
Understanding The Relationship Between Swap and Kubernetes
This feature allows Linux systems to handle intense workloads better while avoiding crashes or sudden slowdowns.
Swap can be a double-edged sword, as using it can lead to decreased overall system performance, especially if the Linux kernel frequently swaps data in and out of the slower disk-based storage.
Swap Support and Kubernetes
Kubernetes is designed to provide efficient resource utilization, ensuring proper allocation and management of resources like CPU, memory, and storage.
Often, Kubernetes is configured to tightly pack instances to maintain performance and optimally utilize memory and CPU resources.
As Kubernetes relies on Quality of Service (QoS) for managing pod resources, disabling swap makes enforcing memory limits and maintaining the desired QoS levels easier – since we won’t be moving data in and out all the time.
An important consideration for Kubernetes is that the system’s performance can be negatively impact if swap is enabled.
Swap can slow down the performance of containerized workloads due to the increased latency of disk-based memory access compared to RAM.
In Kubernetes 1.22, swap support for the kubelet was introduced as an alpha feature, which means that the Kubernetes community acknowledges the potential benefits of swap under certain conditions.
Enabling this feature requires manually altering the kubelet configuration, and disabling swap by default on a typical Kubernetes node is still recommended for the performance reasons listed above.
Should You Be Disabling Swap for Kubernetes?
How and Why We Disable Swap
When installing Kubernetes, especially using tools like kubeadm, one of the common requirements is to disable swap.
Swap is a feature available on Linux systems, allowing the operating system to use a dedicated portion of the hard drive as extra memory when there isn’t enough RAM.
Disabling swap is done using the swapoff command followed by -a to disable all swap devices (e.g., sudo swapoff -a). This is followed by updating the /etc/fstab file to ensure swap remains disabled after rebooting.
The reason we disable swap for Kubernetes is due to the way it schedules and manages resources for running containers.
The Kubernetes scheduler is designed to efficiently utilize the available resources on a node, such as CPU and memory.
Enabling swap can interfere with these resource allocations, causing performance degradation and unpredictability. Thus, disabling swap is required for Kubernetes to function optimally, if speed is your priority over optimized RAM allocation.
Repercussions Of Permanently Disabling Swap
It’s important to acknowledge that permanently disabling swap might not always be ideal, especially for ephemeral Kubernetes environments or systems with limited memory.
In such cases, using swap can provide some advantages regarding resource utilization and performance.
For most production Kubernetes deployments, the benefits outweigh the potential issues, and disabling swap becomes crucial to ensure stable performance and behavior. We recommend evaluating your Kubernetes environment’s specific needs and use cases before deciding to disable swap permanently. Though, for the production builds I’ve done, I’ve always disabled swap.
Remember that, in some scenarios, Kubernetes can still run with swap enabled if you can manage the resources carefully. However, this is not the recommended approach and can lead to unexpected behaviors.
Understanding Swap in Kubernetes Nodes
Workloads and Memory Management
In Kubernetes, nodes are the worker components that run and manage application containers. Each node’s host system has a certain amount of machine memory available to be allocated amongst its workloads.
As part of managing these workloads, Kubernetes uses a Quality of Service (QoS) feature to prioritize resource allocation for pods based on their specified memory requirements.
When the memory pressure on a node increases, it must find ways to allocate resources efficiently. One method is to use swap partitions or swap files on the disk, allowing the host system to temporarily store data that can’t fit into the node’s memory. Essentially, enabling swap is a way to extend machine memory in times of need.
However, swap usage comes with its drawbacks when managing workloads in Kubernetes Nodes.
Swap Impact on individual Kubernetes Node
When a workload starts to hit its memory limit, the kubelet might let it start spilling over into swap instead of terminating the container. While this might seem like a helpful feature, it can cause significant performance issues.
Swapping data between disk and memory creates latency, which is undesirable for performance-sensitive applications.
This is especially true when using high-performance NVMe swap partitions.
Kubernetes is designed around the assumption that swap is not enabled, and its memory management features depend on knowing the exact amount of memory available.
By including swap, it can be harder for Kubernetes to make accurate decisions about memory allocation and pod eviction policies.
Swap Support Evolution in Kubernetes
Initially, Kubernetes required users to disable their swap space completely before launching a cluster. This was mainly for performance reasons, as swap space can slow things down, and the scheduler should ideally never use swap at all 1 .
Over time, though, the Kubernetes community has acknowledged the need for swap support in certain situations.
Future of Swap Support in Kubernetes
Recently, there has been some exciting news in the Kubernetes space – the platform has started implementing support for swap space in certain scenarios 2.
Although not fully supported yet, it’s a step towards enabling swap support for workloads in Kubernetes clusters.
With the introduction of the new alpha feature, Kubernetes now supports the use of swap space in cluster configurations.
This was a much-awaited update, giving users more flexibility when dealing with memory-intensive workloads.
While we haven’t reached full support for swap in Kubernetes, the recent updates have shown that the community is working towards addressing the issue.
As the system evolves, we’ll likely see more developments and improvements related to swap support in Kubernetes clusters.
Footnotes (and links!)
Exploring and Testing Swap Features
Let’s dive into how we can test this feature and provide valuable feedback to Kubernetes.
Testing Swap Support
For this feature to work, clusters with nodes that have swap enabled must be running a kubelet with specific requirements.
We encourage readers (YOU) to perform benchmarking tests to evaluate the suitability of adding swap support to their cluster environment.
By thoroughly testing the performance of our nodes in various scenarios, we’ll be able to understand better whether using swap memory adds value to our clusters.
Feedback and Contributions
Community feedback and contributions are among the most important aspects of open-source projects like Kubernetes.
As we test swap memory support in Kubernetes nodes, we should report any findings, issues, or improvements to the K8s SIG Node WG via the appropriate channels.
By doing so, we can help develop and refine this feature further, making it a more valuable and integral part of the Kubernetes experience.
There’s value in exploring the first two scenarios where swap support could benefit our Kubernetes clusters. But remember that it might only make a noticeable difference in specific use cases.
So we must share our experiences, findings, and any challenges during our testing and exploration. Together, we can make Kubernetes even more powerful and efficient.
