While modern hardware is reliable, nodes can and do fail. Whether it is due to catastrophic hardware failure, an OS crash or a communication failure between cluster nodes, users with a high availability requirements need volume replication. StorageOS solves this problem for containerized environments using Kubernetes or OpenShift.
Replication for High Availability
Replication is the process by which one or more volumes can be kept in sync with a single master volume. High availability refers to the ability to switch between the master and replicas at will, so if the master is suddenly unavailable (for whatever reason), a replica can be promoted to master. This is essential for any organization wanting to run stateful applications in containers. Without it, the business risks data loss or downtime.
Feature Spotlight: StorageOS Volume Replication
With replication disabled, a StorageOS volume saves data to a single node in a cluster. When a node fails, access to the StorageOS volume is suspended for the duration of the node failure, thus causing outage for the application using the volume.
When enabled, under node failure condition, StorageOS volume replication will transparently promote a replica node to master. Mount endpoints migrate to the new master, and applications continue without requiring maintenance or downtime. From the perspective of the application, the only visible effect is a small pause in IO while the failover takes place.
This allows applications backed by StorageOS volumes to be turned into HA applications without extra development work or application refactoring.
How StorageOS Replication Works
All StorageOS volumes have a master replica to which writes are persisted. When additional replicas are enabled, slave volumes are created on other nodes in the cluster. As applications issue write requests, those writes are persisted to the master and all replicas synchronously, after which StorageOS (via the operating system) acknowledges the write to the application.
Replication traffic travels between nodes using standard TCP/IP and is compressed using the lz4 algorithm.
The number of replicas is tunable on a per-volume basis between 0-5, depending on customer requirements, by setting a StorageOS label (semantically equivalent to Kubernetes labels). Additionally, StorageOS policies allow the tuning of replication defaults for an entire namespace if preferred.
The StorageOS architecture allows us to move the backing store for a volume ‘under the covers’ while maintaining the same client facing devices/inodes. In the case of node failure, StorageOS promotes a replica to be master and no change is observable from the container, other than the aforementioned short delay.
Author: Paul Sobey
Paul Sobey is Head of Product at StorageOS. Paul has worked as a systems and infrastructure engineer for over 15 years, responsible for deploying cloud and on-premises infrastructure as well as deploying Kubernetes and containers in production.