But what are the typical requirements of a stateful application? We could deploy a stateful application such as Apache ZooKeeper, MongoDB, Redis, or MySQL by using a Deployment, which could create a ReplicaSet with  replicas=1  to make it reliable; use a Service to discover its endpoint; and use PersistentVolumeClaim and Persistent‐ Volume as permanent storage for its state. While that is mostly true for a single-instance stateful application, it is not entirely true, as a ReplicaSet does not guarantee at-most-once semantics, and the number of replicas can vary temporarily. Such a situation can be disastrous and lead to data loss. Also, the main challenges come up when it is a distributed stateful service that is composed of multiple instances. A stateful application composed of multiple clus‐ tered services requires multifaceted guarantees from the underlying infrastructure. Let’s see some of the most common long-lived persistent prerequisites for distributed stateful applications. We could easily increase the number of  replicas  in a ReplicaSet and end up with a distributed stateful application. However, how do we define the storage requirements in such a case? Typically a distributed stateful application such as those mentioned previously would require dedicated, persistent storage for every instance. A Replica‐ Set with  replicas=3  and a PersistentVolumeClaim (PVC) definition would result in all three Pods attached to the same PersistentVolume (PV). While the ReplicaSet and the PVC ensure the instances are up and the storage is attached to whichever node the instances are scheduled on, the storage is not dedicated, but shared among all Pod instances. A workaround here would be for the application instances to use shared storage and have an in-app mechanism for splitting the storage into subfolders and using it without conflicts. While doable, this approach creates a single point of a failure with the single storage. Also, it is error-prone as the number of Pods changes during scal‐ ing, and it may cause severe challenges around preventing data corruption or loss during scaling. Another workaround would be to have a separate ReplicaSet (with  replicas=1 ) for every instance of the distributed stateful application. In this scenario, every Replica‐ Set would get its PVC and dedicated storage. The downside of this approach is that it is intensive in manual labor: scaling up requires creating a new set of ReplicaSet, PVC, or Service definitions. This approach lacks a single abstraction for managing all instances of the stateful application as one. 88  |  Chapter 11: Stateful Service Storage