I am evaluating using Cassandra as a large key-value store over about 500 commodity machines. AFAIK, Cassandra requires manual intervention during permanent machine failures (removing the deadnode, adding new machines). I would think that the machine failure rate is high enough to overload the administrator. So is it practically feasible to run Cassandra in such a large cluster?



There are mentions of clusters with size 300-400 & more machines. But plenty of tools are provided with Cassandra that could help with automation of deployment process. Plus you can find a plenty of recipes for Ansible (1, 2, ...), or some other automation frameworks.

More work could be required for other admin tasks, like, making sure that repairs are running, etc. But people are doing it, and there are "3rd-party" tools for them as well.


My team (of five) and I currently support 700+ Cassandra nodes across 30+ clusters in production. We use Spinnaker to help manage Cassandra in our external cloud. If a node gets blown away, it does replace it on its own.

You do want to keep a close eye on it though. If the replacement node comes up with a different IP, the old node will still be in the cluster (just DN). So at some point you'll need to nodetool remove it manually. Also, if multiple node failures happen at once, you'll need to make sure that it doesn't try to bootstrap more than one node at a time (which will also fail).

The bottom line, is that there are other tools like Spinnaker that can help with much of the heavy lifting. But it's not magic, and there is only so much "auto-healing" that it can do on its own.

Your Answer

