Yesterday Salvatore released the long-awaited native Redis Cluster solution as part of Redis 3.0. It takes one specific approach to building a cluster that has operational implications - clients need to know which Redis master node to talk to. The server will redirect, and can be asked directly, when the client submits a command and the client will then need to ensure it connects to the appropriate master. Essentially it is like client-side hashing where the server tells you the results.

This is certain a reasonable choice and has it’s benefits. For example a chief benefit in my eyes is each node having direct connectivity scales network-wise better than a proxy sitting in front. However you do have the price of the discovery, the round trip on a miss, and of course the IP costs - every node in the cluster has to be reachable by clients directly. Another chief benefit is the division of client-based concurrency - Redis is more horizontally scalable. Given the effects on performance of high client concurrency this can be a significant benefit on it’s own.

There are other implications which may not be obvious with this model. In this model if you have a hot spot (and you will almost assuredly get one) which is limited by the network capability of that host you have to do migrations directly in order to split it out - and that may be quite complex.

I am certain there are about to be a plethora of Redis Cluster articles and posts to be written on the rest of the year as more people discover these and other implications, and learn the hard lessons of distributed data as they prematurely move from a simple HA setup to a Redis Cluster. What I want to highlight here are alternatives which may not get much recognition - and in some cases attention which can turn a good bit of software into a great one.

Thus in this post I intend to illustrate different methods to achieve similar results to Redis Cluster. Each will have their own benefits and implications and I’ll try to highlight those. This is the first in a relatively short series on these options. Each of the projects I think have the most potential or benefit will be reported on in greater detail in their own articles, with this post serving as the introduction.

Different Strokes

Direct communication is one option to implementing a Redis cluster, but there is another: the proxy model.

Redis Proxy Model 101

The proxy model is the one followed by perhaps the most famous of the CR (Clusterizing Redis) projects - Twemproxy. It’s even right there in the name. Under this model clients speak to a proxy which then handles Redis directly on behalf of the client. Ideally one could launch several instances of the proxy and provide a means for clients to be distributed among them such as via Round Robin DNS entries or by one or more TCP load balancers in front of the proxies.

Most of the projects I’m going to write about in this series will be following the Proxy model. The proxy model is relatively simple to implement and understand and minimizes client-side support requirements.

Key Distribution

Usually under this model a proxy allows you to run multiple Redis instances as masters with each controlling a “shard” - a portion of the keys. Through consistent hashing the proxy can always determine what slot a key fits in, and then map that slot to a node. By doing this in the proxy the client code doesn’t need to know about the layout on the back end. However, because not all keys will be on the same node not all commands are supported. For example with a single-master Redis deployment SUNIONSTORE results sa sb sc works without caveat.

However under the proxy model if any of those keys are on different nodes the command will not generally operate. In order to complete this command the proxy would have to fetch the data from every node which has the keys, then do the operation the proxy. This may sound easy. It isn’t.

Under normal Redis this command would be atomic without question. As Redis is single-threaded there is zero opportunity for any of the sorted sets in question to be modified. However, if the three keys in the example are each on a different server you now must either accept that the operation is no longer guaranteed to be atomic - or the proxy must “lock” each node with has a key the command needs.

The easiest route for proxy writers is to simply not support that command. Indeed, Redis Cluster itself makes this choice when the keys are not all on the same node. To date the vast majority of CR proxy projects elect to simply not support these commands at all.

Twemproxy

Twemproxy (aka “nutcracker”) was, if memory serves me correctly, the first one Salvatore wrote about. With roots at Twitter Twemproxy was written with the idea of scaling network connections. By operating as remote pool Twemproxy could scale concurrent clients by re-using connections to the back-end nodes.

With the addition of built-in consistent hashing and back-end routing it becomes a CR project.

When using it in this fashion, Twemproxy will indeed allow you to “cluster-ize” Redis by running multiple masters to grow beyond a single-machine’s memory capacity. It will let you run multiple instances of Twemproxy to provide client->proxy scaling. It has long been the gold standard in proxy performance

• most newer implementation compare themselves to it if they can show “better” numbers or to Redis if they can’t.

Where Twemproxy is missing things, is in management of the replication. There is an effort underway as I write this to bring in Sentinel support. Proper Sentinel support will make a huge improvement to Twemproxy, whereas going too simple with it will make it worse than no-support.

Codis

Codis is a newer project written in Go and following the path of the proxy + backend management. Searching Github reveals a plethora of projects trying to implement some portion of Redis or a proxy of Redis written in Go. It is almost as if “Write a Redis implementation” is the exercise used in the process of learning network programming in Go.

Like Twemproxy you can use multiple proxy nodes. Unlike Twemproxy it has dependencies beyond Redis and itself - it needs either Zookeeper or Etcd. That said Codis does take more steps than Twemproxy - and certainly takes some bold steps. It provides a system for migrating slots, automatic memory based balancing of slots/node, and to some extent node replication management. The boldest step of them all is essentially a fork of Redis to provide native slot migration capabilities.

XCodis

XCodis is a fork of Codis which makes some interesting choices. It removes several of what I feel are salient features in Codis and exchanges them for different ones. Namely, and on the positive side IMO, it removes the requirement to run a modified Redis. It does this primarily, I suspect, in order to support using LedisDB (same author) as the backing store.

However, it removes the Codis dashboard and auto rebalance features - and in the process occludes much of what Codis does for you. Codes follows the proxy model combined with the “no support” option for many multi-key operations. Like Twemproxy you can use multiple proxy nodes. As a fork of Codis it has dependencies beyond Redis and itself - it needs either Zookeeper or Etcd.

Summary

These are the first three Cluster-izing Redis projects I’m going to write about. They each provide the same basic functionality and provide various levels of the same capabilities of Redis Cluster. Redis Cluster will not be the solution for most people given it’s requirements. This isn’t a bad thing, but knowing some alternatives will be very handy for those needing to determine if they need or want it.