A little-known ability of Redis’ Sentinel mode is event based “script” execution. When so configured, Redis can call an external executable file on certain events, passing the event information on to the command. This is useful for monitoring as well as reconfiguration events. Eventilator is a small utility I’ve written to make this process easier.

Handling Sentinel Events

First up, why use this instead of using the event subscription model? The first reason, and it is a big one, is that you don’t have to write a daemon which runs and waits for events. Instead you can have a fire-when-the-event happens model, which is much easier to integrate with monitoring systems which usually rely on single-invocaton semantics.

Another reason is the fact that, for certain events, only the leader sentinel will call these commands. In particular the event `+promoted-slave will only be fired off by the sentinel elected as the leader of the failover. This means not having to de-dupe failover event messages.

Sentinel has two options for these actions. The first is ‘client-reconfig-script’ and is called when a failover successfully completes. This will only be called by the failover leader. Thus if you have a pod in sentinel with client-reconfig-script set to /usr/sbin/reconfigurator, then the leader sentinel will call it with the failover information when the +switch-master event is processed.

The second script mechanism is notification-script, and it is called on every “warning” level event. This includes events such as +sdown, +odown, +switch-master, and so forth. With one exception these events will trigger the reporting sentinel to call the script configured. That exception is +promoted-slave. This option is useful for recording, tracking, and monitoring warning conditions.

Enter Eventilator

To make this easier, as well as codify some “standard” or common actions I wrote “eventilator”. Written in Go it compiles to a static binary, making deployment nice and easy.

It has two modes and detects the mode in standard UNIX style by checking how it was called (ie. argv[0]). It also has a config file to enable various options and set parameters. Redis does not currently allow you to pass command line options, hence the need for the config file. Each mode uses a dedicated config file.

For use with client-reconfig-script, it needs to be called as reconfigurator, whereas eventilator is for the notification-script mode. The resulting config files are expected to be /etc/redis/reconfigurator and /etc/redis/eventilator.conf respectively.

The stock code includes nice features such as communicating events to a Sensu JIT listener, recording metrics in a Redis instance, and Slack integration. When new releases are cut travis.org runs tests and uploads the release binaries to the github repository’s releases page, making for easy download of new versions.

The repo is on GitHub. The releases can be found on the Eventilator Release Page.

As you might be able to guess by the GitHub organization name, Eventialtor is the latest in a line of tools aimed at managing and interacting with Sentinel from an operational standpoint as opposed to as a Redis client. Keeping an eye on that organization will let you keep current on more cool utilities for working with sentinel.