A little-known ability of Redis’ Sentinel mode is event based “script” execution. When so configured, Redis can call an external executable file on certain events, passing the event information on to the command. This is useful for monitoring as well as reconfiguration events. Eventilator is a small utility I’ve written to make this process easier.
Handling Sentinel Events
First up, why use this instead of using the event subscription model? The first reason, and it is a big one, is that you don’t have to write a daemon which runs and waits for events. Instead you can have a fire-when-the-event happens model, which is much easier to integrate with monitoring systems which usually rely on single-invocaton semantics.
Another reason is the fact that, for certain events, only the leader sentinel will call these commands. In particular the event `+promoted-slave will only be fired off by the sentinel elected as the leader of the failover. This means not having to de-dupe failover event messages.
Sentinel has two options for these actions. The first is
‘client-reconfig-script’ and is called when a failover successfully completes.
This will only be called by the failover leader. Thus if you have a pod in
client-reconfig-script set to
the leader sentinel will call it with the failover information when the
+switch-master event is processed.
The second script mechanism is
notification-script, and it is called on every
“warning” level event. This includes events such as
+switch-master, and so forth. With one exception these events will trigger
the reporting sentinel to call the script configured. That exception is
+promoted-slave. This option is useful for recording, tracking, and
monitoring warning conditions.
To make this easier, as well as codify some “standard” or common actions I wrote “eventilator”. Written in Go it compiles to a static binary, making deployment nice and easy.
It has two modes and detects the mode in standard UNIX style by checking how
it was called (ie.
argv). It also has a config file to enable various
options and set parameters. Redis does not currently allow you to pass command
line options, hence the need for the config file. Each mode uses a dedicated
For use with
client-reconfig-script, it needs to be called as
eventilator is for the
The resulting config files are expected to be
The stock code includes nice features such as communicating events to a Sensu JIT listener, recording metrics in a Redis instance, and Slack integration. When new releases are cut travis.org runs tests and uploads the release binaries to the github repository’s releases page, making for easy download of new versions.
As you might be able to guess by the GitHub organization name, Eventialtor is the latest in a line of tools aimed at managing and interacting with Sentinel from an operational standpoint as opposed to as a Redis client. Keeping an eye on that organization will let you keep current on more cool utilities for working with sentinel.