In this Valkey blog post, I will demonstrate how we can set up a simple replication process among the Valkey nodes and then finally use some auto-failover mechanism with the help of the Valkey-sentinel tool.
Replication (Master-Slave)
Basically, we have two setups with the information below.
1 2 3 | IP valkey port sentinel port 172.31.21.83 6379 26379 172.31.58.227 6379 26379 |
Now let’s prepare some basic configurations in file [/etc/valkey/valkey.conf] for both servers.
1 2 3 | bind IP ### Need to bind both IPs separately for each server. requirepass valkey masterauth valkey |
- requirepass – To add authentication in valkey
- Masterauth – To connect the replication securely.
Now restart the Valkey service on each node.
1 | systemctl restart valkey |
Next, we will set up a basic replication setup. To establish the replication, we need to execute the command below from the slave node.
Syntax:
1 | replicaof <masterip> <masterport> |
Command:
1 2 | 172.31.58.227:6379> replicaof 172.31.21.83 6379 OK |
The above setting can be persisted using the [config rewrite] command.
That’s all. The replication setup is successful now!
1 2 3 4 5 6 7 | 172.31.58.227:6379> info Replication # Replication role:slave master_host:172.31.21.83 master_port:6379 master_link_status:up .... |
We need to make sure the firewall/IP whitelisting is done for both servers.
Now, let’s do some data writing on the master[ 172.31.21.83] to confirm if the replication works fine or not.
1 | shell> valkey-cli -h 172.31.21.83 -a valkey |
1 2 3 4 | 172.31.21.83:6379> set key key1 OK 172.31.21.83:6379> get key "key1" |
On slave[172.31.58.227], the changes were very well reflected.
1 2 | 172.31.58.227:6379> get key "key1" |
The only caveat with this setup is if the master[172.31.21.83] goes down, then we have no auto-failover to sustain the incoming connections on other nodes unless we do some manual intervention.
However, there is one way of setting up an auto-failover mechanism in Valkey by using valkey-sentinel tool. In this way, the application can either use the direct sentinel port [26379] to connect to Valkey server or use some middleware/haproxy tool to distribute read/write. Well, we do not go into those details and are confined to the sentinel setup only here.
Auto-Failover using valkey-sentinel
First, we can edit the sentinel configuration file [/etc/valkey/sentinel.conf] on all nodes as below.
1 2 3 4 5 | sentinel monitor valkey-master 172.31.58.227 6379 2 sentinel down-after-milliseconds valkey-master 1500 sentinel failover-timeout va;key-master 3000 sentinel parallel-syncs valkey-master 1 sentinel auth-pass valkey-master valkey |
- sentinel monitor – It tells sentinel to start monitoring a new master with the specified name, ip, port, and quorum.
- sentinel down-after-milliseconds – Sentinel verify based on this if the instance is unreachable/down or not before doing any failover.
- sentinel failover-timeout – In case a sentinel voted another sentinel for the failover of a given master it will wait this many milliseconds to attempt to failover the same master again.
- sentinel parallel-syncs – The number of replicas that can be synced with the new master after a failover at the same time.
- sentinel auth-pass – To perform the secure authentication when requirepass is enabled in valkey configurations.
Then we can restart the valkey/valkey-sentinel services as below.
1 2 | systemctl start valkey-sentinel systemctl restart valkey |
We can see the sentinel process running on the default port [26379].
1 2 | shell> ps aux | grep valkey-sentinel valkey 14607 0.2 0.4 61332 15428 ? Ssl 06:54 0:13 /usr/bin/valkey-sentinel *:26379 [sentinel] |
Let’s do some failover now.
I stopped the Valkey service on the original master[172.31.21.83]. And then after a few seconds, slave[172.31.58.227] becomes the master.
1 2 3 4 | 172.31.58.227:6379> info replication # Replication role:master connected_slaves:0 |
Looking over the sentinel logs [/var/log/valkey/sentinel.log], we can see the master switchover events captured.
1 2 3 4 | 14607:X 04 May 2024 06:57:28.176 # +switch-master valkey-master 172.31.21.83 6379 172.31.58.227 6379 14607:X 04 May 2024 06:57:28.177 * +slave slave 172.31.21.83:6379 172.31.21.83 6379 @ valkey-master 172.31.58.227 6379 14607:X 04 May 2024 06:57:28.180 * Sentinel new configuration saved on disk 14607:X 04 May 2024 06:57:29.693 # +sdown slave 172.31.21.83:6379 172.31.21.83 6379 @ valkey-master 172.31.58.227 6379 |
Again, once we start the service on the old master [172.31.21.83], it automatically connects as a slave.
1 2 3 4 5 6 | 172.31.21.83:6379> info replication # Replication role:slave master_host:172.31.58.227 master_port:6379 master_link_status:up |
The status of the new master also reflects the changes.
1 2 3 4 5 | 172.31.58.227:6379> info replication # Replication role:master connected_slaves:1 slave0:ip=172.31.21.83,port=6379,state=online,offset=705402,lag=0 |
Conclusion
A basic replication setup in Valkey/Redis would be very handy to have multiple copies of data over the different machines and also act as a backup/disaster recovery option. Still, it’s better, especially for production setups, to have a sentinel layer in the topology to have a minimal impact on the application in case the master node goes down due to some reasons.
