Making HAProxy highly available with Keepalived

In the previous recipe, we used HAProxy to give our web servers some redundancy. The challenge with that solution is we now have a failure point in the load balancer itself. When architecting for HA, you need to cover all points of failure to make sure there is redundancy and that there is no SPOF. In this recipe, we will use Keepalived to add some HA to our configuration. Keepalived is a software application that is open source and designed for Linux-based systems. Its main function is to manage network load balancing and failover, ensuring the HA of web services. Keepalived is often used alongside HAProxy. The software primarily uses the Virtual Router Redundancy Protocol (VRRP) to achieve fault tolerance and evenly distribute the load. Keepalived uses the following features to provide its redundancy:

• High availability: With Keepalived, you can establish a cluster of backup servers that utilize a shared Virtual IP (VIP) address. This setup ensures that even if the primary server experiences a failure, a secondary server will automatically take over and handle incoming traffic, resulting in minimal downtime.
• VRRP: VRRP is a commonly used protocol that enables automatic router failover in IP networks. Keepalived utilizes VRRP to keep a VIP address operational, which can be assigned to any node within the cluster as needed.
• Health checking: The monitoring system of Keepalived regularly checks the health of active servers. If a server becomes unresponsive, Keepalived will remove it from the pool and redirect traffic to the healthy servers.
• Notification mechanisms: With Keepalived, it’s possible to set up notifications for failover events or when certain thresholds are exceeded. These notifications are useful for keeping an eye on the overall health of the cluster.

In a cluster, Keepalived designates one node as the master and the others as backups. The master node handles incoming traffic and responds to ARP requests for the VIP address, while the backups act as standby routers and monitor the master’s status. The nodes communicate using the VRRP protocol, with the master periodically sending VRRP advertisements to show its functioning. Often, a VIP is used to allow a single IP address for end user access. This normal operation is seen in the following figure, where we have Keepalived managing two HAProxy systems.

Figure 6.10 – Keepalived normal operations

If the backups stop receiving advertisements or detect any issues with the master, one backup node will take over as the new master. Keepalived checks the servers’ health using mechanisms such as ICMP (ping) checks or Layer 4 checks (e.g., checking whether a specific port is open) and removes failed servers from the pool. The VRRP priority is adjusted to ensure a healthy backup server takes over. This is seen in the following figure, where the VIP that users connect to has migrated over to the second node, and that node is now using HAProxy to manage the workload.

Figure 6.11 – Keepalived failed node