Keepalived

Example loadbalancing configuration for Open-Xchange Cluster

Introduction

this page contains a basic description about how to set up keepalived for Open-Xchange cluster. This example is to work on debian systems. Keepalived mode is Direct Routing.

It is required to have ox servers and loadbalancer connected to the same switch or hub and that there is no filter for network packages between (some virtualization system do filter, too), so that MAC rewriting works.

For more information please see: www.keepalived.org

Directors setup

test1:~# apt-get install keepalived 
dpkg-reconfigure ipvsadm

answer the questions with "Yes" and then "Backup"

create a file

/etc/keepalived/keepalived.conf

with following contend (adapt network adresses)

global_defs {
    router_id OX
}

vrrp_sync_group OX_GROUP {
    group {
        OX_GOUP
    }
}

vrrp_instance OX_VRRP {
    state BACKUP
    interface eth0
    garp_master_delay 10
    virtual_router_id 10
    priority 101
    nopreempt
    advert_int 1
    authentication {
        auth_type AH   # Simple 'PASS' can use
        auth_pass 1234 # example password '1234' 
    }
    virtual_ipaddress {
        10.20.30.77/24 brd 10.20.30.255 dev eth0 # virtual service ip 10.20.30.67
    }
    virtual_ipaddress_excluded {
    }
}

virtual_server_group OX_HTTP {
        10.20.30.77 80         # virtual ip and port 80
}

virtual_server_group OX_OL_PUSH {
        10.20.30.77 44335      # VIP VPORT
}

virtual_server group OX_HTTP {
    delay_loop 3
    lvs_sched  rr
    lvs_method DR
    protocol   TCP
    virtualhost 10.20.30.77

    real_server 10.20.30.123 80 {
        weight 1
        inhibit_on_failure
        HTTP_GET {
            url {
                path /servlet/TestServlet
                status_code 200
            } 
            connect_port 80
            connect_timeout 10
        }
    }

    real_server 10.20.30.321 80 {
        weight 1
        inhibit_on_failure
        HTTP_GET {
            url {
                path /servlet/TestServlet
                status_code 200
            }
            connect_port 80
            connect_timeout 10
        }
    } 
}

virtual_server group OX_OL_PUSH {
    delay_loop 3
    lvs_sched  rr
    lvs_method DR
    protocol   UDP

    real_server 10.20.30.123 44335 {
        weight 1
        inhibit_on_failure
	  TCP_CHECK {
                 connect_port 9999
		  connect_timeout 5
        }
    }

    real_server 10.20.30.321 44335 {
        weight 1
        inhibit_on_failure
        TCP_CHECK {
                 connect_port 9999
		  connect_timeout 5
        }
    }
}

Networking adjustments

For the history:

1. We need the IP alias (or secondary IP) only on the "server" nodes (i.e. the nodes that keepalived will be a loadbalancer for); we dont need it for the keepalived node itself. The keepalived process on the keepalived node will configure itself the virtual IP as secondary IP on the eth0 device (and not on the lo device, as the server nodes will do.
2. The lines with echos to the files .../{lo,all}/arp_ignore and .../{lo,all}/arp_announce are redundant, respectively, as the setting for "all" also is in effect for of "lo".
3. The rp_filter is 0 by default, so lets not fiddle with it

So basically we only need one adjustment for keepalived, which is the ip_forward setting, and we need to configure the alias and the echos into the all/{arp_ignore,arp_announce} files for the server nodes.

On debian we can do this conveniently by the following adjustments:

1. For the keepalived node, configure in /etc/sysctl.conf: net.ipv4.ip_forward = 1
2. For the server nodes, use a stanza in the /etc/network/interfaces file. Adjust the IP.

auto lo:0
iface lo:0 inet static
    address 10.20.30.77
    netmask 255.255.255.255
    pre-up echo 1 > /proc/sys/net/ipv4/conf/all/arp_ignore
    pre-up echo 2 > /proc/sys/net/ipv4/conf/all/arp_announce
    post-up /sbin/route add -host 10.20.30.77 dev lo:0
    pre-down /sbin/route del -host 10.20.30.77 dev lo:0
    # reset to defaults
    post-down echo 0 > /proc/sys/net/ipv4/conf/all/arp_ignore
    post-down echo 0 > /proc/sys/net/ipv4/conf/all/arp_announce

How to do this on other operating systems needs to be documented here.

create a script that runs on every machine start:

#! /bin/sh

set -x

IP=10.20.30.77

echo 0 > /proc/sys/net/ipv4/conf/lo/rp_filter
echo 1 > /proc/sys/net/ipv4/ip_forward

echo 1 > /proc/sys/net/ipv4/conf/lo/arp_ignore
echo 1 > /proc/sys/net/ipv4/conf/all/arp_ignore
echo 2 > /proc/sys/net/ipv4/conf/lo/arp_announce
echo 2 > /proc/sys/net/ipv4/conf/all/arp_announce

/sbin/ifconfig lo:0 $IP netmask 255.255.255.255 broadcast $IP up
/sbin/route add -host $IP dev lo:0