Documentation for getting started with the HA cluster stack on Debian Jessie and beyond.

This page is under construction.

Overview

To begin, you should have two machines, both freshly loaded with your brand new copy of Debian 8+. We'll need to do some configuration on both machines, so here are some things to keep in mind as you follow along:

We'll briefly cover getting the machines configured and ready for - and then get into - installing and configuring the Pacemaker/Corosync2.X HA cluster stack.

Configure the hosts

Plan the Installation

It's assumed that we have two nodes running Debian Jessie, and it's also assumed that both hosts are connected somehow, so one can see each other and viceversa.

What we'll do in this article, is configure a cluster integrated by two nodes, where we'll configure the following resources:

For this purpose, we chose two nodes, which only have one interface with a public IP:

# cat /etc/debian_version
8.1

An easy way to proceed is execute the commands in one of the nodes, and in paralell execute the same command in the other node.

In order to do that we need to copy the ssh-keys.

Prepare SSH between nodes

Generate a key in both nodes and copy over to the other node:

# ssh-keygen -t rsa
# ssh-copy-id root@node01
# ssh-copy-id root@node02

Now we can try to connect from node01 to node02 and viceversa.

Install the Pacemaker/Corosync2.X HA cluster stack

Since the Jessie branch doesn't have the packets for the new stack, we will use two different repositories:

From the testing branch we'll take the packets libqb0 and fence-agents, since they are the latest ones that the ha-team has upload to oficial repository.

From ppa.mmogp.com we will take the other ones, like the new version of pacemaker and corosync

First of all, we have to follow this guide ppa repository in order to set up the repository

Then we add the testing repository:

# echo "deb http://debian.mirror.iphh.net/debian testing main" > /etc/apt/sources.list.d/testing.list

And we should add also a preferences file, so in this way we won't mess with the system:

# cat << EOF > /etc/apt/preferences.d/preferences
 Package: *
   Pin: release a=jessie
   Pin-Priority: 900
 
   Package: *
   Pin: release a=testing
   Pin-Priority: -10
 EOF

We are ready to get the packages:

# apt-get update
# apt-get install -t testing libqb0 fence-agents
# apt-get install pacemaker corosync

Now we have the modern stack:

# apt-cache policy corosync pacemaker
corosync:
  Installed: 2.3.4-1
  Candidate: 2.3.4-1
  Version table:
 *** 2.3.4-1 0
        500 https://ppa.mmogp.com/apt/debian/ jessie/main amd64 Packages
        100 /var/lib/dpkg/status
     1.4.6-1.1 0
        500 http://debian.mirror.iphh.net/debian/ jessie/main amd64 Packages
        500 http://debian.mirror.iphh.net/debian/ testing/main amd64 Packages
pacemaker:
  Installed: 1.1.12-1
  Candidate: 1.1.12-1
  Version table:
 *** 1.1.12-1 0
        500 https://ppa.mmogp.com/apt/debian/ jessie/main amd64 Packages
        100 /var/lib/dpkg/status

And we have also the right version of libqb0 and fence-agents:

# apt-cache policy fence-agents libqb0
fence-agents:
  Installed: 4.0.18-1
  Candidate: 4.0.18-1
  Version table:
 *** 4.0.18-1 0
        500 http://debian.mirror.iphh.net/debian/ testing/main amd64 Packages
        100 /var/lib/dpkg/status
     4.0.17-2 0
        500 https://ppa.mmogp.com/apt/debian/ jessie/main amd64 Packages
     3.1.5-2 0
        500 http://debian.mirror.iphh.net/debian/ jessie/main amd64 Packages
libqb0:
  Installed: 0.17.1-3
  Candidate: 0.17.1-3
  Version table:
 *** 0.17.1-3 0
        500 http://debian.mirror.iphh.net/debian/ testing/main amd64 Packages
        100 /var/lib/dpkg/status
     0.17.1-1 0
        500 https://ppa.mmogp.com/apt/debian/ jessie/main amd64 Packages
     0.11.1-2 0
        500 http://debian.mirror.iphh.net/debian/ jessie/main amd64 Packages

# apt-get install nginx

Everything is ready to configure the cluster

Configure the Pacemaker/Corosync2.X HA cluster stack

Corosync supports multicast and unicast, but here we'll show an unicast configuration.

Before put our hands into it, we need to prepare the firewall.

Corosync uses the port 5405 udp to communicate with the other members, so we have to creare a rule to allow a traffic.

For example:

Iptables:

iptables -A INPUT -p udp --dport 5405 -d [node] -j ACCEPT

Shorewall:

ACCEPT                  net:[node]                      $FW    udp         5405

Now, the corosync configuration could be something like:

totem {
        version: 2
        token: 3000
        token_retransmits_before_loss_const: 10
        clear_node_high_bit: yes
        crypto_cipher: none
        crypto_hash: none
        transport: udpu
        interface {
                ringnumber: 0
                bindnetaddr: [public ip]
        }
}

logging {
        to_logfile: yes
        logfile: /var/log/corosync/corosync.log
        debug: off
        timestamp: on
        logger_subsys {
                subsys: QUORUM
                debug: off
        }
}

quorum {
        provider: corosync_votequorum
        two_node: 1
        wait_for_all: 1
}

nodelist {
        node {
                ring0_addr: node01
        }
        node {
                ring0_addr: node02
        }
}

This is a very simple setup that it works, but we can see all the available options here:

man corosync.conf 5

Now we can start the services:

# service corosync start
# service pacemaker start

# crm status
Last updated: Thu Jun 11 10:42:19 2015
Last change: Thu Jun 11 10:41:46 2015
Stack: corosync
Current DC: node02 (1053402613) - partition with quorum
Version: 1.1.12-561c4cf
2 Nodes configured
0 Resources configured


Online: [ node01 node02 ]

Adding Resources

Ok, now we can start configuring the resources.

We'll configure Nginx, a shared IP when Nginx will run, also we'll tell to the cluster that we want that both resources run in the same node, and since we need the IP to start Nginx, we'll set up a properly order.

Also, since both nodes are equal, we won't specify any location preference, and we'll also specify that if one nodes crash, and the other takes the resources, the resource will not be migrated back to the node that crashed.

# crm configure
crm(live)configure# property stonith-enabled=no
crm(live)configure# property no-quorum-policy=ignore
crm(live)configure# property default-resource-stickiness=100
crm(live)configure# primitive IP-rsc_nginx ocf:heartbeat:IPaddr2 params ip="xx.xx.xxx.xx" nic="eth0" cidr_netmask="xx.xx.xx.xy" meta migration-threshold=2 op monitor interval=20 timeout=60 on-fail=restart
crm(live)configure# primitive Nginx-rsc ocf:heartbeat:nginx meta migration-threshold=2 op monitor interval=20 timeout=60 on-fail=restart
crm(live)configure# colocation lb-loc inf: IP-rsc_nginx Nginx-rsc
crm(live)configure# order lb-ord inf: IP-rsc_nginx Nginx-rsc
crm(live)configure# commit

We'll receive a warning related to the timeout for the operations start/stop on Nginx. You are free to configure it, but for the purpose of this, that is enough.

We also disabled stonith for now, but later we'll come back to it again.

# crm status
Last updated: Thu Jun 11 11:05:45 2015
Last change: Thu Jun 11 11:04:35 2015
Current DC: node02 (1053402613) - partition with quorum
2 Nodes configured
2 Resources configured


Online: [ node01 node02 ]

 IP-rsc_nginx   (ocf::heartbeat:IPaddr2):       Started node01
 Nginx-rsc      (ocf::heartbeat:nginx): Started node01

Fencing

After configure the normal resources, we saw that everything is running as expected. Now it's time to dig into fencing.

Fence is used to put a node into a known-state, so if we're having a problem inside of the cluster, and one node is flapping or behaiving bad, we'll put this node into an state that we know is safe. ( Fence-Clusterlabs )

To see all the stonith devices/agents that you have, you can run:

# stonith_admin -I

This command will return you all the stonith devices you can use for fencing purposes.

* JFI: fence-agents package stores all fence-agents in /usr/sbin ( ls /usr/sbin/fence_* )

You have to choose properly which fence-agent do you need, because it depends on which hardware/software is relaying your cluster.

Once you have choosen it, you can inspect all the options by:

man [fence_agent]

For this article I created a cluster of two machines running on top of KVM, so I'll use fence_virsh agent.

Since our cluster is already running, we'll create a new shadow config to put our new changes there, so in this way we can do some changes and check further without mess up with the working configuration.

crm(live)# cib new fencing
INFO: cib.new: fencing shadow CIB created
crm(fencing)# configure
crm(fencing)configure# property stonith-enabled=yes
crm(fencing)configure# primitive fence_node01 stonith:fence_virsh \
   >         params ipaddr=virtnode01 port=node01 action=off login=root passwd=passwd pcmk_host_list=node01 \
   >         op monitor interval=60s
crm(fencing)configure# primitive fence_node02 stonith:fence_virsh \
   >         params ipaddr=virtnode02 port=node02 action=off login=root passwd=passwd delay=15 pcmk_host_list=node02 \
   >         op monitor interval=60s
crm(fencing)configure# location l_fence_node01 fence_node01 -inf: node01
crm(fencing)configure# location l_fence_node02 fence_node02 -inf: node02
crm(fencing)configure# end
There are changes pending. Do you want to commit them (y/n)? y
crm(fencing)#

Ok, we've written the changes, but the new configuration is not deployed yet.

Now we can simulate what will happen:

crm(fencing)# cib cibstatus simulate

Current cluster status:
Online: [ node01 node02 ]

 IP-rsc_nginx   (ocf::heartbeat:IPaddr2):       Started node01 
 Nginx-rsc      (ocf::heartbeat:nginx): Started node01 
 fence_node01   (stonith:fence_virsh):  Stopped 
 fence_node02   (stonith:fence_virsh):  Stopped 

Transition Summary:
 * Start   fence_node01 (node02)
 * Start   fence_node02 (node01)

Executing cluster transition:
 * Resource action: fence_node01 monitor on node02
 * Resource action: fence_node01 monitor on node01
 * Resource action: fence_node02 monitor on node02
 * Resource action: fence_node02 monitor on node01
 * Pseudo action:   probe_complete
 * Resource action: fence_node01 start on node02
 * Resource action: fence_node02 start on node01
 * Resource action: fence_node01 monitor=60000 on node02
 * Resource action: fence_node02 monitor=60000 on node01

Revised cluster status:
Online: [ node01 node02 ]

 IP-rsc_nginx   (ocf::heartbeat:IPaddr2):       Started node01 
 Nginx-rsc      (ocf::heartbeat:nginx): Started node01 
 fence_node01   (stonith:fence_virsh):  Started node02  
 fence_node02   (stonith:fence_virsh):  Started node01  

crm(fencing)#

This is a nice way to test changes in the configuration without crash anything.

Once we've seen that everything will work as expected, we can commit the new changes:

crm(fencing)# cib commit
# crm status
Last updated: Tue Jun 13 16:05:26 2015
Last change: Tue Jun 13 16:05:00 2015
Current DC: node01 (1053402612) - partition with quorum
2 Nodes configured
4 Resources configured


Online: [ node01 node02 ] 

 IP-rsc_nginx   (ocf::heartbeat:IPaddr2):       Started node01 
 Nginx-rsc      (ocf::heartbeat:nginx): Started node01 
 fence_node01        (stonith:fence_virsh):  Started node02 
 fence_node02        (stonith:fence_virsh):  Started node01

We can see the stonith resources running, so our fencing is well configured.

Testing the Cluster

Now it's time to test this war-machine in order to see if it's reliable and it'll behavier as we expect.

Possible tests

Migrate a resource [text]

Stop/Start Resources [text]

Kill a Resource

# killall -9 nginx
# pstree | grep nginx <- any process

When op monitor hits again, it detects that Nginx is not running, and pacemaker start again the resource:

# pstree | grep nginx &> /dev/null && echo "Running"
Running

Poweroff one node

Here we'll simulate a power-cut

# crm status
Last updated: Tue Jun 16 17:07:37 2015
Last change: Tue Jun 16 17:03:46 2015
Current DC: node01 (1053402612) - partition with quorum
2 Nodes configured
4 Resources configured


Online: [ node01 node02 ]

 IP-rsc_nginx   (ocf::heartbeat:IPaddr2):       Started node02 
 Nginx-rsc      (ocf::heartbeat:nginx): Started node02 
 fence_node01   (stonith:fence_virsh):  Started node02  
 fence_node02   (stonith:fence_virsh):  Started node01

The resources are running on node02, so this is the node we'll poweroff.

Since I'm working in a virtual environtment, I 'll do a:

virsh destroy node02

crm(live)# status
Last updated: Tue Jun 16 17:10:52 2015
Last change: Tue Jun 16 17:03:47 2015
Current DC: node01 (1053402612) - partition with quorum
2 Nodes configured
4 Resources configured


Online: [ node01 ]
OFFLINE: [ node02 ]

 IP-rsc_nginx   (ocf::heartbeat:IPaddr2):       Started node01 
 Nginx-rsc      (ocf::heartbeat:nginx): Started node01 
 fence_node02   (stonith:fence_virsh):  Started node01

As we can see here, since node02 went away and was marked as OFFLINE, node01 took over the resources.

Put a node as a standby [text]

Testing Stonith

We've tested the cluster. Now it's time to see if our fencing will run as expected.

Tests we'll do:

Kill Corosync

It's assumed that both nodes are running together again. We need to choose one of them, and kill corosync process.

server02

# killall -9 corosync

When Pacemaker detects that, we can see that fences the other node:

Peer server02 was terminated (reboot) by server01 for server01: OK

Loss-connection

To simulate a Loss-connection, we can block the port through Corosync is messaging, so everynode will think is alone and will take over the resources:

# fw-block

Then, we'll see that node that has the delay option, will be fenced in first place. Now we're safe because the resources are only running in one place.

[INCOMPLETE]