rcron is a powerful tool that helps system administrators in setting up cron jobs redundancy and failover over groups of machines. RCRON ensures that a job installed on several machines will only run on the active one at any time.

Working of RCRON

High Availability using RCRON (One Node will be stamped as Active and Second Node will be stamped as Passive), Same cron configuration will be on both, the only difference would be active/passive state in a file.

For automatic Switching of active/passive state, we will be using KEEPALIVED Daemon, which utilizes the keepalive signal for communication between 2 nodes. After a signal is sent, if no reply is received the link is assumed to be down.

In this scenario, One node is marked as KEEPALIVED master and the second as KEEPALIVED backup. The master node will keep the rcron state active, and the backup node will keep the rcron state passive. As soon as the Master Node comes down, it will send a 0 priority signal to the Backup Node, which will switch over and behave as a MASTER NODE and consecutively mark rcron as active as soon as the master node comes up again, the Backup Node will switch over to backup mode.

Installation and Configuration Steps of RCRON On CentOS/RHEL

In My scenario I have Two Linux Machines, one is Primary and the second one is Slave.

Step1: Enable IP Forwarding on both the Nodes. Edit the file /etc/sysctl.conf & make the below change

net.ipv4.ip_forward = 1
# sysctl -p      ; Save the Changes without rebooting the Machine

Step 2: Add the EPEL Repository On Both Nodes.

# wget http://dl.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm
# rpm -ivh epel-release-6-8.noarch.rpm

Step 3: Install the necessary packages on Both the Nodes.

# yum install subversion byacc flex gcc

Step 4: If You getting internet from the proxy server, then set http_proxy for svn.

# mkdir /root/.subversion/
# vi ~/.subversion/servers
http-proxy-exceptions = *.example.com
http-proxy-host = www.example.com
http-proxy-port = 8080
http-compression = no

Step 5: If we are getting Internet connection directly, then skip proxy settings. Download the rcron software from SVN.

# cd /root;
# svn co http://rcron.googlecode.com/svn/trunk rcron
# cd rcron/rcron
# ./configure
# make
# make install

Step 6: Create the directory on both the nodes.

##############################################
For MASTER SERVER Setup RCRON.conf as follows
##############################################
# vi /etc/rcron/rcron.conf
# An arbitrary name
cluster_name        = cluster
# A file containing either the word "active" or the word "passive"
state_file          = /var/run/rcron/state
# The default state in case state_file can't be read
#default_state       = active
syslog_facility     = LOG_CRON
syslog_level        = LOG_INFO
# We can tune jobs niceness/priorities 
nice_level          = 19
+++++++++++++++++++++++++++++++++++++++++++++++
For BACKUP SERVER NODE , Setup RCRON.conf as follows
+++++++++++++++++++++++++++++++++++++++++++++++
vi /etc/rcron/rcron.conf;
# An arbitrary name
cluster_name = cluster
# A file containing either the word "active" or the word "passive"
state_file = /var/run/rcron/state
# The default state in case state_file can't be read
#default_state = passive
syslog_facility = LOG_CRON
syslog_level = LOG_INFO
# We can tune jobs niceness/priorities 
nice_level = 19

On the Master Node Run Below Commands:

# mkdir /var/run/rcron
# touch /var/run/rcron/state
# echo "active" > /var/run/rcron/state

On the Slave Node Run the Below Commands:

# mkdir /var/run/rcron
# touch /var/run/rcron/state
# echo "passive" > /var/run/rcron/state

Step 7: Now install the keepalive package on both the nodes.

a) Download the latest source tarball:

# cd /root
# wget http://cgit.luffy.cx/keepalived/snapshot/keepalived-1.2.7.tar.gz

b) Install the RPM BUILD package:

# yum -y install rpm-build

c) Untar the tar ball & edit keepalived.spec.in file:

# tar -zxvf keepalived-1.2.7.tar.gz
# mkdir -p /root/rpmbuild/SOURCES/
# cp /root/keepalived-1.2.7.tar.gz /root/rpmbuild/SOURCES/

replace from Version 1.2.2 to Version: 1.2.7:

# vi /root/keepalived-1.2.7/keepalived.spec.in
# yum -y install popt*
# cd /root/keepalived-1.2.7
# rpmbuild -ba keepalived.spec.in

Above Command will create a compiled RPM from Source in the location:

/root/rpmbuild/RPMS/x86_64/keepalived-1.2.7-5.x86_64.rpm

Now Install rpm with rpm command:

# rpm -ivh /root/rpmbuild/RPMS/x86_64/keepalived-1.2.7-5.x86_64.rpm

Step 8: SETUP the keepalived configuration.

For the master node, setup the following:

# vi /etc/keepalived/keepalived.conf
vrrp_instance VI_1 { 
state MASTER 
interface eth0 
lvs_sync_daemon_inteface eth0 
virtual_router_id 31 
priority 101 
advert_int 5
vrrp_unicast_bind 172.16.243.144 
vrrp_unicast_peer 172.16.243.145 
authentication { 
auth_type PASS 
auth_pass 1111
}
notify_backup "/bin/echo passive > /var/run/rcron/state" 
notify_master "/bin/echo active > /var/run/rcron/state" 
notify_fault "/bin/echo passive > /var/run/rcron/state"
}

For the backup node:

# vi /etc/keepalived/keepalived.conf
vrrp_instance VI_1 {
state BACKUP 
interface eth0 
lvs_sync_daemon_inteface eth0
virtual_router_id 31
priority 100 
advert_int 5 
vrrp_unicast_bind 172.16.243.145 
vrrp_unicast_peer 172.16.243.144
authentication { 
auth_type PASS 
auth_pass 1111
}
notify_backup "/bin/echo passive > /var/run/rcron/state" 
notify_master "/bin/echo active > /var/run/rcron/state" 
notify_fault "/bin/echo passive > /var/run/rcron/state"
}

Where “172.16.243.145” and “172.16.243.144” are the IP address of Linux Machines.

Step 9: Create Sample crontab entry on both master and slave.

* * * * * /usr/local/bin/rcron --conf /etc/rcron/rcron.conf echo `date` >> /tmp/output

Start the Keepalive service on both the nodes:

# service keepalived start ; chkconfig keepalived on

Note: In case the keepalived daemon gets killed we are going to create a crontab script.

* * * * * * /root/keep-alive-monitor

Contents of the script:

# cat /root/keep-alive-monitor
#!/bin/sh
echo "test" >> /tmp/monitor;
ps -ef|grep -v grep|grep -i keepalived;
if [ $? -eq 0 ] ; then 
exit 0
else 
echo "passive" > /var/run/rcron/state;
/etc/init.d/keepalived restart;
fi