In RHV, KVM Host uses Sanlock to detect the connectivity with Storage domain. When Sanlock enables, it will open wdmd daemon(watchdog multiplexing daemon) and send keepalive with certain heartbeat. sanlock IO to storage does not complete within a fixed time, sanlock stop sending keepalive to wdmd. When it is timeout,the wdmd daemon controlling /dev/watchdog will log errors, warning that the watchdog is not being kept alive, and will soon expire and then fire and reset the KVM host.

$ systemctl status sanlock
● sanlock.service - Shared Storage Lease Manager
Loaded: loaded (/usr/lib/systemd/system/sanlock.service; disabled; vendor preset: disabled)
Active: active (running) since Mon 2020-07-13 15:03:26 NZST; 1 months 19 days ago
Process: 1041 ExecStart=/usr/sbin/sanlock daemon (code=exited, status=0/SUCCESS)
Main PID: 1044 (sanlock)
Tasks: 7
Memory: 18.6M
CGroup: /system.slice/sanlock.service
├─1044 /usr/sbin/sanlock daemon
└─1045 /usr/sbin/sanlock daemon

Warning: Journal has been rotated since unit was started. Log output is incomplete or unavailable.
$ systemctl status wdmd
● wdmd.service - Watchdog Multiplexing Daemon
Loaded: loaded (/usr/lib/systemd/system/wdmd.service; disabled; vendor preset: disabled)
Active: active (running) since Mon 2020-07-13 15:03:26 NZST; 1 months 19 days ago
Process: 1131 ExecStart=/usr/sbin/wdmd (code=exited, status=0/SUCCESS)
Process: 1112 ExecStartPre=/lib/systemd/systemd-wdmd watchdog-check (code=exited, status=0/SUCCESS)
Main PID: 1133 (wdmd)
Tasks: 1
Memory: 2.3M
CGroup: /system.slice/wdmd.service
└─1133 /usr/sbin/wdmd

Warning: Journal has been rotated since unit was started. Log output is incomplete or unavailable.

The sanlock daemon continually writes to storage at a fixed interval to renew its leases. Sanlock will mark KVM host as UNKNOWN, FREE, LIVE, FAIL and DEAD status based on sanlock IO to storage result.

unknown:

KVM host has a lease on the storage, but the clusterlock cannot tell if the host is live or dead yet. Would typically last for 10-20 seconds, but it’s possible that this could persist for up to 80 seconds before the KVM host is considered live or fail.

Free:

There is no lease for this host id.

Live:

KVM host has renewed its lease in the last 80 seconds. It may be renewing its lease now or not, we can tell that only by checking again later.

Fail:

Host has not renewed its lease for 80 seconds. Would last for 60 seconds before host is considered dead. When KVM host staus is marked as ‘Fail’, you will below related logs:

2020-08-31 21:35:01 1665117 [1044]: s1 check_our_lease warning 72 last_success 1665045
2020-08-31 21:35:02 1665118 [1044]: s1 check_our_lease warning 73 last_success 1665045
2020-08-31 21:35:03 1665119 [1044]: s1 check_our_lease warning 74 last_success 1665045
2020-08-31 21:35:04 1665120 [1044]: s1 check_our_lease warning 75 last_success 1665045
2020-08-31 21:35:05 1665121 [1044]: s1 check_our_lease warning 76 last_success 1665045
2020-08-31 21:35:06 1665122 [1044]: s1 check_our_lease warning 77 last_success 1665045
2020-08-31 21:35:07 1665123 [1044]: s1 check_our_lease warning 78 last_success 1665045
2020-08-31 21:35:08 1665124 [1044]: s1 check_our_lease warning 79 last_success 1665045
2020-08-31 21:35:09 1665125 [1044]: s1 check_our_lease failed 80
2020-08-31 21:35:10 1665125 [1044]: s1 all pids clear
2020-08-31 21:35:21 1665137 [3859]: 8d627013 aio timeout RD 0x7f56e00009b0:0x7f56e00009c0:0x7f56f0299000 ioto 10 to_count 4
2020-08-31 21:35:21 1665137 [3859]: s1 delta_renew read timeout 10 sec offset 0 /rhev/data-center/mnt/[mountpoint]/[SD_UUID]/dom_md/ids
2020-08-31 21:35:21 1665137 [3859]: s1 renewal error -202 delta_length 20 last_success 1665045

Dead:

Host has not renewed its lease for 140 seconds.

If sanlock IO to storage does not complete within a fixed time, the sanlock daemon will enter recovery. Recovery begins with the sanlock daemon attempting to kill(SIGTERM) any pids using leases on the affected storage. If any pid does not exit after 10 SIGTERM’s over 10 seconds, sanlock will then attempt kill(SIGKILL). If pids still do not exit within a fixed time, the watchdog will fire, resetting the host. If all pids do exit within the necessary time, the watchdog will be renewed and will not fire.