Identifying Strategies to Reduce Single Points of Failure in RedHat Virtualization
Improving resilience of RedHat Virtualization
As a mission-critical environment, an RHV environment requires the right choice of the underlying infrastructure. Hardware choices for both RHVM and RHV hosts are important to avoid problems, but the configuration of the networking and storage infrastructures are also very important.
A physical data center supporting a mission-critical environment like Red Hat Virtualization should be configured in a mission-critical way. This might include:
- Redundant power to all components.
- Redundant power in the data center.
- Redundant network providers to the data center
Make sure your environment is completely ready for installation. Avoid delays in your deployment by ensuring your server setup is complete, network switches are configured correctly, and you have provisioned plenty of storage.
DNS is critical for RHV to operate correctly. Ensure that forward and reverse name resolution is functioning correctly for hosts and the Red Hat Virtualization Manager and that fully qualified domain names are used.
A number of services, especially related to authentication and TLS/SSL certificates, are sensitive to time skew issues. Use NTP service to ensure that system clocks are synchronized.
If you are using an external authentication provider for RHVM users, such as Red Hat Identity Management (FreeIPA) or Microsoft Active Directory, ensure that it is highly available so that users can access the web portals. You can use local internal authentication profile users, such as the built-in admin superuser, to provide emergency access in case of an authentication outage.
Make sure you properly test your RHV deployment before you go into production.
Using a Standard Hardware Platform for Hosts in a Cluster
Clusters must support a consistent CPU family since they are migration domains. To ease troubleshooting of an RHV environment, it is a good idea to take this a step further and use the same vendor and model of server with the same configuration for all hosts in a cluster. This limits inconsistency and misconfiguration by making all hosts in the cluster as identical as possible. Homogeneous hardware at the cluster level also helps provide consistent performance in the environment, especially when virtual machines migrate from one cluster host to another. Make sure that hardware, such as CPUs (family and number), network interfaces, host bus interfaces (HBA), and RAID cards, are the same across all of the cluster’s hosts. It’s also a good idea to make sure that the firmware and BIOS is up-to-date and on the same version on all hosts in the cluster.
Select the right storage platform is key to avoid unnecessary issues. Good storage performance is critical to the overall performance of the RHV environment. If storage infrastructure choices are not correct, no amount of memory or CPU at the host or VM level can make up for it. Storage infrastructure design should take into account needs for data backup, data replication, and application workload.
A storage environment for RHV should include the following configuration:
- Redundant Ethernet or fibre channel (FC) switches for your storage networks.
- If using iSCSI or NFS, then multiple NICs should be used and bonded.
- If using a SAN, multiple HBAs (FC) or initiators (iSCSI) should be used to provide multiple paths to the SAN. Make sure you use the same make, model, firmware version, and driver versions in the same systems and clusters to ensure consistent performance and ease troubleshooting.
- Consider using SAN-based boot if there is already a SAN available to store VMs. This configuration avoids issues related to a local host’s storage and improves performance on tasks like hypervisor images cloning, speeding up virtual machine deployment times.
A networking infrastructure for RHV should include the following configuration:
- Use redundant network switches.
- Use bonded network interfaces, preferably in LACP mode.
NOTE:: While Red Hat Enterprise Linux and Red Hat Virtualization support all bonding modes (0-6), logical networks that support VM traffic can only use modes 1, 2, 3, or 4. Modes 0, 5, and 6 do not support the Linux bridge needed for VM networks.
– If using Ethernet, plan at least 10GbE links for VM traffic and any Ethernet storage traffic to avoid network traffic congestion issues. Use 40GbE links if available, potentially partitioning them using VLANs as needed.
– Segregate different traffic types, like VM traffic, using Virtual LANs (VLANs). Grant different VLANs priority and available bandwidth based on their traffic, like VM live migration, user-to-VM communication or communication with the engine.
Networks for storage and VM live migration generally need high bandwidth and may need dedicated networks for performance and security. The bandwidth needed for virtual machine traffic varies depending on your applications. RHVM management traffic and console display traffic is relatively low bandwidth and can use slower networks.
VLANs, 40GbE networking, and advanced quality-of-service settings in RHV can be used together to efficiently and flexibly manage physical network configuration while segregating types of traffic and controlling bandwidth appropriately.
Configuring Network Bonds on RHVH
Configuring two NICs as a bonded interface on RHVH is simple and can be configured in the Administration Portal after hosts have been added to Red Hat Virtualization Manager.
On the Hosts tab, select a host from the list and select its Network Interfaces tab at the bottom of the interface. Click Setup Host Networks, just as you would to configure logical networks.
In the window, drag the icon for one physical interface onto the other interface you want to bond. This opens the Create New Bond window. Select a Bond Name and Bonding Mode, and then click OK. You can treat the new bonded interface just like any other interface, adding and removing logical networks as desired.
Note: Remember to configure your networking hardware as needed to support your bonding mode. For example, the default mode used by RHV, IEEE 802.3ad/ LACP (mode 4), requires bonding in that mode to be enabled for the switch ports
connected to the participating NICs. You also need to remember to configure your switch ports to permit the correct VLANs to be passed to the interfaces on your hosts.
RHV supports hosts based on Red Hat Enterprise Linux as well as Red Hat Virtualization Host.Red Hat Enterprise Linux-based hosts can be useful for environments requiring customization at the OS level, for example, because of hardware support. However, because of the manual configuration and updates performed on those hosts, Red Hat Enterprise Linux based hosts can cause unexpected issues in an RHV environment.
Red Hat recommends Red Hat Virtualization Host as the preferred operating system for hosts, because of the following features:
- Only the required packages and services supporting VMs and the hypervisor are part of RHVH.This approach reduces operating system overhead. As an additional benefit, it also reduces the overall security “attack surface” by restricting the default configuration.
- The latest version of RHVH allows you to install additional RPM packages if you need them, which reduces the need for “thick” Red Hat Enterprise Linux-based hosts.
- RHVH includes the recommended configuration for an RHV host, so it does not require any manual configuration. This approach eliminates issues related to manual configuration of a system.
- RHVH includes the Cockpit web administration tool pre-installed. This tool improves the troubleshooting of issues related to a host and its VMs.
Note: Use RHVH whenever possible to minimize the number of services installed in a host,and to ease troubleshooting. Only use a full host installation to support hardware that requires a certain software feature not supported in RHVH.
A RHV host should also include:
- Available out-of-band (OOB) management to enable features like remote power control.
- Up-to-date hardware firmware and BIOS.
- Memory scaled to avoid memory swapping, which significantly degrades VM performance.
- RAID configuration of the host’s local boot disks to reduce the chance of VMs going down due to host failure
You should perform backups of RHVM on a regular basis. Although an all-in-one (default) RHVM installation is the preferred approach for deployment of Red Hat Virtualization Manager, for certain scenarios you may want to run some RHVM components on separate hosts for higher performance. It is possible to deploy RHVM components, like the PostgreSQL database, the data warehouse, and the websocket proxy, to other hosts. This does complicate RHVM deployment and requires careful thought about redundancy, availability, and backup scenarios.
Another strategy is to deploy RHVM as a self-hosted engine running as a virtual machine on one of its hosts. In combination with highly available virtual machines, this can provide a high availability solution for RHVM. The remainder of this chapter looks at these topics.