Level: Intermediate

CheKim Chhuor (chhuor@cn.ibm.com), Software Engineer, IBM　

04 Mar 2008

As the Xen open source hypervisor gains traction in many enterprises for production deployment, you'll discover a need to provide fully redundant storage to the Xen environment from the host adapter all the way down to the hard drives. In this article, learn how to use Red Hat Enterprise Linux® 5.1 to set up Xen and multipath storage access to the IBM System Storage™ DS4800.

Virtualization technology was part of the IBM mainframe decades before virtualization gained its current industry adoption. The recent broader adoption of virtualization is due to the fact that enterprise datacenters can no longer keep up with the demands of new applications by adding racks of servers. Datacenters are already overstretched in terms of server count, electricity consumed, and heat generated per square foot of raised floor space.

While virtualization, which reduces server count by increasing utilization efficiency and lowering operating costs, is an excellent solution to the problem of overwhelming demand, it can also add a burden to system availability since more applications are now running in each box. Any failure can now cause more disruption to business.

One way to counter this is by using a fully redundant path to access external SAN (storage area network) devices—in this way, you can reduce failures caused by storage subsystem access problems.

This article uses Red Hat Enterprise Linux 5 update 1 (RHEL 5.1) to show you how to enable multipath access in Xen host and guests with the IBM-provided RDAC (redundant disk array controller) multipath driver for accessing the IBM System Storage DS4800.

Virtualization and Xen

Much virtual ink has been put to virtual paper to explain the various types of virtualization technologies out there, such as hardware partition, logical partition, hypervisor, and so on. Xen is a paravirtualization hypervisor technology. In other words, the Xen hypervisor is a thin layer of software that sits on top of the hardware; each guest operating system's kernel and drivers have to be modified to run on it.

The advantage is that by collaborating between guest OS and hypervisor when making low-level calls, a Xen guest can achieve near native performance in most cases (which is to say that this type of virtualization is the most cost-effective implementation on commodity hardware).

The downside is that only modified kernels are supported, so you can not use it to consolidate older operating systems such as Windows® 2000. However, since virtualization is becoming such a hot selling point, chip manufacturers are rushing to provide further support to this trend so processors that have Intel-VT or AMD Pacifica feature are able to support full virtualization for running unmodified guests on the Xen hypervisor, albeit with some performance hit.

With the Xen hypervisor now at version 3 and widely used, some vendors have decided to include it as part of their Linux distribution. For instance, in Red Hat Enterprise Linux 5 and Novell SUSE Linux Enterprise Server 10, Xen is robust, supported, and last but not least, free!

Back to top

Xen architecture

When you start up a Xen host, the Xen hypervisor takes control of the system first, then it loads the first guest OS, which is called the domain 0 (or Dom0) in Xen lingo. User interacts with Xen through the Dom0. All the device drivers for accessing hardware are also loaded in Dom0. Dom0 provides virtual block device and virtual network device to the other guest OS, which is sometime referred to as domain U (or DomU). From here on, I won't consider Dom0 as a guest OS anymore, although it is essentially just a guest instance with some special capabilities.

Within each guest (DomU), there are some lightweight device drivers for accessing virtual block devices and virtual network devices; they are called frontend drivers to distinguish them from the heavy-lifting backend drivers running in Dom0. Figure 1 shows the Xen architecture.

Actually, starting from Xen 3.0.3, the virtual block frontend is implemented in the module xenblk, while the virtual block backend is handled by the modules blkbk and blktap. The virtual network frontend is handled by the module xennet and the backend by netbk. To see which modules are loaded in your Linux system, use the commands lsmod and modinfo for module information.

Back to top

Multipath SAN storage

As people move storage from direct host attachment into a shared SAN storage infrastructure, they're adding complexity to the picture. What was once a dedicated SCSI or Fibre Channel cable between the host and storage enclosure now is a bunch of FC cables, SAN switches and director, and a storage server with its own controller, disk enclosures, cables, and so on. While the benefits of SAN storage are indisputable, there are also disadvantages due to higher probability of component failures between the hosts and the actual hard drives. Multipath data access is a remedy to this issue.

Essentially, the host has more than one independent channel to access the same logical drive (using the logical unit number, or LUN) on the storage server. A multipath driver combines all the paths to show only one block device to the operating system. Each LUN has a preferred path, and if that fails, the multipath driver routes I/O requests to the alternate paths. Multipath drivers are not made equal: some are able to handle more intelligent load balancing, while others can handle only failover. For the DS4800 storage server, IBM recommends and provides the RDAC driver to use on top of the HBA (host bust adapter) driver. Remember, the multipath driver does not replace the HBA driver: they complement each other. The multipath driver is analogous to the brain, and the HBA driver is analogous to the arms and legs. Modern operating systems also provide their own multipath driver, but make sure it has been certified by your storage vendor before using it.

Back to top

Hardware and software

Enough theory; let's look at the architecture of the test environment. I have an IBM x3550 server with an HBA card that has two FC ports. Each port is connected to a different SAN switch. The DS4800 storage server has two controllers that have four FC ports on each. For simplicity, I show only two FC connections in Figure 2; each one is connected to the same set of switches as the host. So, any interruption to one of the access path does not affect storage access from the host. The RDAC driver installed in domain 0 takes care of using the preferred path in normal circumstances and switching to the alternate path in case of interruption. The guest OS doesn't need any special driver to have multipath access to the backend storage.

Here's the hardware and software in my test environment:

• IBM System x3550
  • 2 Dual-Core Xeon, 4 GB memory
  • QLogic HBA adapter QLE2462
  • Red Hat Enterprise Linux 5 update 1
  • IBM RDAC driver, version 09.01.C5.11
  • QLogic driver, version qla2xxx 8.01.07.15
• IBM System Storage DS 4800 model 1815-84A
  • 2 controllers
  • 16 FC disks of 136GB
  • 8 FC ports at 4Gbps
• Brocade switches, model 2109-F32
  • 32 ports at 2Gbps

Back to top

Setting up the storage

Because the IBM SAN storage servers were designed for multipath access, you don't need to do anything special to make a logical drive accessible from multiple paths. In the case of DS4800, when creating a logical volume, it automatically alternates controller ownership as the preferred path for load balancing. In Figure 3, I created 4 LUNs and mapped them to my host group. Remember that LUN numbers have to be contiguous for the host to be able to scan them all, and it starts at 0. If you remove a logical drive, make sure the other LUN numbers remain contiguous!

At the switch fabric, I put the first WWPN of the host HBA into one zone, along with a WWPN of storage controller A. Then I put the second WWPN of the host HBA with a WWPN of controller B into another zone.

Back to top

Setting up the Xen host

Setting up the Xen hypervisor and domain 0 is straightforward. The procedure goes like this:

1. Install RHEL 5.1 normally on a local drive. You can install for SAN boot as well, but that is outside the scope of this article.
   
   
2. After you have RHEL 5.1 up and running, install the Xen kernel from the installation source, like this:
   
   Listing 1. Installing the Xen kernel packages
   

   rpm -ivh kernel-xen-2.6.18-53.el5.i686.rpm rpm -ivh kernel-xen-devel-2.6.18-53.el5.i686.rpm

   
   
3. Now you should have a new entry in your boot manager, /boot/grub/grub.conf. Make sure it boots the Xen kernel by default. Reboot the system.
   
   
4. Verify that you're now running the Xen kernel after reboot:
   
   Listing 2. Verifying the Xen kernel
   

   [root@xenhost ~]# uname -a Linux xenhost 2.6.18-53.el5xen #1 SMP Wed Oct 10 17:06:12 EDT 2007 i686 i686 i386 GNU/Linux

   
   
5. Congratulation! You now have a Xen hypervisor with Dom0 running on top of it.

Installing the Xen tools

Now that Xen is running, you need to install some tools to interact with it. In the RHEL 5.1 installation source, the Xen tools are in the /VT directory. The basic command-line tools are very powerful but not pretty, and the graphical tool is easy to use but not powerful. That's life! Anyway, let's go ahead and install both of them.

These are the dependent packages that you have to install first; get them from the /Server directory on the DVD or CDs:

                
rpm -ivh bridge-utils-1.1-2.i386.rpm
rpm -ivh dnsmasq-2.39-2.el5.i386.rpm
rpm -ivh gnome-python2-gnomekeyring-2.16.0-1.fc6.i386.rpm
rpm -ivh xen-libs-3.0.3-41.el5.i386.rpm  

When installing the Xen tools packages, it seems to me that there is a circular dependency no matter what order I install them in, so I have to force the first package to install without dependency verification and then install the rest in this order:

                
rpm -ivh --nodeps libvirt-0.2.3-9.el5.i386.rpm
rpm -ivh libvirt-devel-0.2.3-9.el5.i386.rpm
rpm -ivh libvirt-python-0.2.3-9.el5.i386.rpm
rpm -ivh python-virtinst-0.103.0-3.el5.noarch.rpm
rpm -ivh xen-3.0.3-41.el5.i386.rpm
rpm -ivh xen-devel-3.0.3-41.el5.i386.rpm
rpm -ivh virt-manager-0.4.0-3.el5.i386.rpm
rpm -ivh gnome-applet-vm-0.1.2-1.el5.i386.rpm
rpm -ivh Virtualization-en-US-5.1.0-12.noarch.rpm

Alright, let's kick some tires and try some commands now! But before you do that, you have to start the xend daemon. All of the Xen management tools communicate with the xend daemon to extract information. To start it, run xend start as root. Normally the xend service is set to start at system boot, so you don't have to do this every time.

Now you can run the command xm info to see some information from the Xen hypervisor.

                
[root@xenhost ~]# xm info
host                   : xenhost
release                : 2.6.18-53.el5xen
version                : #1 SMP Wed Oct 10 17:06:12 EDT 2007
machine                : i686
nr_cpus                : 4
nr_nodes               : 1
sockets_per_node       : 2
cores_per_socket       : 2
threads_per_core       : 1
cpu_mhz                : 2992
hw_caps                : bfebfbff:20100000:00000000:00000140:0004e3bd:00000000:00000001
total_memory           : 4095
free_memory            : 20
xen_major              : 3
xen_minor              : 1
xen_extra              : .0-53.el5
xen_caps               : xen-3.0-x86_32p hvm-3.0-x86_32 hvm-3.0-x86_32p
xen_pagesize           : 4096
platform_params        : virt_start=0xf5800000
xen_changeset          : unavailable
cc_compiler            : gcc version 4.1.2 20070626 (Red Hat 4.1.2-14)
cc_compile_by          : brewbuilder
cc_compile_domain      : build.redhat.com
cc_compile_date        : Wed Oct 10 16:30:55 EDT 2007
xend_config_format     : 2

Then you can run the command xm list to see the running virtual machines. Naturally at this point, you'll only see the domain 0 running. You can use xentop for this as well.

                
[root@xenhost ~]# xm list
Name                                      ID Mem(MiB) VCPUs State   Time(s)
Domain-0                                   0     2928     4 r-----  61790.3

If those two commands run successfully, then you can be assured that your Xen environment is configured properly. If the commands can't connect to the xend daemon, try to reboot the machine. Alternatively, you can look at the Xen logs in the /var/log/xen directory.

If you prefer a graphical tool, log on to a Gnome session, and then go to the menu Applications > System Tools > Virtual Machine Manager. You should see the status of your virtual machines as in Figure 4.

Installing the multipath driver

Now that the Xen environment is up and running, it's time to install the drivers to access the SAN storage. The first thing to install is the HBA adapter driver, which is the QLogic driver in my case. Download the QLogic driver installation package from the vendor's Web site and install it according to its documentation. You must turn off the default failover option afterward by adding an option in /etc/modprobe.conf as follows:

                
options qla2xxx ql2xfailover=0
      

It's time to install the RDAC multipath driver now. Again, download the installation package from the IBM Web site (see Resources for a link) and follow the documentation to install the driver. It should compile the source code, install the module, and create a new initrd image. You should manually modify your boot manager configuration file /boot/grub/grub.conf to use the new initrd image. Be careful here; do not use the example provided by the installation script, since that isn't applicable to a Xen host. Instead use the current format in grub.conf, and substitute initrd-xyz.img with mpp-xyz.img.

My example looks like this:

                
title Red Hat Enterprise Linux Server with RDAC driver (2.6.18-53.el5xen)
        root (hd0,0)
        kernel /xen.gz-2.6.18-53.el5
        module /vmlinuz-2.6.18-53.el5xen ro root=LABEL=/ rhgb quiet
        module /mpp-2.6.18-53.el5xen.img

After reboot, list all the logical drives with the command /opt/mpp/lsvdev. If you're not seeing all the LUNs, run the command mppBusRescan.

                
[root@xenhost ~]# /opt/mpp/lsvdev
        Array Name      Lun    sd device
        -------------------------------------
        ARCJMTDS48K1    0     -> /dev/sdb
        ARCJMTDS48K1    1     -> /dev/sdc
        ARCJMTDS48K1    2     -> /dev/sdd
        ARCJMTDS48K1    3     -> /dev/sde

Now you should have multipath access to your SAN storage from domain 0.

Back to top

Setting up the Xen guests

Before installing the Xen guests, you need to do two things on the host:

1. Make sure that the virtual network is set up properly.
2. Make the installation media available to the Xen guests through the network, which could be HTTP, FTP, or NFS—but it doesn't take a locally mounted ISO image!

For my virtual network, I bound it to the eth0 adapter specifically. You can use the graphical Virtual Machine Manager (VMM) to do that; go to the menu Edit > Host details > Virtual Networks to modify or add a new one.

For the installation media, I simply extract all the files from the DVD image and put them under /var/www/html/rhel51 directory, then start the Apache Web server with service httpd start. All the files are then accessible within the Xen guests as http://192.168.100.1/rhel51. Make sure you don't have iptable blocking incoming connection to port 80.

Guest test0

For the first guest OS, I want it to have three logical drives mapped directly to three LUNs on the DS4800 through Dom0, of course.

Ready...let's go.

1. Click on the New button at the main panel of VMM.
   
   Figure 7. VMM main panel with domain 0
   
   
   
2. Give a name to the guest, such as test0.
3. Choose the Paravirtualized mode.
4. At the next screen, type:
   http://192.168.100.1/rhel51
   in the Install Media URL field.
5. Assign a Normal Disk Partition to the guest; the first one should be /dev/sdb from the perspective of the host. The remaining disks can be added later.
   
   Figure 8. Assigning storage space to the guest test0
   
   
   
6. Choose the Virtual Network that you've set up previously.
7. Allocate memory to the guest. In my case, I set the startup memory to 512MB and max memory to 1GB. For virtual CPU, I set it to 2.
   
   Figure 9. Allocating memory to the guest test0
   
   
   
8. Here is the summary panel that I have:
   
   Figure 10. Guest test0 creation summary panel
   
   
   
9. Click Finish to start the creation process.
10. At this point, you should see the familiar Red Hat installation program kick in. Just follow through to complete the installation. This takes a while, as usual.
    
    Figure 11. Installation begins at the guest test0
    
    
    
11. At any time during or after the installation, you may add the two remaining logical drives to this guest. At the VMM main panel, select test0 and click Details.
    
    Figure 12. VMM main panel with 1 guest
    
    
    
12. At the Hardware tab, click Add.
    
    Figure 13. Hardware details panel of the guest test0
    
    
    
13. Accept the default choice of Storage device.
14. Choose a Normal Disk Partition and type /dev/sdc.
15. Click Finish to complete the process.
16. Repeat the process to add /dev/sdd.
17. Now we should have three disks defined to the guest.
    
    Figure 14. After adding 2 storage devices to guest test0
    
    
    
18. Find an appropriate time to shut down and restart the guest to make the change effective—rebooting won't do here.

To access the guest OS:

• You can use the graphical console as you did during OS installation.
• From the host terminal session, you can run xm console test0 to access a text console.
• You can ssh to test0 if you know its DHCP-assigned IP address.

To see the new disks, you can run fdisk -l and optionally use fdisk to create partition on them.

The guest configuration is stored in a plain text file in the /etc/xen directory. For example, here's the content of /etc/xen/test0:

                
[root@xenhost ~]# cat /etc/xen/test0
name = "test0"
uuid = "dddf02f6-5f90-74a5-0098-365a51b54282"
maxmem = 1000
memory = 500
vcpus = 2
bootloader = "/usr/bin/pygrub"
on_poweroff = "destroy"
on_reboot = "restart"
on_crash = "restart"
vfb = [ "type=vnc,vncunused=1,keymap=en-us" ]
disk = [ "phy:/dev/sdb,xvda,w", "phy:/dev/sdc,xvdb,w", "phy:/dev/sdd,xvdc,w" ]
vif = [ "mac=00:16:3e:79:2f:e1,bridge=vnet0" ]

Creating this file from scratch isn't easy, but it's fairly easy to modify this file manually. For example, if you want to change the memory allocation or number of virtual CPU, you just need to edit this file and shut down/restart the guest with the command xm shutdown test0, then xm create test0. Or you can use the VMM tool.

For the sake of this exercise, I modified the name associated to each disk in the guest from xvdx to hdx like this:

                
disk = [ "phy:/dev/sdb,hda,w", "phy:/dev/sdc,hdb,w", "phy:/dev/sdd,hdc,w" ]

Actually, it doesn't matter how you name it; you can try sdx as well. The guest simply boots the first block device in the list. After creating partitions and filesystems on the virtual disks, I get this from within test0:

                
 [root@test0 ~]# df -h
Filesystem            	           Size  Used Avail Use% Mounted on
/dev/mapper/VolGroup00-LogVol00  3.8G  1.8G  1.8G  51% /
/dev/hda1                         99M   13M   82M  14% /boot
tmpfs                            251M     0  251M   0% /dev/shm
/dev/hdb1                        2.0G   36M  1.9G   2% /mnt/disk1
/dev/hdc1                        2.0G   36M  1.9G   2% /mnt/disk2

This completes the setup process for test0, which now has multipath access to three LUNs on the DS4800.

Guest test1

For the second guest, I want to map image files in Dom0 as virtual disks to the guest. The LUN #3 on DS4800 is made available to Dom0 as /dev/sde. I create a partition on it using fdisk, then create a filesystem on that partition using mkfs. Then I mount the filesystem to the default Xen image file location at /var/lib/xen/images.

Now let's proceed to create the guest test1 with the same procedure as earlier, by clicking New in the VMM main panel, giving it the System Name of test1, and choosing Paravirtualized. At the storage assignment screen, choose Simple File with the location of /var/lib/xen/images/test1-xvda.img. Give it a size and check the option Allocate entire virtual disk now.

Then, step through the other screens as we did before; the OS installation should kick in again at the end. During the installation, open the VMM test1 details panel to add two more disks using the image file name of /var/lib/xen/images/test1-xvdb.img and /var/lib/xen/images/test1-xvdc.img. Then you should have three disks as shown in Figure 17.

This is the configuration file that I have for test1:

                
[root@xenhost ~]# cat /etc/xen/test1
name = "test1"
uuid = "53b39c1e9edc6143a06d4011154beab9"
maxmem = 1000
memory = 600
vcpus = 2
bootloader = "/usr/bin/pygrub"
on_poweroff = "destroy"
on_reboot = "restart"
on_crash = "restart"
vfb = [ "type=vnc,vncunused=1,keymap=en-us" ]
disk = [ "tap:aio:/var/lib/xen/images/test1-xvda.img,xvda,w", 
 "tap:aio:/var/lib/xen/images/test1-xvdb.img,xvdb,w", "tap:aio:/var/lib/xen/images/
 test1-xvdc.img,xvdc,w" ]
vif = [ "mac=00:16:3e:7b:78:63,bridge=vnet0" ]

The tag tap:aio is used to represent a logical disk based on the image file, while the tag phy represents a physical disk. After OS installation, shut down and restart test1, and you should see the two added disks with fdisk -l. After creating partitions and filesystems on these disks, I have this in test1:

                
[root@test1 ~]# df -h
Filesystem                       Size  Used Avail Use% Mounted on 
/dev/mapper/VolGroup00-LogVol00  4.7G  1.8G  2.9G  39% /
/dev/xvda1                        99M   13M   82M  14% /boot
tmpfs                            301M     0  301M   0% /dev/shm
/dev/xvdb1                       2.0G   35M  1.8G   2% /mnt/disk1
/dev/xvdc1                       2.0G   35M  1.8G   2% /mnt/disk2

At this point, both guests are fully installed and configured to access their respective virtual disks with multipath through domain 0. My VMM is showing this now:

Back to top

Pulling some cables

To make sure that multipath access is functional in both guests, I run an IBM internal I/O workload tool in each guest. It writes block of data, then reads and verifies continuously on all three virtual disks. Then I purposely interrupt one of the access paths by pulling off an FC cable between the host and switch. The I/O tool freezes for about 5 seconds, and then resumes activities as usual.

To simulate a SAN switch failure, I disable a port on the other switch where the host is connected to, and again, the I/O tool continues to run normally.

Finally, I disable one of the controllers on the DS4800 and as expected, the I/O tool continues to run flawlessly! Obviously, between each test, I restore the environment back to a functional state and wait a few minutes for the multipath driver to reestablish its connection to the broken path.

Back to top

Conclusion

Share this... Digg this story Post to del.icio.us Slashdot it!

In this article, you've seen how to set up a Xen environment with multipath access to the IBM DS4800 storage server. You've learned how multipath access can be used to dramatically increase the uptime of Xen virtual machines in the event of storage path failure or storage infrastructure maintenance.

If a dual path does not give you sufficient availability and bandwidth, you can have four or up to eight paths! The high-end storage servers even have the intelligence to load-balance access paths if you have a need for that. So, if you're building a virtualized environment with SAN storage, you should definitely consider multipath access in your strategy to achieve high availability, reliability, and serviceability.

Resources

• In "Virtual Linux" (developerWorks, December 2006), take a tour of virtualization methods, architectures, and implementations.
  
  
• "Virtualization with coLinux" (developerWorks, March 2007) introduces coLinux for Linux/Windows virtualization, and gives you another path to virtualization.
  
  
• Downloads for DS4800 Midrange Disk System provides support and downloads, including firmware, storage management and host bus adapter software, and management and diagnostic tools.
  
  
• Xen.org delivers up-to-date info on the Xen hypervisor, which provides a feature set for virtualization of x86, x86_64, IA64, PowerPC™, and other CPU architectures, as well as a wide range of guest operating systems including Windows, Linux, Solaris, and various versions of the BSD operating systems.
  
  
• In the developerWorks Linux zone, find more resources for Linux developers, and scan our most popular articles and tutorials.
  
  
• See all Linux tips and Linux tutorials on developerWorks.
  
  
• Stay current with developerWorks technical events and Webcasts.
  
  

• Order the SEK for Linux, a two-DVD set containing the latest IBM trial software for Linux from DB2®, Lotus®, Rational®, Tivoli®, and WebSphere®.
  
  
• With IBM trial software, available for download directly from developerWorks, build your next development project on Linux.
  
  

• Get involved in the developerWorks community through blogs, forums, podcasts, and community topics in our new developerWorks spaces.

About the author

		CheKim Chhuor works on the OSL storage testing team in the Systems and Technology Lab at IBM in Shanghai, China. His current focus is on storage technologies and systems architecture. Previously he worked at IBM in Poughkeepsie, New York, on various system management products and at IBM in Montreal, Canada, as a consultant. Chhuor has many years of Web infrastructure consulting experience and holds many IBM certifications on WebSphere®, DB2®, and On Demand Business.

Original link: http://www.ibm.com/developerwork...