IBM System Storage™ DS3000/DS4000/DS5000 Linux RDAC multipath driver package version 09.03.0C05.0504 FOR USE IN LINUX 2.6 KERNEL (RedHat 5.u5 and 5.u6, RedhHat 6.0, SLES10 SP3 and SP4 and SLES11 SP1) OPERATING SYSTEM ENVIRONMENTS ONLY. Note: For other Linux 2.4 and 2.6 kernel operating system (OS) environments, please refer to the IBM System Storage Interoperation Center (SSIC) web site for the supported IBM DS Linux RDAC package for that OS environment. (C) Copyright International Business Machines Corporation 1999, 2011. All rights reserved. US Government Users Restricted Rights - Use, duplication, or disclosure restricted by GSA ADP Schedule Contract with IBM Corp. Note: Before using this information and the product it supports, read the general information in section 6.0 "Trademarks and Notices" in this document. Last Update Date: 07/18/2011 Note: Refer to the IBM System Storage™ Support Web Site or CD for the IBM System Storage DS Storage Manager version 10 Installation and Host Support Guide. This guide along with the Storage Manager program Online Help provide the installation and support information. This guide also provides information on other DS3000/DS4000/DS5000 related publications. These publications provide useful information on the installation of all DS3000DS4000/DS5000 Storage Manager host software packages including the RDAC package. Products Supported: ----------------------------------------------------------------- | New Model | Old Model | Machine Type | Model | |----------- |-----------|--------------|-------------------------| | DS5300 | N/A | 1818 | 53A | |----------- |-----------|--------------|-------------------------| | DS5100 | N/A | 1818 | 51A | |----------- |-----------|--------------|-------------------------| | DS5020 | N/A | 1814 | 20A | |----------- |-----------|--------------|-------------------------| | DS4800 | N/A | 1815 | 82A, 82H, 84A, 84H, | | | | | 88A, 88H, 80A, 80H | |----------- |-----------|--------------|-------------------------| | DS4700 | N/A | 1814 | 70A, 70H, 72A, 72H, | | | | | 70T, 70S, 72T, 72S, | |----------- |-----------|--------------|-------------------------| | DS4500 | FAStT 900 | 1742 | 90X, 90U | |----------- |-----------|--------------|-------------------------| | DS4400 | FAStT 700 | 1742 | 1RX, 1RU | |----------- |-----------|--------------|-------------------------| | DS4300 | FAStT 600 | 1722 | 60X, 60U, 60J, 60K | | | | | 60L | |----------- |-----------|--------------|-------------------------| | DS4200 | N/A | 1814 | 7VA, 7VH | |----------- |-----------|--------------|-------------------------| | DS4100 | FAStT 100 | 1724 | 100, 1SC | |----------- |-----------|--------------|-------------------------| | DS3950 | N/A | 1814 | 94H, 98H, | |------------|-----------|--------------|-------------------------| | DCS3700 | N/A | 1818 | 80C | |----------- |-----------|--------------|-------------------------| | DS3500 | N/A | 1746 | C2A, A2S, A2D, C4A, | | | | | A4S, A4D | |----------- |-----------|--------------|-------------------------| | DS3200 | N/A | 1726 | 21X, 22X, 22T, HC2, HC6 | |----------- |-----------|--------------|-------------------------| | DS3300 | N/A | 1726 | 31X, 32X, 32T, HC3, HC7 | |----------- |-----------|--------------|-------------------------| | DS3400 | N/A | 1726 | 41X, 42X, 42T, HC4, HC8 | ----------------------------------------------------------------- Note: This supported subsystem list might change. Please refer to the IBM SSIC web site for the Linux RDAC version that is supported for a given subsystem in a given Linux kernel/patch version. http://www.ibm.com/systems/support/storage/ssic/ CONTENTS --------- 1.0 Overview 2.0 Installation and Setup Instructions 3.0 Configuration Information 4.0 Unattended Mode 5.0 Web Sites and Support Phone Number 6.0 Trademarks and Notices 7.0 Disclaimer 1.0 Overview: 1.1 Description The Linux RDAC provides redundant failover/failback support for the logical drives in the DS3000/DS4000/DS5000 storage subsystem that are mapped to the Linux host server. The Linux host server must have Fibre Channel (FC) connections to the host ports of both controllers A and B of the DS3000/DS4000/DS5000 storage subsystem. It is provided as a redundant path failover/failback driver alternative to the Linux FC host bus adapter failover device driver. There are many versions of the Linux RDAC driver. The version (09.03.0C05.0504) that this README is packaged with is for Linux 2.6 kernel versions RedHat 6, RedHat 5.u5 and 5.u6, SLES10 SP3 and SP4 and SLES11 SP1 only. For other Linux 2.4 and 2.6 kernel operating system (OS) environments, please refer to the IBM System Storage Interoperation Center (SSIC) web site for the supported IBM DS Linux RDAC package for that OS environment. Please refer to the Dependencies section for the required code levels and Linux kernel versions. In addition, please refer to the http://www.ibm.com/systems/support/storage/ssic/ web site and the IBM System Storage™ Disk Storage Systems Technical Support web site for the latest supported DS3000/DS4000/DS5000 storage subsystem models, host adapters, device drivers, Linux kernel versions and updated readme. Note: Deviations from the listed code levels are not supported. Please contact IBM Support or representatives for help if required. 1.2 Limitations New limitations for Linux RDAC multipath driver package version 9.03.0C05.0504. None Legacy Limitations that still apply 1. This version of DS3000/DS4000/DS5000 Linux RDAC will not support any Linux 2.4 kernels like SuSE SLES 8 or Redhat 3 Linux operating system on Intel/AMD architecture or Linux on POWER (LoP) servers. Please refer to the Linux RDAC package driver version 09.00.A5.XX instead. 2. The Linux RDAC driver cannot co-exist with a Fibre Channel HBA-level multipath failover/failback driver. Note: You might have to modify the driver make file for it to be compiled in the non-failover mode. 3. SteelEye's™ LifeKeeper® clustering is supported on LoP, Intel or AMD architecture processors servers. However, it is currently supported with a maximum of 8 luns only. Please use the following additional instructions when setting up the LifeKeeper® cluster in servers with Intel or AMD architecture processors or LoP servers. Also refer to the publications from Steeleye Technology Inc for more information on the cluster configuration/setup. a. Recompile the Fibre Channel Host Bus Adapter driver using the following setting. - Change the setting QLA2X_PERFORMANCE to 0 in the qla_settings.h file. b. Change the following setting in the /etc/default/LifeKeeper file. LCMHBEATTIME=5 LCMNUMHBEATS=16 c. Reboot the Linux host after completing steps a and b. 4. All limitations which apply to the current DS3000/DS4000/DS5000 Storage Server when configured with Linux servers also apply to this Linux RDAC release. 5. In the DS4000 and DS5000 storage subsystem, Auto Logical Drive Transfer(ADT/AVT) mode, is automatically enabled in the Linux storage partitioning host type. This mode causes contention when an RDAc is installed. If using the "Linux" host type, it has to be disabled using the script that is bundled in this Linux RDAC web package or in the \Scripts directory of this DS Storage Manager version 10 support for Linux CD. The name of the script file is DisableAVT_Linux.scr. Open the DS Storage Manager Enterprise Management window, - Highlight the subsystem - Select Tools - Select Execute script. - A script editing window opens. In this window - Select File - Select Load Script - Give the full path name for the script file (eg . /scripts/DisableAVT_Linux.scr) and click OK - Select Tools - Select Verify and Execute Note: This is assumed that the "Linux" host type is selected for the Storage Partitioning partition that the Linux host server HBA port is defined in. This "Linux" host type has ADT/AVT enabled as default. Please refer to the IBM System Storage DS Storage Manager version 10 Installation and Support Guide for appropriate OS or the Enterprise Window Online help for more information on how to execute a script. 6. The Linux SCSI layer does not support skipped (sparse) LUNs. If the mapped LUNs are not contiguous, the Linux kernel will not scan the rest of the LUNs. Therefore, the LUNs after the skipped LUN will not be available to the host server. The LUNs should always be mapped using consecutive LUN numbers. The SMutil hot_add command can be used to scan for sparse LUNs. However, after the system is reboot, the LUNs after the skipped LUN might not be available until the hot-add utility was performed. 7. Fibre channel configuration. If a host server has multiple HBA ports and each HBA port sees both controllers (via an un-zoned switch), the Linux RDAC driver may return I/O errors during controller failover. The recommendation is to use multiple unconnected FC switches to zone the FC switch into multiple zones so that each HBA port sees only one controller in a DS4000/DS5000 storage subsystem. 8. The RDAC driver reports I/O failures immediately after all paths are failed. When the RDAC driver detects that all paths to a DS3000/DS4000/DS5000 storage subsystem are failed, it will report I/O failure immediately. This behavior is different from the IBM Fibre Channel (FC) HBA failover device driver. The FC HBA failover device driver will wait for a certain time out/retry period before reporting an I/O failure to the host application. There is no work-around. 9. The virtual HBA driver can not be loaded if there are no storage arrays attached. The work-around is to connect the host server to the DS4000/DS5000 storage subsystem and run the "mppBusRescan" utility, which is previously known as the "hot_add" utility. Note: If no LUN is mapped to the host server partition on the DS3000/DS4000/DS5000 storage subsystem, "mppBusRescan"/"hot_add" will not load the mpp_Vhba module. If the server did not recognize any new mapped LUNs after running the "mppBusRescan" utility a few times, consider rebooting the server. 10. The cluster support feature in the Linux RDAC driver is only available for controller 05.4x.xx.xx or later firmware. If a SCSI-2 reserve/release command is addressed to a LUN on an array running a firmware version lower than 05.40, a check condition 05-02-00 (key-asc-ascq) (illegal request, invalid command operation code) will be returned. LoP servers do not support clustering with this Linux RDAC. 11. (For LoP Servers) The "modprobe -r" command is not supported with this Linux RDAC release on SLES 9 when the Linux RDAC driver is installed. Use "rmmod" to remove and recover the device driver stack. On LoP pSeries servers with more than 3 processors, using the modprobe -r command may result in a hung server or panic's. 12. (For LoP Servers) SAN Boot is not supported on IBM JS20 server blades with the initial release SLES9 kernel. The SAN Boot is only supported on JS20 server blades with SLES9 w/SP1 kernel installed. (6-3-2005) 13. Universal Access Luns (UTM LUNs) should not be assigned to LUN ID 0, especially with RDAC's installed. Placing an Universal Access Lun at LUN id 0, may lead to loss of LUN recognition of next sequence LUN ids or partial list of Virtual Luns reported with the RDAC driver. These UTM LUNs should be assigned LUN ID 31. 14. (For LoP Servers) HBA Hot-Swap command to remove a live, fully functioning FC HBA, will cause a system panic if the I/O is not diverted from that path first. This functionality is only supported on current 2.6 kernel versions. The work-around is to pull the Fibre Channel cable first, wait 2 - 5 minutes for fail-over to complete, then issue the "drslot_chrp_pci -r -s slot-number" command. 15. The RDAC driver stack, which includes the mppUpper, Physical HBA driver and the mppVhba, should not be loaded or unloaded using the modprobe kernel module utility. Any use of modprobe on the RDAC driver stack is entirely the responsibility of the user. MPP does not recommend or support the use of modprobe with the RDAC driver stack. Recommendation: Reboot the system whenever a need to unload the driver stack arises. Use of rmmod to remove all the drivers in the RDAC driver stack, in order, is also not supported. Reasoning: MPP creates special initrd image with the required driver modules and their module parameters.User should be aware of the correct module settings and order to perform the modprobe. Since the unloading of the mpp drivers and physical hba drivers will lose the connectivity to storage the best way is to reboot the system. See section "Physical HBA driver update" for correct procedure. (added 6/11/2010) 16. - Rootboot/SAN boot is not supported in server with IA63 processors - CR109190 - In IA64 servers with QLogic HBA, failover occurs immediately on switch port fail in RHEL5.4 - CR162496. - SMagent in Storage Manager version 10.60 gets terminated with error messages when detecting storage in IA64 servers. Do not install SMagent in IA64 servers. 17. RH5 update 4 is not supported in servers with LoP version of the operating systems. Please check the www.ibm.com/systems/support/storage/config/ssic/index.jsp IBM System Storage™ Disk Storage Systems Interoperation Center web site for this support in the future. 18. For x86 or x86_64 servers, support for RH5 update 4 (RH5u4) and SLES 10 SP3 (SLES10SP3) is in FC environment with Linux RDAC version 09.03.0C05.0331. For RH5u4 and SLES10SP3 with SAS environment, use the Linux RDAC version 09.03.0C05.0439 or greater instead. There is not any support for RH5.4 nor SLES10SP3 in DS3K iSCSI environments. (added 6/13/2011) 19. KDUMP does not work in RHEL 6 with MPP driver installed. RHEL 6.0 has changed the way kdump image is created and hangs when executed with the Mpp driver installed. A bugzilla was opened for this problem: https://bugzilla.redhat.com/show_bug.cgi?id=626606 This bugzilla is resolved in RHEL 6.1. 20. RHEL 6 SAN boot with default values will result in the Operating System installation LUN being managed by Device Mapper. Multi-Path (DMMP) failover. If a different failover is the preferred failover, steps must be taken during install to prevent DMMP from managing the LUN. The OS will be managed by DMMP by default. If the user installs the OS with default values then they will have this issue. The OS LUN could be managed by two different failovers and this could cause unforeseen errors and is not a tested/supported configuration. Workaround: To not allow the SAN boot LUN be managed by DMMP the user must configure the partition table manually during OS installation. 1. Select the "Review and modify partitioning layout" during installation 2. Delete all volumes in the LVM volume group 3. Delete the "physical volume (LVM)" from the hard drive group 4. Select "Free Space" from the hard drive group and select "Create" 5. Select "Standard Partition" 6. Add partition setup - mount point as / - file system type as ext4 - fill to maximum allowable size 7. Select "Next" and continue installation. 1.3 Enhancements version 09.03.0C05.0504 - New Linux MPP architecture allows for multi-threaded processing. - Provides a performance enhancement over the previous Linux MPP version. - Provide support for RedHat 6 and SLES 10 SP4 environment. 1.4 Recommendations and prerequisites for the update: SLES Linux on Power - package glibc-devel-64bit required for build 1.5 Dependencies: Refer to the www.ibm.com/systems/support/storage/config/ssic/index.jsp IBM System Storage™ Disk Storage Systems Interoperation Center web site for the latest update to the supported host adapters, device driver levels, and DS4000/DS5000 controller firmware versions. It is always recommended that the latest versions are used to receive up to date fixes and features. New linux kernel versions supported with this release: - SLES 10-SP3: 2.6.16.60-0.54.5 - SLES 10-SP4: 2.6.16.60-0.85.1 - SLES 11.1 : 2.6.32.12-0.7 - RHEL5-u5 : 2.6.18-194 - RHEL5-u6 : 2.6.18-238 - RHEL6 : 2.6.32-71 2.0 Installation and Setup Instructions ----------------------------------------- 2.1 Prerequisites 1. The Linux RDAC driver is released as a source-code package in the format of a gunzip compressed tar file. To unpack the driver source package, type the following command in the Linux console window: tar -zxvf rdac-LINUX-09.03.0C05.0504-source.tar.gz where xx.xx.xx.xx is the release version of the RDAC driver. The source files will uncompress to the "linuxrdac-09.03.0C05.0504" directory. 2. The Host server must have the non-failover Fibre Channel HBA device driver properly built and installed before the Linux RDAC driver installation. Refer to the FC HBA device driver README or the FC HBA User Guide for instructions on installing the non-failover version of the device driver. The driver source tree is included in the package if you need to build it from the source tree. 3. Although the Host server can have different FC HBAs from multiple vendors or different FC HBA models from the same vendors, only one model of the fibre channel HBAs can be connected to the IBM DS3000/DS4000/DS5000 storage subsystems in each host server. 4. Check for the following low level fibre channel host bus adapter (FC HBA) driver names entries in the appropriate file for a given Linux OS distribution. - For SuSE SLES Linux OS distribution: low level driver FC HBA names should be included in the INITRD_MODULES string in /etc/sysconfig/kernel file. Examples: - For the systems that have the IBM FC2-133 adapter or JS20 fibre channel expansion card, the INITRD_MODULES string should include a qla2xxx and qla2300 ( or qla2300). - For the systems that have IBM Emulex HBAs, the INITRD_MODULES string should include lpfcdd or lpfc. - For Redhat EL Linux OS distribution: There should be a scsi_hostadapter entry for the physical FC HBA driver(s) in the /etc/modules.conf file. Examples: - For the systems that have the IBM FC2-133 adapter or JS20 fibre channel expansion card, there is a scsi_hostadapter entry such as: alias scsi_hostadapterx qla2300 - For the systems that have IBM Emulex HBAs, there is a scsi_hostadapter entry such as: alias scsi_hostadapterx lpfc(or lpfcdd) 5. The kernel source tree for the Linux kernel version to be built against should already be installed before RDAC driver package installation. 2.2 Build and Installation 2.2.1. This step is for SUSE kernel distribution only. For Redhat kernel distribution, proceed to step 2.2.2. a. Install the kernel-source from the SUSE distribution. b. create a soft link to the kernel source. ln -sf /usr/src/ /usr/src/linux c. To ensure kernel version synchronization between the driver and running kernel, type the following in the Linux console window. Press Enter after each command. - cd /usr/src/linux - make mrproper (completely clean the kernel tree) - cp /boot/config-`uname -r` .config (copy the new config) - make oldconfig (update configuration using .config) - make dep (rebuild the dependencies) - make modules (build the modules-not require on newer kernel version) 2.2.2. RDAC Driver Building for SuSE and Redhat a. Change to the "linuxrdac" directory. b. To remove the old driver modules in that directory, type the following and press Enter: make clean c. To compile all driver modules and utilities in a multiple CPUs server (SMP kernel), type the following and press Enter: make 2.2.3. RDAC Driver Installation for SuSE and Redhat a. Type the following and press Enter: make install This command will do the following things: * copy the driver modules to the kernel module tree. * build the new RAMdisk image (mpp-`uname -r`.img) which includes the RDAC driver modules and all driver modules that are needed at boot. b. Follow the instructions displayed at the end of of the build process to add a new boot menu option which uses /boot/mpp-`uname -r`.img as the initial RAMdisk image. The following is an example for /boot/grub/menu.lst (Note that it may vary in accordance with different system setup configuration): title SUSE Linux with MPP support kernel (hd0,1)/boot/vmlinuz root=/dev/sda2 vga=0x317 selinux=0 splash=silent console=tty0 resume=/dev/sda3 elevator=cfq showopts initrd (hd0,1)/boot/mpp-2.6.5-7.139-smp.img The following is an example of /etc/yaboot.conf for IBM® eServer pSeries Architectures (ppc64) systems (Note entries may vary in accordance with your system setup configuration) image = /boot/vmlinux label = MPP root = /dev/sdb3 append = "selinux=0 elevator=cfq" initrd = /boot/mpp-2.6.5-7.139-pseries64.img 2.3 RDAC Driver Post-Installation a. Reboot the system using the new boot menu option. b. To verify that sd_mod, sg, mpp_Upper, qla2300, and mpp_Vhba were loaded after the reboot, type the following and press Enter: /sbin/lsmod On SuSE platforms, verify sg, mppUpper, mppVhba and low level HBA drivers (e.g., qla2xxx, qla2300) are loaded. c. To verify that the RDAC driver discovered the available physical Luns and created virtual Luns for them, type the following and press Enter: ls -lR /proc/mpp d. You can now issue I/Os to the Luns. e. If you make any changes to MPP configuration file (/etc/mpp.conf) or persistent binding file (/var/mpp/devicemapping), please run mppUpdate to re-build RAMdisk image to include the new file so that the new configuration file (or persistent binding file) can be used on the next system reboot. f. To dynamically re-load the driver stack (scsi_mod, sd_mod, sg, mpp_Upper, , mpp_Vhba) without rebooting the system, you need to first comment out all "scsi_hostadapter" entries in /etc/module.conf. Then, issue "modprobe -r mpp_Upper" to unload the driver stack and follow by "modprobe mpp_Upper" to re-load the driver stack. See Recommendation. Recommendation: Reboot the system whenever a need to unload the driver stack arises. The use of modprobe with the RDAC driver stack or the use of rmmod to remove all the drivers in the RDAC driver stack, in order, is not recommended/ supported. g. Installation of the Emulex Application Kit, such as lpfcapp-2.10g-1.ppc64.tgz, on SLES9 servers, automatically installs HBAnyware. HBAnyware interferes with the unloading the lpfcdfc and lpfc modules. The work around for this is to either stop HBAnyware (/usr/sbin/HBAnyware stop) or uninstall the Application kit. See the Emulex Application Kit Readme for further instructions. h. The file /etc/syslog.conf should have entries for kern.debug, kern.notice, kern.info and kern.warning in order to capture all debug and error messages for MPP driver. 2.4 Disable AVT Open the DS4000 Storage Manager Enterprise Management window, - Highlight the subsystem - Select Tools - Select Execute script. - A script editing window opens. In this window - Select File - Select Load Script - Give the full path name for the script file (eg . /usr/scripts/DisableAVT_Linux.scr) and click OK - Select Tools - Select Verify and Execute Please refer to the IBM System Storage DS4000 Storage Manager version 9 Installation and Support Guide for appropriate OS or the Enterprise Window Online help for more information on how to execute a script. 2.5 Persistent Naming Linux SCSI device names have the possibility of changing when the host system reboots. We recommend using a utility such as devlabel to create user-defined device names that will map devices based on a unique identifier, called a UUID. 2.6 RDAC driver unload The unload order for the MPP driver stack is as follows: a. rmmod mppVhba b. rmmod (low-level HBA driver, this can be qla2xxx followed by qla2300 or mptscsih followed by mptbase or lpfcdd or lpfc c. rmmod mppUpper 2.7 RDAC Driver Package Uninstall and Reinstall/Update To un-install the RDAC driver package, type the following and press Enter: make uninstall Note: To Reinstall/Update the RDAC driver package, you need to issue the command "make uninstall" followed by the "make install" command under the /linuxrdac directory in the new RDAC driver source code bundle. 2.8 Kernel Update After applying the kernel update and booting into the new kernel, please do the following things to build MPP Initrd image for the new kernel: a. Change directory to Linuxrdac source code directory. b. do "make uninstall". c. Follow driver building and installation procedure described in Section 2.2.2 and Section 2.2.3. 2.9 Initial Ramdisk image (initrd) Initial RamDisk Image (initrd image) is automatically created when the driver is installed using "make install". Boot loader Configuration file needs to have entry for this newly created image. The initrd image is located in boot partition with file name :'mpp-`uname -r`.img. { Example : mpp-2.4.21-9.ELsmp.img } For a Driver Update, If the system already has an old entry for MPP, the system administrator has to just modify the existing MPP entry accordingly in the boot loader configuration file. In most of the cases, no change is required if the kernel version is same. To create new 'initrd image' user has to run 'mppUpdate' which will overwrite old image file with the new image file. If 3rd party drivers needed to be added to initrd image, edit the /etc/sysconfig/kernel ( SUSE ) with the 3rd party driver entries. Run 'mppUpdate' to create new ramdisk image. 2.10 Adding and Deleting mapped logical drives (LUNs) Linux RDAC supports re-scanning to recognize a newly mapped LUN without rebooting the server. The utility program is packed with the Linux RDAC driver. It can be invoked by using either "hot_add" or "mppBusRescan" command. "hot_add" is a symbolic link to mppBusRescan. There are man pages for both commands. However, prior to Linux rdac version 09.03.0C05.0214, the Linux RDAC driver doesn't support LUN deletion. One has to reboot the server after deleting the mapped logical drives. With rdac version 09.03.0C05.0214 and later, one can run the command "hot_add -d" to delete the LUNs without rebooting the server. 2.11 Proc entries To verify that the RDAC driver discovered the available physical LUNs and created virtual LUNs for them, type the following and press Enter: ls -lR /proc/mpp The linux system provides a /proc file system that is a special memory-only, software-created filesystem used by the kernel to export driver information to the user space. The proc pseudo- filesystem is mounted to /proc. The linux SCSI subsystem's proc filesystem directory is /proc/scsi. By convention each HBA driver creates a sub directory under /proc/scsi. The linux RDAC driver's proc entries are: 1. /proc/mpp Entry for the RDAC driver itself 2. /proc/scsi/mpp/ The entry for the MPP virtual host adapter. This can differ from system to system and is assigned by the scsi middle-layer. 3. /proc/mpp/ There is an entry for each array that is visible to the host. 4. /proc/mpp//controllerA /proc/mpp//controllerB These are the entries for the 2 targets on each array seen by the host. Each storage array has 2 controllers. 5. /proc/mpp//controller/ Here the low level driver can be any one of qlogic or emulex HBAs. The HCT# is the Host#, channel#, Target#, where Host# is the Host No of the low-level driver and is assigned by the scsi mid-layer. Channel# depends on whether the HBA card is single or dual channeled. Target# is the Target # assigned to that controller on that array and is assigned by the Low-level HBA driver. 6. /proc/mpp//controller//LUN # Lastly are the Volumes or LUN # of Volumes as mapped on host partition on the storage array and seen through that path or Initiater(Host)-Target combination. 2.12 MAN pages There are 4 MAN pages that can provide more information man RDAC - This man page talks about the RDAC(MPP) driver and the RDAC(MPP) configurable parameters. man mppUtil - This man page tells you how to use the mppUtil - the utility for the RDAC(MPP) driver. man mppBusRescan - This man page describes the hot_add (a.k.a mppBusRescan) script that is available to add newly mapped luns and newly added storage arrays. man mppUpdate - This man page instructs you as to how you can use the persistant device mapping facility available with the RDAC driver and how to change the configuration values for the RDAC driver and build a new RDAC ramdisk. 2.13 Physical HBA driver update NOTE: This section applies to DS4000/5000 products. For DS3000 products, please refer to the DS3000 controller firmware readme. The following is the recommended procedure to update physical HBA driver : a. Get the latest hba drivers supported by MPP. b. install the drivers c. execute "mppUpdate" to create a new mpp initrd image. d. reboot the system to boot with the new initrd image. The following is the non-recommended procedure to update the physical HBA driver using modprobe : a. unload MPP Virtual HBA Driver. execute "modprobe -r mppVhba". b. Unload the physical HBA driver stack. example "modprobe -r lpfc". c. Unload MPP Upper Level Driver. execute "modprobe -r mppUpper". d. Get the latest hba drivers supported by MPP. e. compile / install the physical HBA driver. f. execute "mppUpdate" to create a new mpp initrd image. This step is important to pick up new physical HBA drivers in the mpp initrd image for the next boot. g. load MPP Upper Level Driver. execute "modprobe mppUpper". h. execute "cat /opt/mpp/modprobe.conf.mppappend" to see the mpp generated physical module parameters if any. For example: options lpfc lpfc_nodev_tmo=60 i. load the physical HBA driver stack using the module parameters. example "modprobe lpfc lpfc_nodev_tmo=60". j. load MPP Virtual HBA Driver. execute "modprobe mppVhba". NOTE : Please see mpp supported module options in section "Physical HBA Module options". NOTE : Qlogic hba driver has a module parameter "ql2xfailover" If the module parameter is not set to 0, qlogic driver acts as a failover driver itself and hides the paths. 2.14 Physical HBA Module options MPP Driver sets physical HBA moudle options during installation. Preset options can be found in /opt/mpp/modprobe.conf.mppappend after MPP driver installation. Following are the recommendations for various HBA drivers: 2.14.0 Qlogic HBA module options ( Module Name - qla2xxx ) ----------------------------------------------------------- ql2xfailover -> 0 Disable failover support in Qlogic driver since failover is handled by MPP driver. ql2xretrycount -> 3 Default retry count value is 20, we are setting it to a lower number and handle the IO return condition accordingly. This allows the IO to return to MPP driver from Qlogic driver before the MPP Driver's ControllerIOWaitTime ( Default 300 seconds ) is expired. ql2xprocessnotready -> 0 Disable the handling of target not ready feature in Qlogic driver, MPP will control this. ( SK/ASC/ASCQ 02/04/01 ) qlport_down_retry -> 35 The above value needs to be set for any storage array types which have FC frond-end and back-end. The HBA driver doubles the value (70). This is to handle the port down timeout. According to our testing setting this value to 35 will buy us approximately 70 seconds, after that qlogic driver returns DID_NO_CONNECT if the target is still gone. If the target comes back with in 70 seconds we do not need to perform a failover, else we will perform failover after qlogic responds with DID_NO_CONNECT after the 70 seconds. qlport_down_retry -> 72 The above value needs to be set for any storage array types which have FC frond-end and SAS back-end. The HBA driver doubles the value (144). 2.14.1 Emulex HBA module options ( Module Name - lpfc ) ----------------------------------------------------------- lpfc_nodev_tmo -> 60 The above value needs to be set for any storage array types which have FC frond-end and back-end. This enables the lpfc driver to hold IO waiting for device to come back. This handles the target loss scenario. If the target comes back with in 60 seconds we do not need to perform a failover, else we will perform failover after lpfc responds with DID_NO_CONNECT after the 60 seconds. lpfc_nodev_tmo -> 144 The above value needs to be set for any storage array types which have FC frond-end and SAS back-end. 2.14.2 Brocade HBA module options ( Module Name - bfa ) ----------------------------------------------------------- rport_del_timeout -> 60 The above value needs to be set for any storage array types which have FC frond-end and back-end. This enables the bfa driver to hold IO waiting for device to come back. This handles the target loss scenario. If the target comes back with in 60 seconds we do not need to perform a failover, else we will perform failover after bfa responds with DID_NO_CONNECT after the 60 seconds. rport_del_timeout -> 144 The above value needs to be set for any storage array types which have FC frond-end and SAS back-end. NOTE : These options are used in building the mpp initrd ramdisk. For example, if you have to override the changes then module options can be specified in /etc/modprobe.conf for Redhat AS 4, and /etc/modprobe.conf.local for SUSE SLES 9. 3.0 Configuration Information ------------------------------- 3.1 Configuration installation N/A 3.2 Configuration Settings N/A 3.3 Hardware Status and Information N/A 4.0 Unattended Mode ------------------------------- N/A 5.0 WEB Sites and Support Phone Number ---------------- 5.1 IBM System Storage™ Disk Storage Systems Technical Support web site: http://www.ibm.com/systems/storage/support 5.2 IBM System Storage™ Marketing Web Site: http://www.ibm.com/systems/storage/disk 5.3 IBM System Storage™ Interoperation Center (SSIC) web site: http://www.ibm.com/systems/support/storage/ssic/ 5.4 If you have any questions about this update, or problem applying the update go to the following HelpCenter World Telephone Numbers URL: http://www.ibm.com/planetwide IMPORTANT: It is recommended that you download the latest version of the DS4000 Storage Manager host software, the DS3000/DS4000/DS5000 storage subsystem controller firmware, the DS3000/DS4000/DS5000 drive expansion enclosure ESM firmware and the drive firmware at the time of the initial installation and when product updates become available. For more information about how to register for support notifications, see the following IBM Support Web page: www.ibm.com/systems/support/storage/subscribe/moreinfo.html You can also check the Stay Informed section of the IBM Disk Support Web site, at the following address: http://www.ibm.com/systems/storage/support 6.0 Trademarks and Notices ---------------------------- The following terms are trademarks of the IBM Corporation in the United States or other countries or both: IBM FAStT System Storage™ the e-business logo xSeries HelpCenter Linux is a registered trademark of Linus Torvalds. Microsoft, Windows, and Windows NT are trademarks of Microsoft Corporation in the United States, other countries, or both. Java and all Java-based trademarks and logos are trademarks or registered trademarks of Sun Microsystems, Inc. in the United states, other countries, or both. UNIX is a registered trademark of The Open Group in the United States and other countries. STEELEYE are TRADEMARKS and LIFEKEEPER is a registered trademark of STEELEYE TECHNOLOGY, INC. 7.0 Disclaimer -------------- 7.1 THIS DOCUMENT IS PROVIDED "AS IS" WITHOUT WARRANTY OF ANY KIND. IBM DISCLAIMS ALL WARRANTIES, WHETHER EXPRESS OR IMPLIED, INCLUDING WITHOUT LIMITATION, THE IMPLIED WARRANTIES OF FITNESS FOR A PARTICULAR PURPOSE AND MERCHANTABILITY WITH RESPECT TO THE INFORMATION IN THIS DOCUMENT. BY FURNISHING THIS DOCUMENT, IBM GRANTS NO LICENSES TO ANY PATENTS OR COPYRIGHTS. 7.2 Note to U.S. Government Users -- Documentation related to restricted rights -- Use, duplication or disclosure is subject to restrictions set forth in GSA ADP Schedule Contract with IBM Corporation.