IBM POWER9 Systems LC Server Firmware
Applies to: AC922 (8335-GTH) and AC922 (8335-GTX)
This document provides information about the installation of Licensed Machine or Licensed Internal Code, which is sometimes referred to generically as microcode or firmware.
This package provides firmware for the Power System AC922 (8335-GTH) and AC922 (8335-GTX) servers only.
The firmware level in this package is:
•OP920.02 / PNOR OP9_v2.0.8-2.2 and openBMC ibm-v2.1-438-r15
This section specifies the "Minimum ipmitool Code Level" required by the System Firmware for managing the system. OpenPOWER requires ipmitool level v1.8.15 or later to execute correctly on the OP910 and later firmware. It must be capable of establishing a IPMI v2 session with the ipmi support on the BMC.
Verify your ipmitool level on your Linux workstation using the following command:
bash-4.1$ ipmitool -V
ipmitool version 1.8.15
If you are need to update or add impitool to your Linux workstation , you can compile ipmitools (current level 1.8.15) for Linux as follows from Sourceforge:
1.1.1 Download impitool tar from http://sourceforge.net/projects/ipmitool/ to your linux system
1.1.2 Extract tarball on Linux system
1.1.3 cd to top-level directory
1.1.4 ./configure
1.1.5 make
1.1.6 ipmitool will be under src/ipmitool
You may also get the ipmitool package directly from your workstation Linux packages.
For specific fix level information on key components of IBM Power Systems LC and Linux operating systems, please refer to the documentation in the IBM Knowledge Center for the AC922 (8335-GTH) and AC922(8335-GTX):
https://www.ibm.com/support/knowledgecenter/en/POWER9/p9hdx/8335_gth_landing.htm
https://www.ibm.com/support/knowledgecenter/en/POWER9/p9hdx/8335_gtx_landing.htm
If using xCAT on the host OS to do firmware updates, the minimum xCAT level that should be used is 2.13.4 because it has stability improvements for the firmware update process. See the xCAT 2.13.4 release notes below for more information.
https://github.com/xcat2/xcat-core/wiki/XCAT_2.13.4_Release_Notes
The Linux OS has a NVIDIA CUDA driver that must be at recommended level 396.44 or later, or minimum level 396.26 to be compatible with OP920.00 and later levels. Without this driver, a GPU which has faulted and gone through a GPU reset can cause a Terminate Immediate (TI) for the system. The recommended level for the NVIDIA CUDA driver is level 396.44 to get ATS performance improvements.
The Power AC922 server delivers four Tesla V100 with NVLink GPUs supported in two processor sockets.
The Tesla CUDA driver can be obtained at the download NVIDIA link of “https://www.nvidia.com/content/DriverDownload-March2009/confirmation.php?url=/tesla/396.44/nvidia-driver-local-repo-rhel7-396.44-1.0-1.ppc64le.rpm&lang=us&type=Tesla”
The NVIDIA "http://www.nvidia.com/Download/index.aspx?lang=en-us" link using the following information can be used to do a manual search for the driver:
Manually find drivers for my NVIDIA products.
Product Type: Tesla
Product Series: V-Series
Product: Tesla V100
Operating System: Linux POWER LE RHEL 7
CUDA Toolkit: 9.2
Language: English(US)
Search results:
Version: 396.44
Release Date: 2018.8.6
Operating System: Linux POWER LE RHEL 7
CUDA Toolkit: 9.2
Language: English (US)
File Size: 47.28 MB
On Red Hat Enterprise Linux (RHEL) for PPC, RHEL-Alt 7.5, The Trusted Platform Module (TPM) device driver is not loaded automatically at boot time. Without this driver, the TPM device will not be accessible.
This affects any user-space application needing to access the TPM, as well as kernel security functions, such as the Integrity Measurement Architecture subsystem (IMA) in the Linux kernel. Without the TPM driver loaded, IMA will be unable to record trusted measurements to the TPM.
To load the driver manually, as root:
# modprobe tpm_i2c_nuvoton
To load the driver automatically at boot time:
# echo "tpm_i2c_nuvoton" > /etc/modules-load.d/tpm.conf"
The TPM device driver will be integrated as a built-in kernel module in a future release 7 of RHEL-Alt. Once this is done, it will be loaded automatically and this procedure will no longer be necessary.
Downgrading firmware from any given release level to an earlier release level is not recommended.
If you feel that it is necessary to downgrade the firmware on your system to an earlier release level, please contact your next level of support.
Concurrent Firmware Updates not available for LC servers.
Concurrent system firmware update is not supported on LC servers.
The 8335-GTG model is not supported for the OP920.xx release. However, the 8335-GTG may be upgraded to a 8335-GTH model by a SSR. These steps involve replacing the hardware processor features in the 8335-GTG and then updating to the alternative PNOR and BMC images, which can be found in Fix Central as part of the initial OP920.00 delivery. At the successful conclusion of the upgrade steps, the system model will be 8335-GTH with the OP920.00 release firmware. You may then update to this latest level of firmware.
The existing processors being replaced during a model or feature conversion become the property of IBM and must be returned. Feature conversions are always implemented on a "quantity of one for quantity of one" basis. Multiple existing features may not be converted to a single new feature. Single existing features may not be converted to multiple new features.
Feature conversions for 8335-GTG to 8335-GTH for processor features:
From FC | To FC | Return Parts? |
EP0K - 16-core 2.60 GHz (3.09 GHz Turbo) POWER9 Processor | EP0P - 16-core 2.7 GHz (3.3 GHz Turbo) POWER9 Processor | Yes |
EP0M - 20-core 2.0 GHz (2.87 GHz Turbo) POWER9 Processor | EP0R - 20-core 2.4 GHz (3.0 GHz Turbo) POWER9 Processor | Yes |
•witherspoon-IBM-OP9-v2.0-2.14-cfm_prod.pnor.squashfs.tar - alternative pnor image for use in the gtg to gth upgrade option.
•obmc-witherspoon-ibm-v2.1-cfm.ubi.mtd.tar - alternative bmc image for use in the GTG to GTH upgrade option.
(Note that these files exist only under the OP920.00 delivery in Fix Central)
Use the following examples as a reference to determine whether your installation will be concurrent or disruptive.
For the LC server systems, the installation of system firmware is always disruptive.
The BMC and PNOR image tar files are used to update the primary side of the PNOR and the primary side of the BMC only, leaving the golden sides unchanged.
Filename | Size | Checksum |
obmc-witherspoon-ibm-v2.1.ubi.mtd.tar | 22251520 | b934ea320980108f72753c3cb28a164d |
witherspoon-IBM-OP9-v2.0.8-2.2_prod.pnor.squashfs.tar | 23531520 | 029e4728975035b7bf439b3cdf122d5f |
|
|
|
Note: The Checksum can be found by running the Linux/Unix/AIX md5sum command against the Hardware Platform Management (hpm) file (all 32 characters of the checksum are listed), ie: md5sum <filename>
After a successful update to this firmware level, the PNOR components and BMC should be at the following levels.
To display the PNOR level, use the following BMC command: "cat /var/lib/phosphor-software-manager/pnor/ro/VERSION"
And the BMC command line command "cat" can be used to display the BMC level: "cat /etc/os-release".
Note: FRU information for the PNOR level does not show the updated levels via the fru command until the system has been booted once at the updated level.
PNOR firmware level: driver content
display pnor FW level using this cmd: "cat /var/lib/phosphor-software-manager/pnor/ro/VERSION"
IBM-witherspoon-ibm-OP9-v2.0.8-2.2-prod
op-build-v2.0.8-1-gc51594f
buildroot-2018.02.1-6-ga8d1126
skiboot-v6.0.7
hostboot-d033213-pf93db89
occ-084756c
linux-4.16.13-openpower1-p7c1bcc5
petitboot-v1.7.2-p221b844
machine-xml-7cd20a6
hostboot-binaries-hw080418a.op920
capp-ucode-p9-dd2-v4
sbe-55d6eb2
hcode-hw080418a.op920
openBMC level:
display BMC FW level via ssh session on the BMC , using this cmd root@witherspoon:~# cat /etc/os-release
ID: openbmc-phosphor
NAME: Phosphor OpenBMC (Phosphor OpenBMC Project Reference Distro)
VERSION: "ibm-v2.1"
VERSION_ID "ibm-v2.1-438-g0030304-r15-0-g19832d3"
PRETTY_NAME "Phosphor OpenBMC (Phosphor OpenBMC Project Reference Distro) ibm-v2.1"
BUILD_ID "ibm-v2.1-438-g0030304-r15"
OP920 | |
|
|
OP9_v2.0.8-2.2/ OP920.02
08/14/2018
| Impact: Availability Severity: SPE
New features and functions
Support was added for parity error checking of the GPU data on the NVLink Datalink Layer (NDL), providing earlier memory fault detection and recovery retries to eliminate transient faults.
System firmware changes that affect all systems
A problem was fixed for PCIe4 CX5 adapter performance with an increase of performance of 40% for DMA read requests. The adapter affected is the Mellanox CX5 PCIe4 100Gb IB CAPI with feature codes #EC62 with CCIN 2CF1 and #EC64 with CCIN 2CF2. Without the fix, each read request requires a retry to work.
A problem was fixed for user code running on a GPU that can perform invalid commands to the MMIO space and cause an HMI that brings down the system. With the fix, ill-formatted commands to the MMIO space from the GPU will not be processed as a fatal exception but responses will be set to 0xFFFFFFFF and the GPU will receive a normal response code. The user GPU application can look for the bad response and fail, but the system will continue running without taking an HMI, allowing all other workloads to continue normally.
A problem was fixed for a GPU NVLINK writing out of range to a MMIO section of memory with byte-enabled writes that caused a machine check. With the fix, the out of range write is handled (detected) to cause a process core dump, but leaves the system in a usable state.
A problem was fixed for GPU workloads using unified memory with address translation service (ATS) sometimes hanging after resetting the GPUs. The trigger for the failure was putting the NPU in the fenced state via the "NPU Fence State" register with SCOM address 0x5011696. With the fix, the GPU fencing is handled using the NTL (NVLink Transaction Layer) reset register bits instead.
A problem was fixed for user applications timing out on the GPU operation for accessing the address translation service (ATS) unified memory, causing an HMI and system termination. With the fix, the ATSD timeout has been disabled, so the user applications can wait for GPU read or write operations to be completed without regard for the time needed for the operation.
A problem was fixed for the On-Chip Controller inadvertently disabling the MMIO ATSD flush bits, thereby potentially reducing the performance of the address translation service (ATS) unified memory for the GPU.
A problem was fixed for the SBE timer being stuck and unavailable to the host applications. This forces OPAL to use legacy timer loops for timers at the cost of additional processor bandwidth. Here are the messages that are logged for the problem that occurs on every boot: [ 194.494559313,3] SBE: Timer stuck, falling back to OPAL pollers. [ 194.494624185,3] SBE: You will likely have slower I2C and may have experienced increased jitter.
A problem was fixed for NPU log messages that were missing the CPU chip identifiers. With the fix, CPU taking the HMI (Host Maintenance Interrupt) is listed along with the NPU FIR register values.
A problem was fixed for various I2C devices such as DIMMs, SEEPROMs, and TPMs failing during the IPL with an I2C arbitration loss condition. This results in unusable hardware or IPL failures. Frequency of this intermittent error varies with the number of I2C devices in the system. A re-IPL of the system can recover from the problem.
|
OP9_v2.0.3-2.17 / OP920.01
06/22/2018 | Impact: Data Severity: HIPER
New features and functions
Support for the Simple Network Management Protocol (SNMP) was added to the BMC. SNMP is an Internet standard protocol for collecting and organizing information about managed devices on IP networks and for modifying that information to change device behavior.
Support for network configuration was added to the BMC web GUI.
System firmware changes that affect all systems
HIPER/Pervasive: A firmware change was made to address a rare case where a memory correctable error on POWER9 servers may result in an undetected corruption of data.
A problem has been fixed for a PCIe adaper running in CAPP mode having a missing MMIO Base Address Register (BAR) entry that causes a failure of the adapter and a fence off of two of the four ports of the adapter.
A problem has been fixed for a slow start up of a process that can occur when the system had been previously in an idle state.
A problem was fixed for a failure in DDR4 RCD (Register Clock Driver) memory initialization that causes half of the DIMM memory to be unusable after an IPL. This is an intermittent problem where the memory can sometimes be recovered by doing another IPL. The error is not a hardware problem with the DIMM but it is an error in the initialization sequence needed get the DIMM ready for normal operations.
SW429936(HOSTBOOT): A problem has been fixed for systems unexpectedly running with all processors at lower frequencies than would be expected for Workload Optimized Frequency (WOF) ultra-turbo mode.
SW429049(HCODE): A problem was fixed for a processor core that cannot be awakened or a timeout in the On-Chip Controller (OCC) when switching Workload Optimized Frequency (WOF) modes from disabled to enabled. These errors can cause a reduction in performance by running with fewer cores or by running at the safe mode frequencies
|
OP9_v2.0_2.14 / OP920.00
05/25/2018 | Impact: New Severity: New
New features and functions for MTM 8335-GTH and 8335-GTX:
GA Level
|
OS levels supported by the LC 8335 servers:
- Minimum level is Red Hat Enterprise Linux 7.5 for IBM Power LE (POWER9), also known as RHEL 7.5-ALT LE, with third Z-stream or later (https://access.redhat.com/errata/RHBA-2018:2467 has the needed kernel "kernel-alt-4.14.0-49.10.1.el7a.src.rpm" )
- NVIDIA Telsa CUDA recommended driver level 396.44 or later, or minimum driver level 396.26 from the CUDA 9.2 toolkit
Additional OS level supported by the AC922(8335-GTH) server:
- Ubuntu 18.04, or later (with no GPU option)
IBM Power LC 8335 servers supports Linux which provides a UNIX like implementation across many computer architectures. Linux supports almost all of the Power System I/O and the configurator verifies support on order. For more information about the software that is available on IBM Power Systems, see the Linux on IBM Power Systems website:
http://www.ibm.com/systems/power/software/linux/index.html
The Linux operating system is an open source, cross-platform OS. It is supported on every Power Systems server IBM sells. Linux on Power Systems is the only Linux infrastructure that offers both scale-out and scale-up choices.
A supported version of Linux on the Power LC 8335 is Red Hat Enterprise Linux 7.5 for IBM Power LE (POWER9) (RHEL 7.5-ALT LE).
For additional questions about the availability of this release and supported Power servers, consult the Red Hat Hardware Catalog at
https://access.redhat.com/products/red-hat-enterprise-linux/#addl-arch.
For the AC922 (8335-GTH) that is configured without GPUs, there is the option of using Linux Ubuntu 18.04 or later as the OS.
For more information about Linux on Power, see the Linux on Power developer center at https://developer.ibm.com/linuxonpower/
For information about the features and external devices that are supported by Linux, see this website:
http://www.ibm.com/systems/power/software/linux/index.html
Use one of the following commands at the Linux command prompt to determine the current Linux level:
•cat /proc/version
•uname -a
The output string from the command will provide the Linux version level.
The opal-prd package on the Linux system collects the OPAL Processor Recovery Diagnostics messages to log file /var/log/syslog. It is recommended that this package be installed if it is not already present as it will help with maintaining the system processors by alerting the users to processor maintenance when needed.
On Red Hat Linux, perform command "rpm -qa | grep -i opal-prd ". The command output indicates the package is installed on your system if the rpm for opal-prd is found and displayed. This package provides a daemon to load and run the OpenPOWER firmware's Processor Recovery Diagnostics binary. This is responsible for run-time maintenance of Power hardware. If the package is not installed on your system, the following command can be run on Red Hat to install it:
sudo yum update opal-prd
To display the PNOR level, use the following BMC command: "cat /var/lib/phosphor-software-manager/pnor/ro/VERSION"
And the BMC command line command "cat" can be used to display the BMC level: "cat /etc/os-release".
Note: the "cat" commands are run after ssh to the BMC as root and the default password is 0penBmc (where 0 is the zero character).
Follow the instructions on Fix Central. You must read and agree to the license agreement to obtain the firmware packages.
The updating and upgrading of system firmware depends on several factors, such as the current firmware that is installed, and what operating systems is running on the system.
These scenarios and the associated installation instructions are comprehensively outlined in the firmware section of Fix Central, found at the following website:
http://www.ibm.com/support/fixcentral/
Any hardware failures should be resolved before proceeding with the firmware updates to help insure the system will not be running degraded after the updates.
The process of updating firmware on the OpenBMC managed servers is documented below.
The sequence of events that must happen is the following:
•Power off the Host
•Update and Activate BMC
•Update and Activate PNOR
•Reboot the BMC (applies new BMC image)
•Power on the Host (applies new PNOR image)
The OpenBMC firmware updates (BMC and PNOR) for the LC 8335 servers can be managed via the command line with the openbmctool.
The openbmctool is obtained using the IBM Support Portal.
1.Go to the IBM Support Portal.
2.In the search field, enter your machine type and model. Then click the correct product support entry for your system.
3.From the Downloads list, click the openbmctool for your machine type and model.
4.Follow the instructions to install and run the openbmctool. You will need to provide the file locations of the BMC firmware image tar and PNOR firmware image tar that must be downloaded from Fix Central for the update level needed.
Information on the openbmctool and the firmware update process can be found in the IBM Knowledge Center:
https://www.ibm.com/support/knowledgecenter/POWER9/p9ei8/p9ei8_update_firmware_openbmctool.htm .
The service processor, or baseboard management controller (BMC), provides a hypervisor and operating system-independent layer that uses the robust error detection and self-healing functions that are built into the POWER processor and memory buffer modules. OpenPOWER application layer (OPAL) is the system firmware in the stack of POWER processor-based Linux-only servers.
The service processor, or baseboard management controller (BMC), is the primary control for autonomous sensor monitoring and event logging features on the LC server.
The BMC supports the Intelligent Platform Management Interface (IPMI) for system monitoring and management. The BMC monitors the operation of the firmware during the boot process and also monitors the OPAL hypervisor for termination.
The OpenPOWER Abstraction Layer (OPAL) provides hardware abstraction and run time services to the running host Operating System.
For the 8335 servers, only the OPAL bare-metal installs can be used.
Find out more about OPAL skiboot here:
https://github.com/open-power/skiboot
The Intelligent Platform Management Interface (IPMI) is an open standard for monitoring, logging, recovery, inventory, and control of hardware that is implemented independent of the main CPU, BIOS, and OS. The LC 8335 servers provide one 10M/100M baseT IPMI port.
The ipmitool is a utility for managing and configuring devices that support IPMI. It provides a simple command-line interface to the service processor. You can install the ipmitool from the Linux distribution packages in your workstation, sourceforge.net, or another server (preferably on the same network as the installed server).
For installing ipmitool from sourceforge, please see section 1.1 "Minimum ipmitool Code Level".
For more information about ipmitool, there are several good references for ipmitool commands:
The man page
The built-in command line help provides a list of IPMItool commands:
# ipmitool help
You can also get help for many specific IPMItool commands by adding the word help after the command:
# ipmitool channel help
For a list of common ipmitool commands and help on each, you may use the following link:
www.ibm.com/support/knowledgecenter/linuxonibm/liabp/liabpcommonipmi.htm
To connect to your host system with IPMI, you need to know the IP address of the server and have
a valid password. To power on the server with the ipmitool, follow these steps:
1. Open a terminal program.
2. Power on your server with the ipmitool:
ipmitool -I lanplus -H bmc_ip_address -P ipmi_password power on
3. Activate your IPMI console:
ipmitool -I lanplus -H bmc_ip_address -P ipmi_password sol activate
Petitboot is a kexec based bootloader used by IBM POWER9 systems for doing the bare-metal installs on the 8335 servers.
After the POWER9 system powers on, the petitboot bootloader scans local boot devices and network interfaces to find boot options that are available to the system. Petitboot returns a list of boot options that are available to the system. If you are using a static IP or if you did not provide boot arguments in your network boot server, you must provide the details to petitboot. You can configure petitboot to find your boot with the following instructions:
https://www.ibm.com/support/knowledgecenter/linuxonibm/liabp/liabppetitbootadvanced.htm
You can edit petitboot configuration options, change the amount of time before Petitboot automatically boots, etc. with these instructions:
https://www.ibm.com/support/knowledgecenter/linuxonibm/liabp/liabppetitbootconfig.htm
After you select to boot the ISO media for the Linux distribution of your choice, the installer wizard for that Linux distribution walks you through the steps to set up disk options, your root password, time zones, and so on.
You can read more about the petitboot bootloader program here:
https://www.kernel.org/pub/linux/kernel/people/geoff/petitboot/petitboot.html
This guide helps you install Linux on Power Systems server.
Overview
Use the information found in http://www.ibm.com/support/knowledgecenter/linuxonibm/liabw/liabwkickoff.htm to install Linux on a non-virtualized (bare metal) IBM Power LC server.
Date | Description |
08/14/2018 | Added OP920.02 |
06/22/2018 | Added OP920.01 |
06/01/2018 | OP920.00 delivery - Added files to support an upgrade from 8335-GTG to 8335-GTH |
05/25/2018 | New for AC922 LC servers for the OP920.00 release |