From FC

To FC

Return Parts?

EP0K - 16-core 2.60 GHz

(3.09 GHz Turbo) POWER9 Processor

EP0P - 16-core 2.7 GHz

(3.3 GHz Turbo) POWER9 Processor

Yes

EP0M - 20-core 2.0 GHz

(2.87 GHz Turbo) POWER9 Processor

EP0R - 20-core 2.4 GHz

(3.0  GHz Turbo) POWER9 Processor

Yes

Filename

Size

Checksum

obmc-witherspoon-ibm-v2.3.ubi.mtd.tar

24023040

 5ff23b60c6cad376ce823b0237bfed11

witherspoon-IBM-OP9-v2.0.15-2.9_prod.pnor.squashfs.tar

 23777280

 ab77d13dde644ff87e369caffcd2cf6e

OP920
For Impact, Severity and other Firmware definitions, Please refer to the below 'Glossary of firmware terms' url: 
http://www14.software.ibm.com/webapp/set2/sas/f/power5cm/home.html#termdefs

OP9_v2.0.15-2.9/ OP920.21

09/12/2019

Impact: Data    Severity: HIPER

New features and functions

A change was made to enable OS software checkstops by default.  This prevents hangs of multiple hours in failed reboots if the CPUs become stuck at the start of a kdump.

System firmware changes that affect all systems

HIPER/Pervasive:   A change was made to fix an intermittent processor anomaly that may result in issues such as operating system or hypervisor termination, application segmentation fault, hang, or undetected data corruption.  The only issues observed to date have been operating system or hypervisor terminations.

A problem was fixed for failed FRUs associated with checkstops in hostboot not being guarded.  This is an intermittent timing problem related to error log entries that have been created but not written or flushed yet at the time of the guard processing.

A problem was fixed for an IPL failure caused by an IPMI timeout.  This is a rare problem and the reIPL after the failure recovers from the problem.  The eSEL for the IPMI timeout may be ignored.

A problem was fixed for an OS boot that fails because the BMC is going through a reboot itself.  The OS boot can fail when it needs to use BMC services when accessing the flash memory.  This can happen if the BMC is not ready to receive commands.  With the fix, the boot waits for the BMC to become ready instead of failing immediately on the errant flash access.

A problem was fixed for a fast reboot of the OS failing if VFs (Virtual Funcitions) were enabled and disabled prior to the reboot.

A problem was fixed for bad flashes caused by data size of memory to flash not being block aligned.  This error can intermittently cause partial data to be written to the flash.  

A problem was fixed for a hang in the OS reboot caused by a TOD failure.

OP9_v2.0.14-2.6/ OP920.20

04/04/2019

Impact: Data   Severity:  HIPER

New features and functions

Support was added for alternate fan settings to increase cooling for air-cooled system configurations that have optical cables.  In cases of lower processor and GPU usage but high I/O activity over the optical cables, the system may have an over-temperature warning near the I/O adapter because the fans are running too slowly with the default setting.  The alternate "custom" setting for the higher fan speeds can be enabled using the following openbmctool commands at version 1.13 and later (this version supports reading the current/supported thermal modes and changing the current mode):

Get list of available thermal control zones

- openbmctool.py -H [BMC] -U root -P 0penBmc thermal zones

Get the current and supported thermal modes

- openbmctool.py -H [BMC] -U root -P 0penBmc thermal modes get -z 0

Set the thermal mode to "Custom"

- openbmctool.py -H [BMC] -U root -P 0penBmc thermal modes set -z 0 -m Custom

Set the thermal mode to "Default"

- openbmctool.py -H [BMC] -U root -P 0penBmc thermal modes set -z 0 -m Default

System firmware changes that affect all systems

HIPER/Pervasive:  A problem was fixed where, under certain conditions, a Power Management Reset (PM Reset) event may result in undetected data corruption.  PM Resets occur under various scenarios such as power management mode changes between Dynamic Performance and Maximum Performance, power management controller recovery procedures, or system boot.

A problem was fixed for intermittent user and user level privilege errors for the OS ipmipower command.

The following error message is issued:  " privilege level cannot be obtained for this user".

A problem was fixed for a re-IPL failure of OPAL with BB821410 logged.  This is an intermittent and infrequent error that can occur if Skiboot fails to get notified of the BMC mailbox shutdown prior to the re-IPL attempt.  The problem can be circumvented by doing another IPL.

A problem was fixed for IPMI power down and power on raw commands failing when issued in IPMI Restriction Mode.  For this error, the host goes unresponsive with the following SEL list and message logged to the BMC gui after the raw commands are issued from ipmitool:

> ipmitool -I lanplus -H 9.40.192.54 -P 0penBmc sel list

 195 | 02/05/2019 | 09:48:19 | System Event #0x01 | Undetermined system hardware failure | Asserted

 196 | 02/05/2019 | 10:36:54 | System Event #0x01 | Undetermined system hardware failure | Asserted

 197 | 02/05/2019 | 10:40:53 | System Event #0x01 | Undetermined system hardware failure | Asserted

And the following error logged in the BMC GUI:

FQPSPCR0023M: Hostboot has become unresponsive

 _PID=2868 MESSAGE=org.open_power.Host.Boot.Error.WatchdogTimedOut

A problem was fixed for not being able to change the "IPMI admin" password away from the BMC default.  With the fix, the "IPMI admin" password can be changed using the ipmitool command.  Note: The "IPMI admin" password is independent of the "normal admin" password on the BMC, such as that used by the REST APIs.  When REST is used to change the admin password, it is changing the "normal admin" user ID password, not the "IPMI admin" user ID password.  Consider changing both of the "admin" user passwords to provide better security.  The following is an example of using the ipmitool command to change the "IPMI admin" user ID password (The "1" represents userid 1, which corresponds to the " IPMI admin" user):

ipmitool user set password 1 your-new-IPMI-admin-password

Support was added to recognize a port parameter in the URL path for the Preboot eXecution Environment (PXE) in the ethernet adapters.  Without the fix, there could be PXE discovery failures if a port was specified in the URL for the PXE.  

A problem was fixed for a skiboot hang that could occur rarely for a i2C request if the i2c  bus is in error or locked by the On-Chip Controller (OCC).

A problem was fixed for "Unexpected TCE size" error messages when Linux tried the default P9 PHB4 pages size and used the unsupported  2M and 1G page sizes.  The TCE page size property is now set correctly with 4K/64K/16M and 256M supported.

A problem was fixed for PCIe ECC protection in the response data path for Power 9 processor parts.  With the fix,  PCIe ECC errors detected from the adjacent AIB (Adapter Interface Board) receive data path escalate to a checkstop so that the defective parts can be replaced.

A problem was fixed for an intermittent rare processor core lock failure that is not a real hardware problem.  The erroneous failure looks like this in the logs:

LOCK ERROR: Releasing lock we don't hold depth @0x30493d20 (state: 0x0000000000000001)

      [13836.000173140,0] Aborting!

      CPU 0000 Backtrace:

       S: 0000000031c03930 R: 000000003001d840   ._abort+0x60

       S: 0000000031c039c0 R: 000000003001a0c4   .lock_error+0x64

       S: 0000000031c03a50 R: 0000000030019c70   .unlock+0x54

       S: 0000000031c03af0 R: 000000003001a040   .drop_my_locks+0xf4

A problem was fixed for the power-capping range allowed for the user.  Changes were made to allow the user to access the entire powercap range, with two minimums exported into the OS:  soft power cap minimum  "powercap-min" and the hard power cap minimum limit "powercap-hard-min".

A problem was fixed for an OS reboot after a shutdown that intermittently fails after the shutdown.  This can happen if the BMC is not ready to receive commands.  With the fix, the messages to the BMC are validated and retried as needed.  To recover from this error, the system can be rebooted from the BMC interface.

A problem was fixed for a kernel hard lock up that could occur if IPMI synchronous messages were sent from the OS to BMC while the BMC was rebooting.  For these type of messages, a processor thread remains waiting in OPAL until a response is returned from the BMC.

A problem was fixed for a rare Nest Memory Management Unit (NMMU) hang calling out processor hardware incorrectly, masking the real cause of the problem which was an NPU failure.   The incorrect error messages take this form on the system:

3      | FQPSPPU0093G  | 2018-10-01 01:25:40  | Yes          | Warning   | CPU 1 has exceeded a correctable error threshold

4      | FQPSPPU0093G  | 2018-10-01 03:20:55  | Yes          | Warning   | CPU 0 has exceeded a correctable error threshold

5      | FQPSPAA0008M  | 2018-10-01 04:35:40  | Yes          | Critical  | Hostboot procedure callout

A problem was fixed for certain errors from OCC and Power Management not having a SEL log created when it should have for failures such as B1xx2AD3.

A problem was fixed for DDR4 2933 MHZ and 3200 MHZ DIMMs not defaulting to the 2666 MHZ speed on a new DIMM plug, thus preventing the system  from IPLing.

Another fix was delivered for an intermittent IPL failure with BC131705 and  BC8A1703 logged with a processor core called out.  This is a rare error and does not have a real hardware fault, so the processor core can be unguarded and used again on the next IPL.  This fix adds additional synchronization to the core threads over what was delivered in OP920.10.

A problem was fixed for memory bandwidth degradation that can occur if memory DIMMs have failed locations noted with symbol marks.  Without the fix, the marked failed memory locations were wrongly subject to ECC corrector retries by the memory controller.

A problem was fixed for call home error logs from OCC having incorrect data regarding the memory bandwidth averages.

OP9_v2.0.10-2.2/ OP920.10

12/14/2018

Impact: Data    Severity:  HIPER

New features and functions

The POWER9 default for the Spectre mitigation protection was changed to Kernel protection instead of Kernel + User protection.  This was done to improve performance for user workloads such as Python scripts.  The security vulnerabilities, CVE-2017-5753 and CVE-2017- 5715 (collectively known as Spectre) allow user-level code to infer data from unauthorized memory by using speculative execution to perform side-channel information disclosure attacks.  If the default Spectre protection level had been changed on the system previously, this modified level of Spectre protection will persist across the firmware update.  

The following are the steps that can be used to override the default Spectre protection to provide for more security by fully engaging the Spectre protection or provide for more performance by fully disengaging the Spectre protection.  Note that disengaging the protections will leave the system vulnerable to attack via Spectre variant 2, and could result in data leakage and/or system compromise.  The override is controlled by the BMC and requires a reboot of the POWER9 to take effect.

To override the protection level:

   1) Create/edit the "/var/lib/obmc/cfam_overrides" on BMC

   2)Add the following contents:

# Control speculative execution mode

0 0x283a 0x00000000  # bits 28:31 are used for init level – in this case 0 for Kernel + User

0 0x283F 0x20000000  # Indicate override register is valid

   3) Re-IPL to apply changes.

Key:

    init level 0 == Kernel + User protection (safest, old default)

    init level 1 == Kernel protection only (safest, new default)

    init level 2 == No protection

Support was added for increasing the number of BMC error logs from 100 to 200 and changing the error log to roll over old entries when full instead of stopping the logging of errors.  Without this feature, the error log would get full at 100 entries and error logging would be stopped until some of the error logs were purged to make room for new entries.

System firmware changes that affect all systems

HIPER/Non-pervasive: Fixes included to address potential scenarios that could result in undetected data corruption.

A problem was fixed for an intermittent opal-prd crash that can happen on the host OS.  This is the fault signature:  " opal-prd[2864]: unhandled signal 11 at 0000000000029320 nip 00000 00102012830 lr 0000000102016890 code 1"

A problem was fixed for a PCI Host Bridge (PHB) configuration write error that caused the incorrect PCIe device to be frozen.  The fault will be attributed to the last device to have a memory-mapped I/O operation (MMIO).  With this fix,  the freeze action for PHB configuration write errors is disabled in order to not impact functional hardware.

A problem was fixed for diagnostic code trying to read sensor values for PCI Host Bridge (PHB) entries that are unused, which causes debug output to have incorrect values for the unused entries.  With the fix, only the used entries are processed by the diagnostic code.

A problem was fixed for a IPL loop/hang with a fatal MCE exception log caused by a probe of a failed PCI Host Bridge (PHB) that had been guarded.   This is an infrequent error because it requires a PHB to have previously failed. The exception log has the following format:

Fatal MCE at 000000003006ecd4   .probe_phb4+0x570

 CFAR : 00000000300b98a0

 <snip>

Aborting!

CPU 0018 Backtrace:

S: 0000000031cc37e0 R: 000000003001a51c   ._abort+0x4c

S: 0000000031cc3860 R: 0000000030028170   .exception_entry+0x180

S: 0000000031cc3a40 R: 0000000000001f10 *

S: 0000000031cc3c20 R: 000000003006ecb0   .probe_phb4+0x54c

S: 0000000031cc3e30 R: 0000000030014ca4   .main_cpu_entry+0x5b0

S: 0000000031cc3f00 R: 0000000030002700   boot_entry+0x1b8

A problem was fixed for an intermittent error message when activating firmware during a firmware update.  This extraneous error message occurred with moderate frequency.  This is internal server 500 error message returned on the REST enumerate request.  The error message can be ignored because there is not a problem with the firmware activate.

A problem was fixed for an On-Chip Controller (OCC) read failure with ERRNO=19 during a power off of the system.  This intermittent problem is an extraneous errror log and can be ignored because the power off is successful.

A problem was fixed for an intermittent power on failure with message "Error in mapper call to get service name".  To recover from this problem. power cycle the BMC and try the boot again.  

A problem was fixed for not being able to set the Power Supply Redundancy by using a REST API command.  Without the fix, this was a read-only attribute.

A problem was fixed for memory Over-Temperature (OT) throttling not occurring when a DIMM reaches the throttle temperature.  Although the frequency to the memory DIMMs is not reduced, the fan speeds do increase to provide more cooling for the DIMMs.

A problem was fixed for error logs occurring on the IPL following a DIMM error recovery.  These logs, related to failed memory scrubbing, have the following "Signature Description":  "mba(n0p15c1) () ERROR: command complete analysis failed".  These error logs do not indicate a hardware problem and may be ignored.

A problem was fixed for system termination for a re-IPL with power on.  The system can be recovered by powering off and then IPLing.  This problem occurs infrequently and can be avoided by powering off the system between IPLs.

A problem was fixed for certain system boot failures not propagating to the BMC before the boot firmware shuts down.  Some details of the error log may still appear in the console output trace, but the details will not be available with the BMC queries.  This problem is timing dependent and intermittently possible depending on the timing of the shutdown path.  However, immediate shutdowns exacerbate the problem and increase the chance it can occur.

OP9_v2.0.8-2.7/ OP920.03

09/25/2018

System firmware changes that affect all systems

A problem was fixed for a MSI-X checkstop in CAPI mode.  This occurred intermittently when a DMA from the CAPI device targeted an address lower than 4GB and was confused for a 32-bit MSI operation.  This is now avoided by disabling the 32-bit MSI when in CAPI mode.

A performance problem was fixed for certain cases of DMA operations from the GPU to an untranslated virtual memory location.  With the fix, as much as a 10X performance improvement can occur for this type of DMA from the GPU.

OP9_v2.0.8-2.2/ OP920.02

08/14/2018

Impact: Availability       Severity:  SPE

New features and functions

Support was added for parity error checking of the GPU data on the NVLink Datalink Layer (NDL), providing earlier memory fault detection and recovery retries to eliminate transient faults.  

System firmware changes that affect all systems

A problem was fixed for PCIe4 CX5 adapter performance with an increase of performance of 40% for DMA read requests.  The adapter affected is the Mellanox CX5 PCIe4 100Gb IB CAPI with feature codes  #EC62 with CCIN 2CF1 and  #EC64 with CCIN 2CF2.  Without the fix, each read request requires a retry to work.

A problem was fixed for user code running on a GPU that can perform invalid commands to the MMIO space and cause an HMI that brings down the system.  With the fix, ill-formatted commands to the MMIO space from the GPU will not be processed as a fatal exception but responses will be set to 0xFFFFFFFF and the GPU will receive a normal response code.  The user  GPU application can look for the bad response and fail, but the system will continue running without taking an HMI, allowing all other workloads to continue normally.

A problem was fixed for a GPU NVLINK writing out of range to a MMIO section of memory with byte-enabled writes that caused a machine check.  With the fix, the out of range write is handled (detected) to cause a process core dump, but leaves the system in a usable state.

A problem was fixed for GPU workloads using unified memory with address translation service (ATS) sometimes hanging after resetting the GPUs.  The trigger for the failure was putting the NPU in the fenced state via the "NPU Fence State" register with SCOM address 0x5011696.  With the fix, the GPU fencing is handled using the NTL (NVLink Transaction Layer) reset register bits instead.

A problem was fixed for user applications timing out on the GPU operation for accessing the address translation service (ATS) unified memory, causing an HMI and system termination.  With the fix, the ATSD timeout has been disabled, so the user applications can wait for GPU read or write operations to be completed without regard for the time needed for the operation.

A problem was fixed for the On-Chip Controller inadvertently disabling the MMIO ATSD flush bits, thereby potentially reducing the performance of the address translation service (ATS) unified memory for the GPU.

A problem was fixed for the SBE timer being stuck and unavailable to the host applications.  This forces OPAL to use legacy timer loops for timers at the cost of additional processor bandwidth.  Here are the messages that are logged for the problem that occurs on every boot:

[  194.494559313,3] SBE: Timer stuck, falling back to OPAL pollers.

[  194.494624185,3] SBE: You will likely have slower I2C and may have experienced increased jitter.

A problem was fixed for NPU log messages that were missing the CPU chip identifiers.  With the fix, CPU taking the HMI (Host Maintenance Interrupt) is listed along with the NPU FIR register values.

A problem was fixed for various I2C devices such as DIMMs, SEEPROMs, and TPMs failing during the IPL with an I2C arbitration loss condition.  This results in unusable hardware or IPL failures.  Frequency of this intermittent error varies with the number of I2C devices in the system.   A re-IPL of the system can recover from the problem.

OP9_v2.0.3-2.17 / OP920.01

06/22/2018

Impact:  Data    Severity:  HIPER

New features and functions

Support for the Simple Network Management Protocol (SNMP) was added to the BMC.  SNMP is an Internet standard protocol for collecting and organizing information about managed devices on IP networks and for modifying that information to change device behavior.

Support for network configuration was added to the BMC web GUI.

System firmware changes that affect all systems

HIPER/Pervasive:   A firmware change was made to address a rare case where a memory correctable error on POWER9 servers may result in an undetected corruption of data.

A problem has been fixed for a PCIe adaper running in CAPP mode having a missing MMIO Base Address Register (BAR) entry that causes a failure of the adapter and  a fence off of two of the four ports of the adapter.

A problem has been fixed for a slow start up of a process that can occur when the system had been previously in an idle state.

A problem was fixed for a failure in DDR4 RCD (Register Clock Driver) memory initialization that causes half of the DIMM memory to be unusable after an IPL.  This is an intermittent problem where the memory can sometimes be recovered by doing another IPL.  The error is not a hardware problem with the DIMM but it is an error in the initialization sequence needed get the DIMM ready for normal operations.

SW429936(HOSTBOOT):  A problem has been fixed for systems unexpectedly running with all processors at lower frequencies than would be expected for Workload Optimized Frequency (WOF)  ultra-turbo mode.

SW429049(HCODE):   A problem was fixed for a processor core that cannot be awakened or a timeout in the On-Chip Controller (OCC) when switching Workload Optimized Frequency (WOF) modes from disabled to enabled.  These errors can cause a reduction in performance by running with fewer cores or by running at the safe mode frequencies

OP9_v2.0_2.14 / OP920.00

05/25/2018

Impact:  New    Severity:  New

New features and functions for MTM  8335-GTH and 8335-GTX:

GA Level

Date

Description

09/11/19

08/02/2019

04/04/2019

Added OP920.20

12/14/2018

10/02/2018

09/25/2018

08/14/2018

06/22/2018

06/01/2018

05/25/2018