Power9 System Firmware

Applies to:   9040-MR9

This document provides information about the installation of Licensed Machine or Licensed Internal Code, which is sometimes referred to generically as microcode or firmware.


Contents


1.0 Systems Affected

This package provides firmware for Power Systems E950 (9040-MR9) servers only.

The firmware level in this package is:

1.1 Minimum HMC Code Level

This section is intended to describe the "Minimum HMC Code Level" required by the System Firmware to complete the firmware installation process. When installing the System Firmware, the HMC level must be equal to or higher than the "Minimum HMC Code Level" before starting the system firmware update.  If the HMC managing the server targeted for the System Firmware update is running a code level lower than the "Minimum HMC Code Level" the firmware update will not proceed.

The Minimum HMC Code levels for this firmware for HMC x86,  ppc64 or ppc64le are listed below.

x86 -  This term is used to reference the legacy HMC that runs on x86/Intel/AMD hardware for the Virtual HMC that can run on the Intel hypervisors (KVM, XEN, VMWare ESXi).
ppc64 or ppc64le - describes the Linux code that is compiled to run on Power-based servers or LPARS (Logical Partitions)
The Minimum HMC level supports the following HMC models:
x86 - KVM, XEN, VMWare ESXi (6.0/6.5)
ppc64le - 7063-CR1,vHMC on PowerVM (POWER8 and POWER9 systems

For information concerning HMC releases and the latest PTFs,  go to the following URL to access Fix Central:
http://www-933.ibm.com/support/fixcentral/

For specific fix level information on key components of IBM Power Systems running the AIX, IBM i and Linux operating systems, we suggest using the Fix Level Recommendation Tool (FLRT):
http://www14.software.ibm.com/webapp/set2/flrt/home


NOTES:

                -You must be logged in as hscroot in order for the firmware installation to complete correctly.
                - Systems Director Management Console (SDMC) does not support this System Firmware level

2.0 Important Information

Boot adapter microcode requirement

Update all adapters which are boot adapters, or which may be used as boot adapters in the future, to the latest microcode from IBM Fix Central.  The latest microcode will ensure the adapters support the new Firmware Secure Boot feature of Power Systems. This requirement applies when updating system firmware from a level prior to FW940 to levels FW940 and later.
The latest adapter microcode levels include signed boot driver code. If a boot-capable PCI adapter is not installed with the latest level of adapter microcode, the partition which owns the adapter will boot, but error logs with SRCs BA5400A5 or BA5400A6 will be posted.  Once the adapter(s) are updated, the error logs will no longer be posted.

Downgrading firmware from any given release level to an earlier release level is not recommended

Firmware downgrade warnings:
1) Adapter feature codes (#EC2S/#EC2U and #EC3M and #EC66) when configured in SR-IOV shared mode in FW930 or later, even if originally configured in shared mode in a pre-FW930 release, may not function properly if the system is downgraded to a pre-FW930 release. The adapter should be configured in dedicated mode first (i.e. take the adapter out of SR-IOV shared mode) before downgrading to a pre-FW930 release.
2) If partitions have been run in POWER9 compatibility mode in FW940, a downgrade to an earlier release (pre-FW940) may cause a problem with the partitions starting.  To prevent this problem, the "server firmware" settings must be reset by rebooting partitions in "Power9_base" before doing the downgrade.

If you feel that it is necessary to downgrade the firmware on your system to an earlier release level, please contact your next level of support.

2.1 IPv6 Support and Limitations

IPv6 (Internet Protocol version 6) is supported in the System Management Services (SMS) in this level of system firmware. There are several limitations that should be considered.

When configuring a network interface card (NIC) for remote IPL, only the most recently configured protocol (IPv4 or IPv6) is retained. For example, if the network interface card was previously configured with IPv4 information and is now being configured with IPv6 information, the IPv4 configuration information is discarded.

A single network interface card may only be chosen once for the boot device list. In other words, the interface cannot be configured for the IPv6 protocol and for the IPv4 protocol at the same time.

2.2 Concurrent Firmware Updates

Concurrent system firmware update is supported on HMC Managed Systems only.

Ensure that there are no RMC connections issues for any system partitions prior to applying the firmware update.  If there is a RMC connection failure to a partition during the firmware update, the RMC connection will need to be restored and additional recovery actions for that partition will be required to complete partition firmware updates.

2.3 Memory Considerations for Firmware Upgrades

Firmware Release Level upgrades and Service Pack updates may consume additional system memory.
Server firmware requires memory to support the logical partitions on the server. The amount of memory required by the server firmware varies according to several factors.
Factors influencing server firmware memory requirements include the following:
Generally, you can estimate the amount of memory required by server firmware to be approximately 8% of the system installed memory. The actual amount required will generally be less than 8%. However, there are some server models that require an absolute minimum amount of memory for server firmware, regardless of the previously mentioned considerations.

Additional information can be found at:
https://www.ibm.com/support/knowledgecenter/9040-MR9/p9hat/p9hat_lparmemory.htm

2.4 SBE Updates

Power 9 servers contain SBEs (Self Boot Engines) and are used to boot the system.  SBE is internal to each of the Power 9 chips and used to "self boot" the chip.  The SBE image is persistent and is only reloaded if there is a system firmware update that contains a SBE change.  If there is a SBE change and system firmware update is concurrent, then the SBE update is delayed to the next IPL of the CEC which will cause an additional 3-5 minutes per processor chip in the system to be added on to the IPL.  If there is a SBE change and the system firmware update is disruptive, then SBE update will cause an additional 3-5 minutes per processor chip in the system to be added on to the IPL.  During the SBE update process, the HMC or op-panel will display service processor code C1C3C213 for each of the SBEs being updated.  This is a normal progress code and system boot should be not be terminated by the user.  Additional time estimate can be between 12-20 minutes.

The SBE image is only updated with this service pack if the starting firmware level is less than FW950.10.


3.0 Firmware Information

Use the following examples as a reference to determine whether your installation will be concurrent or disruptive.

For systems that are not managed by an HMC, the installation of system firmware is always disruptive.

Note: The concurrent levels of system firmware may, on occasion, contain fixes that are known as Deferred and/or Partition-Deferred. Deferred fixes can be installed concurrently, but will not be activated until the next IPL. Partition-Deferred fixes can be installed concurrently, but will not be activated until a partition reactivate is performed. Deferred and/or Partition-Deferred fixes, if any, will be identified in the "Firmware Update Descriptions" table of this document. For these types of fixes (Deferred and/or Partition-Deferred) within a service pack, only the fixes in the service pack which cannot be concurrently activated are deferred.

Note: The file names and service pack levels used in the following examples are for clarification only, and are not necessarily levels that have been, or will be released.

System firmware file naming convention:

01VMxxx_yyy_zzz

NOTE: Values of service pack and last disruptive service pack level (yyy and zzz) are only unique within a release level (xxx). For example, 01VM900_040_040 and 01VM910_040_045 are different service packs.

An installation is disruptive if:

            Example: Currently installed release is 01VM900_040_040, new release is 01VM910_050_050.

            Example: VM910_040_040 is disruptive, no matter what level of VM910 is currently installed on the system.

            Example: Currently installed service pack is VM910_040_040 and new service pack is VM910_050_045.

An installation is concurrent if:

The release level (xxx) is the same, and
The service pack level (yyy) currently installed on the system is the same or higher than the last disruptive service pack level (zzz) of the service pack to be installed.

Example: Currently installed service pack is VM910_040_040, new service pack is VM910_041_040.

3.1 Firmware Information and Description

 
Filename Size Checksum md5sum
01VM950_087_045.rpm 161116512
10549
948572967bbd2b58cdf2756fcb46943f  

Note: The Checksum can be found by running the AIX sum command against the rpm file (only the first 5 digits are listed).
ie: sum 01VM950_087_045.rpm

VM950
For Impact, Severity and other Firmware definitions, Please refer to the below 'Glossary of firmware terms' url:
http://www14.software.ibm.com/webapp/set2/sas/f/power5cm/home.html#termdefs

The complete Firmware Fix History for this Release Level can be reviewed at the following url:
http://download.boulder.ibm.com/ibmdl/pub/software/server/firmware/VM-Firmware-Hist.html
VM950_087_045 / FW950.20

09/16/21
Impact: Data    Severity:  HIPER

New Features and Functions
  • Support added for a mainstream 800GB NVME U.2 7 mm SSD (Solid State Drive) PCIe4 drive in a 7 mm carrier with Feature Code #EC7Q and CCIN 59B4 for AIX, Linux, and VIOS.
    This PCIe4 drive is also compatible with PCIe3 slots on the system.
  • Support was changed to disable Service Location Protocol (SLP) by default for newly shipped systems or systems that are reset to manufacturing defaults.  This change has been made to reduce memory usage on the service processor by disabling a service that is not needed for normal system operations.  This change can be made manually for existing customers by changing it in ASMI with the options "ASMI -> System Configuration -> Security -> External Services Management" to disable the service.
  • Support was added to generate a service processor fipsdump whenever there is Hostboot (HB) TI and HB dump.  Without this new support, a HB crash (with a HB dump) does not generate a fipsdump and the FSP FFDC at that point in time. So it was difficult to correlate what was seen in the HB dump to what was happening on the FSP at the time of the HB fail.

System firmware changes that affect all systems

  • HIPER:  A problem was fixed which may occur on a target system following a Live Partition Mobility (LPM) migration of an AIX partition utilizing Active Memory Expansion (AME) with 64 KB page size enabled using the vmo tunable: "vmo -ro ame_mpsize_support=1".  The problem may result in AIX termination, file system corruption, application segmentation faults, or undetected data corruption.
    Note:  If you are doing an LPM migration of an AIX partition utilizing AME and 64 KB page size enabled involving a POWER8 or POWER9 system, ensure you have a Service Pack including this change for the appropriate firmware level on both the source and target systems.
  • A problem was fixed for a missing hardware callout and guard for a processor chip failure with SRC BC8AE540 and signature "ex(n0p0c5) (L3FIR[28]) L3 LRU array parity error".
  • A problem was fixed for a missing hardware callout and guard for a processor chip failure with Predictive Error (PE) SRC BC70E540 and signature "ex(n1p2c6) (L2FIR[19]) Rc or NCU Pb data CE error".  The PE error occurs after the number of CE errors reaches a threshold of 32 errors per day.
  • A problem was fixed for an infrequent SRC of B7006956 that may occur during a system power off.  This SRC indicates that encrypted NVRAM locations failed to synchronize with the copy in memory during the shutdown of the hypervisor. This error can be ignored as the encrypted NVRAM information is stored in a redundant location, so the next IPL of the system is successful.
  • A problem was fixed for a service processor mailbox (mbox) timeout error with SRC B182953C during the IPL of systems with large memory configurations and "I/O Adapter Enlarged Capacity" enabled from ASMI.  The error indicates that the hypervisor did not respond quickly enough to a message from the service processor but this may not result in an IPL failure.  The problem is intermittent, so if the IPL does fail, the workaround is to retry the IPL.
  • A problem was fixed for a misleading SRC B7006A20 (Unsupported Hardware Configuration) that can occur for some error cases for PCIes #EMX0 expansion drawers that are connected with copper cables.  For cable unplug errors, the SRC B7006A88 (Drawer TrainError) should be shown instead of the B7006A20.   If a B7006A20 is logged against copper cables with the signature "Prc UnsupportedCableswithFewerChannels" and the message "NOT A 12CHANNEL CABLE", this error should instead follow the service actions for a B7006A88 SRC.  
  • Problems were fixed for DLPAR operations that change the uncapped weight of a partition and DLPAR operations that switch an active partition from uncapped to capped.  After changing the uncapped weight, the weight can be incorrect.  When switching an active partition from uncapped to capped, the operation can fail.
  • A problem was fixed where the Floating Point Unit Computational Test, which should be set to "staggered" by default, has been changed in some circumstances to be disabled. If you wish to re-enable this option, this fix is required.  After applying this service pack, do the following steps:
    1) Sign in to the Advanced System Management Interface (ASMI).
    2) Select Floating Point Computational Unit under the System Configuration heading and change it from disabled to what is needed: staggered (run once per core each day) or periodic (a specified time).
    3) Click "Save Settings".
  • A problem was fixed for a hypervisor hang and HMC Incomplete error as a secondary problem after an SR-IOV adapter has gone into error recovery for a failure.  This secondary failure is infrequent because it requires an unrecovered error first for an SR-IOV adapter.
  • A problem was fixed for a system termination with SRC B700F107 following a time facility processor failure with SRC B700F10B.  With the fix, the transparent replacement of the failed processor will occur for the B700F10B if there is a free core, with no impact to the system. 
  • A problem was fixed for an incorrect "Power Good fault" SRC logged for an #EMX0 PCIe3 expansion drawer on the lower CXP cable of B7006A85 (AOCABLE, PCICARD).  The correct SRC is B7006A86 (PCICARD, AOCABLE).
  • A problem was fixed for a Live Partition Mobility (LPM) migration that failed with the error "HSCL3659 The partition migration has been stopped because orchestrator detected an error" on the HMC.  This problem is intermittent and rare that is triggered by the HMC being overrun with unneeded LPM message requests from the hypervisor that can cause a timeout in HMC queries that result in the LPM operation being aborted.  The workaround is to retry the LPM migration which will normally succeed.
  • A problem was fixed for an SR-IOV adapter in shared mode configured as Virtual Ethernet Port Aggregator (VEPA) where unmatched unicast packets were not forwarded to the promiscuous mode VF.
    Update instructions:  https://www.ibm.com/support/knowledgecenter/en/POWER9/p9efd/p9efd_updating_sriov_firmware.htm.
  • A problem was fixed for certain SR-IOV adapters in SR-IOV Shared mode which may cause a network interruption and SRCs B400FF02 and B400FF04 logged.  The problem occurs infrequently during normal network traffic..
    This fix updates the adapter firmware to 11.4.415.38 for the following Feature Codes and CCINs: #EN15/#EN16 with CCIN 2CE3, #EN17/#EN18 with CCIN 2CE4, #EN0H/#EN0J with CCIN 2B93, #EN0M/#EN0N with CCIN 2CC0, and #EN0K/#EN0L with CCIN 2CC1.
    Update instructions:  https://www.ibm.com/support/knowledgecenter/en/POWER9/p9efd/p9efd_updating_sriov_firmware.htm.
  • A problem was fixed for the Device Description in a System Plan related to Crypto Coprocessors and NVMe cards that were only showing the PCI vendor and device ID of the cards.  This is not enough information to verify which card is installed without looking up the PCI IDs first.  With the fix, more specific/useful information is displayed and this additional information does not have any adverse impact on sysplan operations.  The problem is seen every time a System Plan is created for an installed Crypto Coprocessor or NVMe card.
  • A problem was fixed for some serviceable events specific to the reporting of EEH errors not being displayed on the HMC.  The sending of an associated call home event, however, was not affected.  This problem is intermittent and infrequent.
  • A problem was fixed for possible partition errors following a concurrent firmware update from FW910 or later. A precondition for this problem is that DLPAR operations of either physical or virtual I/O devices must have occurred prior to the firmware update.  The error can take the form of a partition crash at some point following the update. The frequency of this problem is low.  If the problem occurs, the OS will likely report a DSI (Data Storage Interrupt) error.  For example, AIX produces a DSI_PROC log entry.  If the partition does not crash, it is also possible that some subsequent I/O DLPAR operations will fail.
  • A problem was fixed for Platform Error Logs (PELS) not being logged and shown by the OS if they have an Error Severity code of "critical error".   The trigger is the reporting by a system firmware subsystem of an error log that has set an Event/Error Severity in the 'UH' section of the log to a value in the range, 0x50 to 0x5F.  The following error logs are affected:
    B200308C ==> PHYP ==>  A problem occurred during the IPL of a partition.  The adapter type cannot be determined. Ensure that a valid I/O Load Source is tagged.
    B700F104 ==> PHYP ==> Operating System error.  Platform Licensed Internal Code terminated a partition.
    B7006990 ==> PHYP ==> Service processor failure
    B2005149 ==> PHYP ==>  A problem occurred during the IPL of a partition.
    B700F10B ==> PHYP ==>  A resource has been disabled due to hardware problems
    A7001150 ==> PHYP ==> System log entry only, no service action required. No action needed unless a serviceable event was logged.
    B7005442 ==> PHYP ==> A parity error was detected in the hardware Segment Lookaside Buffer (SLB).
    B200541A ==> PHYP ==> A problem occurred during a partition Firmware Assisted Dump
    B7001160 ==> PHYP ==> Service processor failure.
    B7005121 ==> PHYP ==> Platform LIC failure
    BC8A0604 ==> Hostboot  ==> A problem occurred during the IPL of the system.
    BC8A1E07 ==> Hostboot  ==>  Secure Boot firmware validation failed.
    Note that these error logs are still reported to the service processor and HMC properly.  This issue does not affect the Call Home action for the error logs.
  • A problem was fixed for Live Partition Mobility (LPM) migrations from non-trusted POWER9 systems to POWER10 systems. The LPM migration failure occurs every time a LPM migration is attempted from a non-trusted system source to FW1010 and later.  For POWER9 systems, non-trusted is the default setting.  The messages shown on the HMC for the failure are the following:
     HSCL365C The partition migration has been stopped because platform firmware detected an error (041800AC).
     HSCL365D The partition migration has been stopped because target MSP detected an error (05000127).
     HSCL365D The partition migration has been stopped because target MSP detected an error (05000127).
    A workaround for the problem is to enable the trusted system key on the POWER9 FW940/FW950 source system which can be done using an intricate procedure.  Please contact IBM Support for help with this workaround.
  • A problem was fixed for a missing error log SRC for an SR-IOV adapter in Shared mode that fails during the IPL because of adapter failure or because the system has insufficient memory for SR-IOV Shared mode for the adapter.  The error log SRC added is B7005308, indicating a serviceable event and providing the adapter and error information.
  • A problem was fixed for a Live Partition Mobility (LPM) migration failure from a POWER9 FW950 source to a POWER10 FW1010 target.  This will fail on every attempt with the following message on the HMC:
    "HSCLA2CF The partition migration has been stopped unexpectedly.  Perform a migration recovery for this partition, if necessary."
  • A problem was fixed for error logs not being sent over to HMC when disconnecting/reconnecting power cords that caused a flood on informational SRCs of B1818A37 and B18187D7.  After the flood of error logs, the reporting of error logs to the HMC stopped, which also prevented Call Home from working.  To recover from the error, the service processor can have a reset/reload done using ASMI.

System firmware changes that affect certain systems

  • For a system with a partition running AIX 7.3, a problem was fixed for running Live Update or Live Partition Mobility (LPM).  AIX 7.3 supports Virtual Persistent Memory (PMEM) but it cannot be used with these operations, but the problem was making it appear that PMEM was configured when it was not.  The Live Update and LPM operations always fail when attempted on AIX 7.3.  Here is the failure output from a Live Update Preview:
    "1430-296 FAILED: not all devices are virtual devices.
    nvmem0
    1430-129 FAILED: The following loaded kernel extensions are not known to be safe for Live Update:
    nvmemdd
    ...
    1430-218 The live update preview failed.
    0503-125 geninstall:  The lvupdate call failed.
    Please see /var/adm/ras/liveupdate/logs/lvupdlog for details."
  • On systems with only Integrated Facility for Linux ( IFL) processors and AIX partitions, a problem was fixed for performance issues for IFL VMs (Linux and VIOS).  This problem occurs if AIX partitions are active on a system with IPL only cores.  As a workaround, AIX partitions should not be activated on an IFL only system.  With the fix, the activation of AIX partitions is blocked on an IFL only system.  If this fix is installed concurrently with AIX partitions running, these partitions will be allowed to continue to run until they are powered off.  Once powered off, the AIX partitions will not be allowed to be activated again on the IFL-only system.
  • For systems with an AIX partition and Platform Keystore (PKS) enabled for the partition, a problem was fixed for AIX not being able to access the PKS during a Main Store Dump (MSD) IPL.  This may prevent the dump from completing.  This will happen for every MSD IPL when the partition PKS is enabled and in use by the AIX OS.
  • For a system with an AIX or Linux partition, a problem was fixed for a boot hang in RTAS for a partition that owns I/O which uses MSI-X interrupts.  A BA180007 SRC may be logged prior to the hang.  The frequency of this RTAS hang error is very low. 
VM950_075_045 / FW950.11

06/08/21
Impact:  Availability     Severity:  HIPER

System firmware changes that affect all systems
  • HIPER/Pervasive:  A problem was fixed for a checkstop due to an internal Bus transport parity error or a data timeout on the Bus.  This is a very rare problem that requires a particular SMP transport link traffic pattern and timing.  Both the traffic pattern and timing are very difficult to achieve with customer application workloads.  The fix will have no measurable effect on most customer workloads although highly intensive OLAP-like workloads may see up to 2.5% impact.
VM950_072_045 / FW950.10

04/28/21
Impact:  Availability      Severity:  SPE

New Features and Functions
  • Support added to Redfish to provide a command to set the ASMI user passwords using a new AccountService schema. Using this service, the ASMI admin, HMC, and general user passwords can be changed.
  • PowerVM support for the Platform KeyStore (PKS) for partitions has removed the FW950.00 restriction where the total amount of PKS for the system that could be configured was limited to 1 MB across all the partitions. This restriction has been removed for FW950.10.
  • Support was added for Samsung DIMMs with part number 01GY853.  If these DIMMs are installed in a system with older firmware than FW950.10, the DIMMs will fail and be guarded with SRC BC8A090F logged with HwpReturnCode " RC_CEN_MBVPD_TERM_DATA_UNSUPPORTED_VPD_ENCODE".
  • Support was added for a new service processor command that can be used to 'lock' the power management mode, such that the mode can not be changed except by doing a factory reset.
  • Support for new mainstream 931 GB, 1.86 TB, 3.72 TB, and 7.44 TB capacity SSDs.  A 2.5-inch serial-attached SCSI (SAS) SSD is mounted on an SFF-3 carrier or tray for a POWER9 system unit or mounted on an SFF-2 for placement in an expansion drawer, such as the EXP24SX drawer, when attached to a POWER9 server. The drive is formatted to use 4224-byte (4k) sectors and does not support the 4k JBOD 4096-byte sector. Firmware level FW950.10 or later is required for these drives. The following are the feature codes and CCINs for the new drives:
    #ESKJ/#ESKK with CCIN 5B2B/5B29 – 931 GB Mainstream SAS 4k SFF-3/SFF-2 SSD for AIX/Linux
    #ESKL/#ESKM with CCIN 5B2B/5B29 - 931GB Mainstream SAS 4k SFF-3/SFF-2 SSD for IBM i
    #ESKN/#ESKP with CCIN 5B20/5B21- 1.86TB Mainstream SAS 4k SFF-3/SFF-2 SSD for AIX/Linux
    #ESKQ/#ESKR with CCIN 5B20/5B21- 1.86TB Mainstream SAS 4k SFF-3/SFF-2 SSD for IBM i
    #ESKS/#ESKT with CCIN 5B2C/5B2D - 3.72TB Mainstream SAS 4k SFF-3/SFF-2 SSD for AIX/Linux
    #ESKU/#ESKV with CCIN 5B2C/5B2D - 3.72TB Mainstream SAS 4k SFF-3/SFF-2 SSD for IBM i
    #ESKW/#ESKX with CCIN 5B2E/5B2F- 7.44TB Mainstream SAS 4k SFF-3/SFF-2 SSD for AIX/Linux
    #ESKY/#ESKZ with CCIN 5B2E/5B2F -7.44TB Mainstream SAS 4k SFF-3/SFF-2 SSD for IBM i
  • Support for new enterprise SSDs refresh the previously available 387 GB, 775 GB, and 1550 GB capacity points for POWER9 servers. These are 400 GB, 800 GB, and 1600 GB SSDs that are always formatted either to 4224 (4k) byte sectors or to 528 (5xx) byte sectors for additional protection, resulting in 387 GB, 775 GB, and 1550 GB capacities. The 4096-byte sector, the 512-byte sector, and JBOD are not supported.  Firmware level FW950.10 or later is required for these drives. The following are the feature codes and CCINs for the new drives:
    #ESK0/#ESK1 with CCIN 5B19/ 5B16  - 387GB Enterprise SAS 5xx SFF-3/SFF-2 SSD for AIX/Linux
    #ESK2/#ESK3 with CCIN 5B1A/5B17 - 775GB Enterprise SAS 5xx SFF-3/SFF-2 SSD for AIX/Linux
    #ESK6/#ESK8 with CCIN 5B13/5B10.- 387GB Enterprise SAS 4k SFF-3/SFF-2 SSD for AIX/Linux
    #ESK7/#ESK9 with CCIN 5B13/5B10- 387GB Enterprise SAS 4k SFF-3/SFF-2 SSD for IBM i
    #ESKA/#ESKC with CCIN 5B14/5B11- 775GB Enterprise SAS 4k SFF-3/SFF-2 SSD for AIX/Linux
    #ESKB/#ESKD with CCIN 5B14/5B11- 775GB Enterprise SAS 4k SFF-3/SFF-2 SSD for IBM i
    #ESKE/#ESKG with CCIN 5B15/5B12- 1.55TB Enterprise SAS 4k SFF-3/SFF-2 SSD for AIX/Linux
    #ESKF/#ESKH with CCIN 5B15/5B12- 1.55TB Enterprise SAS 4k SFF-3/SFF-2 SSD for IBM i
  • Support for new PCIe 4.0 x8 dual-port 32 Gb optical Fibre Channel (FC) short form adapter based on the Marvell QLE2772 PCIe host bus adapter (6.6 inches x 2.731 inches). The adapter provides two ports of 32 Gb FC capability using SR optics. Each port can provide up to 6,400 MBps bandwidth. This adapter has feature codes #EN1J/#EN1K with CCIN 579C. Firmware level FW950.10 or later is required for this adapter.
  • Support for new PCIe 3.0 16 Gb quad-port optical Fibre Channel (FC)l x8 short form adapter based on the Marvell QLE2694L PCIe host bus adapter (6.6 inches x 2.371 inches). The adapter provides four ports of 16 Gb FC capability using SR optics. Each port can provide up to 3,200 MBps bandwidth. This adapter has feature codes #EN1E/#EN1F with CCIN 579A. Firmware level FW950.10 or later is required for this adapter.
System firmware changes that affect all systems
  • DEFERRED:  A problem was fixed for a rare Voltage Regulator Module (VRM) power fault with an SRC 11002700 logged for the VRM failure followed by an SRC 11002610 system crash.  The trigger for this problem is intense workloads that cause what appear to be input over-current conditions.  A re-IPL of the system is needed to activate this fix.
  • A problem was fixed for certain SR-IOV adapters that have a rare, intermittent error with B400FF02 and B400FF04 logged, causing a reboot of the VF.  The error is handled and recovered without any user intervention needed.  The SR-IOV adapters affected have the following Feature Codes and CCINs: #EC2R/#EC2S with CCIN 58FA; #EC2T/#EC2U with CCIN 58FB; #EC3L/#EC3M with CCIN 2CE; and #EC66/#EC67 with CCIN 2CF3.
  • A problem was fixed for initiating a Remote Restart from a PowerVC/NovaLink source system to a remote target.  This happens whenever the source system is running FW950.00.  The error would look like this from PowerVC (system name, release level would be specific to the environment):
    "Virtual machine RR-5 could not be remote restarted to Ubu_AX_9.114.255.10. Error message: PowerVM API failed to complete for instance=RR-5-71f5c2cf-0000004e.HTTP error 500 for method PUT on path /rest/api/uom/ManagedSystem/598c1be4-cb4c-3957-917d-327b764d8ac1/LogicalPartition: Internal Server Error -- [PVME01040100-0004] Internal error PVME01038003 occurred while trying to perform this command."
  • A problem was fixed for a B1502616 SRC logged after a system is powered off.  This rare error, "A critical error occurred on the thermal/power management device (TPMD); it is being disabled. " is not a real problem but occurred because the Power Management (PM) complex was being reset during the power off.  No recovery is needed as the next IPL of the system is successful.
  • A problem was fixed for the error handling of a system with an unsupported memory configuration that exceeds available memory power. Without the fix, the IPL of the system is attempted and fails with a segmentation fault with SRCs B1818611 and B181460B logged that do not call out the incorrect DIMMs.
  • A problem was fixed for an error in the HMC GUI (Error launching task) when clicking on "Hardware Virtualized IO". This error is infrequent and is triggered by an optical cable to a PCIe3 #EMX0 expansion drawer that is failed or unplugged.  With the fix, the HMC can show the working I/O adapters.
  • A problem was fixed for performance degradation of a partition due to task dispatching delays.  This may happen when a processor chip has all of its shared processors removed and converted to dedicated processors. This could be driven by DLPAR remove of processors or Dynamic Platform Optimization (DPO).
  • A problem was fixed for an unrecoverable UE SRC B181BE12 being logged if a service processor message acknowledgment is sent to a Hostboot instance that has already shutdown. This is a harmless error log and it should have been marked as an informational log.
  • A problem was fixed for Time of Day (TOD) being lost for the real-time clock (RTC) with an SRC B15A3303 logged when the service processor boots or resets.  This is a very rare problem that involves a timing problem in the service processor kernel.  If the server is running when the error occurs, there will be an SRC B15A3303 logged, and the time of day on the service processor will be incorrect for up to six hours until the hypervisor synchronizes its (valid) time with the service processor.  If the server is not running when the error occurs, there will be an SRC B15A3303 logged, and If the server is subsequently IPLed without setting the date and time in ASMI to fix it, the IPL will abort with an SRC B7881201 which indicates to the system operator that the date and time are invalid.
  • A problem was fixed for the Systems Management Services ( SMS) menu "Device IO Information" option being incorrect when displaying the capacity for an NVMe or Fibre Channel (FC) NVMe disk. This problem occurs every time the data is displayed.
  • A problem was fixed for intermittent failures for a reset of a Virtual Function (VF) for SR-IOV adapters during Enhanced Error Handling (EEH) error recovery.  This is triggered by EEH events at a VF level only, not at the adapter level. The error recovery fails if a data packet is received by the VF while the EEH recovery is in progress. A VF that has failed can be recovered by a partition reboot or a DLPAR remove and add of the VF.
  • A problem was fixed for a logical partition activation error that can occur when trying to activate a partition when the adapter hardware for an SR-IOV logical port has been physically removed or is unavailable due to a hardware issue. This message is reported on the HMC for the activation failure:  "Error:  HSCL12B5 The operation to remove SR-IOV logical port <number> failed because of the following error: HSCL1552 The firmware operation failed with extended error" where the logical port number will vary.  This is an infrequent problem that is only an issue if the adapter hardware has been removed or another problem makes it unavailable.  The workaround for this problem is to physically add the hardware back in or correct the hardware issue.  If that cannot be done, create an alternate profile for the logical partition without the SR-IOV logical port and use that until the hardware issue is resolved.
  • A problem was fixed for incomplete periodic data gathered by IBM Service for #EMXO PCIe expansion drawer predictive error analysis. The service data is missing the PLX (PCIe switch) data that is needed for the debug of certain errors.
  • A problem was fixed for a partition hang in shutdown with SRC B200F00F logged.  The trigger for the problem is an asynchronous NX accelerator job (such as gzip or NX842 compression) in the partition that fails to clean up successfully.  This is intermittent and does not cause a problem until a shutdown of the partition is attempted.  The hung partition can be recovered by performing an LPAR dump on the hung partition.  When the dump has been completed, the partition will be properly shut down and can then be restarted without any errors.
  • A problem was fixed for a rare failure for an SPCN I2C command sent to a PCIe I/O expansion drawer that can occur when service data is manually collected with hypervisor macros "xmsvc -dumpCCData and xmsvc -logCCErrBuffer".   If the hypervisor macro "xmsvc "is run to gather service data and a CMC Alert occurs at the same time that requires an SPCN command to clear the alert, then the I2C commands may be improperly serialized, resulting in an SPCN I2C command failure. To prevent this problem,  avoid using xmsvc -dumpCCData and xmsvc -logCCErrBuffer to collect service data until this fix is applied.
  • The following problems were fixed for certain SR-IOV adapters:
    1) An error was fixed that occurs during a VNIC failover where the VNIC backing device has a physical port down or read port errors with an SRC B400FF02 logged.
    2) A problem was fixed for adding a new logical port that has a PVID assigned that is causing traffic on that VLAN to be dropped by other interfaces on the same physical port which uses OS VLAN tagging for that same VLAN ID.  This problem occurs each time a logical port with a non-zero PVID that is the same as an existing VLAN is dynamically added to a partition or is activated as part of a partition activation, the traffic flow stops for other partitions with OS configured VLAN devices with the same VLAN ID.  This problem can be recovered by configuring an IP address on the logical port with the non-zero PVID and initiating traffic flow on this logical port.  This problem can be avoided by not configuring logical ports with a PVID if other logical ports on the same physical port are configured with OS VLAN devices.
    This fix updates the adapter firmware to 11.4.415.37 for the following Feature Codes and CCINs: #EN15/#EN16 with CCIN 2CE3, #EN17/#EN18 with CCIN 2CE4, #EN0H/#EN0J with CCIN 2B93, #EN0M/#EN0N with CCIN 2CC0, and #EN0K/#EN0L with CCIN 2CC1.
    The SR-IOV adapter firmware level update for the shared-mode adapters happens under user control to prevent unexpected temporary outages on the adapters.  A system reboot will update all SR-IOV shared-mode adapters with the new firmware level.  In addition, when an adapter is first set to SR-IOV shared mode, the adapter firmware is updated to the latest level available with the system firmware (and it is also updated automatically during maintenance operations, such as when the adapter is stopped or replaced).  And lastly, selective manual updates of the SR-IOV adapters can be performed using the Hardware Management Console (HMC).  To selectively update the adapter firmware, follow the steps given at the IBM Knowledge Center for using HMC to make the updates:  https://www.ibm.com/support/knowledgecenter/en/POWER9/p9efd/p9efd_updating_sriov_firmware.htm.
    Note: Adapters that are capable of running in SR-IOV mode, but are currently running in dedicated mode and assigned to a partition, can be updated concurrently either by the OS that owns the adapter or the managing HMC (if OS is AIX or VIOS and RMC is running).
  • A problem was fixed for a system hang or terminate with SRC B700F105 logged during a Dynamic Platform Optimization (DPO) that is running with a partition in a failed state but that is not shut down.  If DPO attempts to relocate a dedicated processor from the failed partition, the problem may occur.  This problem can be avoided by doing a shutdown of any failed partitions before initiating DPO.
  • A problem was fixed for a system crash with HMC message HSCL025D and SRC B700F103 logged on a Live Partition Mobility (LPM) inactive migration attempt that fails. The trigger for this problem is inactive migration that fails a compatibility check between the source and target systems.
  • A problem was fixed for time-out issues in Power Enterprise Pools 1.0 (PEP 1.0) that can affect performance by having non-optimal assignments of processors and memory to the server logical partitions in the pool.  For this problem to happen, the server must be in a PEP 1.0 pool and the HMC must take longer than 2 minutes to provide the PowerVM hypervisor with the information about pool resources owned by this server.  The problem can be avoided by running the HMC optmem command before activating the partitions.
  • A problem was fixed for certain SR-IOV adapters not being able to create the maximum number of VLANs that are supported for a physical port. There were insufficient memory pages allocated for the physical functions for this adapter type. The SR-IOV adapters affected have the following Feature Codes and CCINs:  #EC66/#EC67 with CCIN 2CF3.
  • A problem was fixed for certain SR-IOV adapters that can have B400FF02 SRCs logged with LPA dumps during a vNIC remove operation.  The adapters can have issues with a deadlock in managing memory pages.  In most cases, the operations should recover and complete.  This fix updates the adapter firmware to XX.29.2003 for the following Feature Codes and CCINs: #EC2R/EC2S with CCIN 58FA; #EC2T/EC2U with CCIN 58FB; #EC3L/EC3M with CCIN 2CE; and #EC66/EC67 with CCIN 2CF3.
    The SR-IOV adapter firmware level update for the shared-mode adapters happens under user control to prevent unexpected temporary outages on the adapters.  A system reboot will update all SR-IOV shared-mode adapters with the new firmware level.  In addition, when an adapter is first set to SR-IOV shared mode, the adapter firmware is updated to the latest level available with the system firmware (and it is also updated automatically during maintenance operations, such as when the adapter is stopped or replaced).  And lastly, selective manual updates of the SR-IOV adapters can be performed using the Hardware Management Console (HMC).  To selectively update the adapter firmware, follow the steps given at the IBM Knowledge Center for using HMC to make the updates:   https://www.ibm.com/support/knowledgecenter/en/POWER9/p9efd/p9efd_updating_sriov_firmware.htm.
    Note: Adapters that are capable of running in SR-IOV mode, but are currently running in dedicated mode and assigned to a partition, can be updated concurrently either by the OS that owns the adapter or the managing HMC (if OS is AIX or VIOS and RMC is running).
VM950_045_045 / FW950.00

11/23/20
Impact:  New      Severity:  New

GA Level with key features included listed below
  • All features and fixes from the FW930.30 and FW940.20 service packs (and below) are included in this release.
New Features and Functions
  • Host firmware support for anti-rollback protection.  This feature implements firmware anti-rollback protection as described in NIST SP 800-147B "BIOS Protection Guidelines for Servers".  Firmware is signed with a "secure version".  Support added for a new menu in ASMI called "Host firmware security policy" to update this secure version level at the processor hardware.  Using this menu, the system administrator can enable the "Host firmware secure version lock-in" policy, which will cause the host firmware to update the "minimum secure version" to match the currently running firmware. Use the "Firmware Update Policy" menu in ASMI to show the current "minimum secure version" in the processor hardware along with the "Minimum code level supported" information. The secure boot verification process will block installing any firmware secure version that is less than the "minimum secure version" maintained in the processor hardware.
    Prior to enabling the "lock-in" policy, it is recommended to accept the current firmware level.
    WARNING: Once lock-in is enabled and the system is booted, the "minimum secure version" is updated and there is no way to roll it back to allow installing firmware releases with a lesser secure version.
    Note:  If upgrading from FW930.30 or FW940.20, this feature is already applied.
  • This server firmware level includes the SR-IOV adapter firmware level 11.4.415.33 for the following Feature Codes and CCINs: #EN15/EN16 with CCIN 2CE3, #EN17/EN18 with CCIN 2CE4, #EN0H/EN0J with CCIN 2B93, #EN0M/EN0N with CCIN 2CC0, and #EN0K/EN0L with CCIN 2CC1.
  • This server firmware includes the SR-IOV adapter firmware level 1x.25.6100 for the following Feature Codes and CCINs: #EC2R/EC2S with CCIN 58FA; #EC2T/EC2U with CCIN 58FB; #EC3L/EC3M with CCIN 2CEC; and #EC66/EC67 with CCIN 2CF3.
  • Support for PCIe4 x8 1.6/3.2/6.4 TB NVMe Adapters that are Peripheral Component Interconnect Express (PCIe) Generation 4 (Gen4) x8 adapters with the following feature codes and CCINs:
    #EC7A/#EC7B with CCIN 594A ; #EC7C/#EC7D with CCIN 594B; and #EC7E/#EC7F with CCIN 594C for AIX/Linux.
    #EC7J/#EC7K with CCIN 594A ; #EC7L/#EC7M with CCIN 594B; and #EC7N/#EC7P with CCIN 594C for IBM i.
  • PowerVM boot support for AIX for NVMe over Fabrics (NVMf) for 32Gb Fibre Channel.  Natively attached adapters are supported with the following feature codes and CCINs: #EN1A/#EN1B with CCIN 578F.
  • Support added for a PCIe2 2-Port USB 3.0 adapter with the following feature codes and CCIN: #EC6J/#EC6K with CCIN 590F.
  • Support added for dedicated processor partitions in IBM Power Enterprise Pools (PEP) 2.0.  Previously, systems added to PEP 2.0 needed to have all partitions as shared processor partitions.
  • Support added for SR-IOV Hybrid Network Virtualization (HNV) for Linux.   This capability allows a Linux partition to take advantage of the efficiency and performance benefits of SR-IOV logical ports and participate in mobility operations such as active and inactive Live Partition Mobility (LPM) and Simplified Remote Restart (SRR).  HNV is enabled by selecting a new Migratable option when an SR-IOV logical port is configured. The Migratable option is used to create a backup virtual device.  The backup virtual device must be a Virtual Ethernet adapter (virtual Network Interface Controller (vNIC) adapter not supported as a backup device). In addition to this firmware, HNV support in a production environment requires HMC 9.1.941.0 or later, RHEL 8., SLES 15, and VIOS 3.1.1.20 or later.
  • Enhanced  Dynamic DMA Window (DDW) for I/O adapter slots to enable the OS to use 64KB TCEs.   The OS supported is Linux RHEL 8.3 LE.
  • PowerVM support for the Platform KeyStore (PKS) for partitions.  PowerVM has added new h-call interfaces allowing the partition to interact with the Platform KeyStore that is maintained by PowerVM.  This keystore can be used by the partition to store items requiring confidentiality or integrity like encryption keys or certificates.
    Note:  The total amount of PKS for the system is limited to 1 MB across all the partitions for FW950.00.
  • Support for 64 GB 16Gbit DDR4 system memory running at 2666 Mhz with feature code #EM9B and part number 78P6815.
  • Support for 128 GB 16Gbit DDR4 system memory running at 2666 Mhz with feature code #EM9C and part number 78P6925.
System firmware changes that affect all systems
  • HIPER/Pervasive:  A problem was fixed to be able to detect a failed PFET sensing circuit in a core at runtime, and prevent a system fail with an incomplete state when a core fails to wake up.  The failed core is detected on the subsequent IPL.  With the fix. a core is called out with the PFET failure with SRC BC13090F and hardware description "CME detected malfunctioning of PFET headers." to isolate the error better with a correct callout.
  • A problem was fixed for a VIOS, AIX, or Linux partition hang during an activation at SRC CA000040.  This will occur on a system that has been running more than 814 days when the boot of the partition is attempted if the partitions are in POWER9_base or POWER9 processor compatibility mode.
    A workaround to this problem is to re-IPL the system or to change the failing partition to POWER8 compatibility mode.
    Note:  If upgrading from FW930.30, this fix is already applied.

4.0 How to Determine The Currently Installed Firmware Level

You can view the server's current firmware level on the Advanced System Management Interface (ASMI) Welcome pane. It appears in the top right corner. Example: VM920_123.


5.0 Downloading the Firmware Package

Follow the instructions on Fix Central. You must read and agree to the license agreement to obtain the firmware packages.

Note: If your HMC is not internet-connected you will need to download the new firmware level to a USB flash memory device or ftp server.


6.0 Installing the Firmware

The method used to install new firmware will depend on the release level of firmware which is currently installed on your server. The release level can be determined by the prefix of the new firmware's filename.

Example: VMxxx_yyy_zzz

Where xxx = release level

Instructions for installing firmware updates and upgrades can be found at https://www.ibm.com/support/knowledgecenter/9040-MR9/p9eh6/p9eh6_updates_sys.htm

IBM i Systems:

For information concerning IBM i Systems, go to the following URL to access Fix Central: 
http://www-933.ibm.com/support/fixcentral/

Choose "Select product", under Product Group specify "System i", under Product specify "IBM i", then Continue and specify the desired firmware PTF accordingly.

HMC and NovaLink Co-Managed Systems (Disruptive firmware updates only):

A co-managed system is managed by HMC and NovaLink, with one of the interfaces in the co-management master mode.
Instructions for installing firmware updates and upgrades on systems co-managed by an HMC and Novalink is the same as above for a HMC managed systems since the firmware update must be done by the HMC in the co-management master mode.  Before the firmware update is attempted, one must be sure that HMC is set in the master mode using the steps at the following IBM KnowledgeCenter link for NovaLink co-managed systems:
https://www.ibm.com/support/knowledgecenter/9009-22A/p9eig/p9eig_kickoff.htm

Then the firmware updates can proceed with the same steps as for the HMC managed systems except the system must be powered off because only a disruptive update is allowed.   If a concurrent update is attempted, the following error will occur: " HSCF0180E Operation failed for <system name> (<system mtms>).  The operation failed.  E302F861 is the error code:"
https://www.ibm.com/support/knowledgecenter/9009-22A/p9eh6/p9eh6_updates_sys.htm

7.0 Firmware History

The complete Firmware Fix History (including HIPER descriptions)  for this Release level can be reviewed at the following url:
http://download.boulder.ibm.com/ibmdl/pub/software/server/firmware/VM-Firmware-Hist.html