Power9 System Firmware

Applies to:   9009-22G; 9009-41G; 9009-42G; 5105-22E .

This document provides information about the installation of Licensed Machine or Licensed Internal Code, which is sometimes referred to generically as microcode or firmware.

Contents

1.0 Systems Affected

This package provides firmware for Power Systems S922 (9009-22G), Power Systems S914 (9009-41G), Power Systems S924 (9009-42G), and IBM ESS (5105-22E) servers only.

The firmware level in this package is:

1.1 Minimum HMC Code Level

This section is intended to describe the "Minimum HMC Code Level" required by the System Firmware to complete the firmware installation process. When installing the System Firmware, the HMC level must be equal to or higher than the "Minimum HMC Code Level" before starting the system firmware update.  If the HMC managing the server targeted for the System Firmware update is running a code level lower than the "Minimum HMC Code Level" the firmware update will not proceed.

The Minimum HMC Code levels for this firmware for HMC x86,  ppc64 or ppc64le are listed below.

x86 -  This term is used to reference the legacy HMC that runs on x86/Intel/AMD hardware for both the 7042 Machine Type appliances and the Virtual HMC that can run on the Intel hypervisors (KVM, VMWare, Xen).
ppc64 or ppc64le - describes the Linux code that is compiled to run on Power-based servers or LPARS (Logical Partitions)
For information concerning HMC releases and the latest PTFs,  go to the following URL to access Fix Central:
http://www-933.ibm.com/support/fixcentral/

For specific fix level information on key components of IBM Power Systems running the AIX, IBM i and Linux operating systems, we suggest using the Fix Level Recommendation Tool (FLRT):
http://www14.software.ibm.com/webapp/set2/flrt/home


NOTES:
                -You must be logged in as hscroot in order for the firmware installation to complete correctly.
                - Systems Director Management Console (SDMC) does not support this System Firmware level

1.2 IBM i Minimum Levels

Reference the following URL for IBM i Support: Recommended fixes:

https://www-01.ibm.com/support/docview.wss?uid=nas8N1021194

2.0 Important Information

NovaLink levels earlier than "NovaLink 1.0.0.16 Feb 2020 release" with partitions running certain SR-IOV capable adapters are NOT supported at this firmware release

NovaLink levels earlier than "NovaLink 1.0.0.16 Feb 2020 release" do not support IO adapter FCs EC2R/EC2S, EC2T/EC2U, EC3L/EC3M, EC66/EC67.  If the system contains any of these feature codes, then NovaLink must first be updated to "NovaLink 1.0.0.16 Feb 2020 release"  or later in order to manage the system via NovaLink / PowerVC combination.

Downgrading firmware from any given release level to an earlier release level is not recommended

Firmware downgrade warning:
It is not possible to downgrade from this level to an earlier level.  FW941.00 is a release specifically for the 9009-22G, 9009-41G and 9009-42G servers which will not operate with any firmware below FW941.00 

2.1 IPv6 Support and Limitations

IPv6 (Internet Protocol version 6) is supported in the System Management Services (SMS) in this level of system firmware. There are several limitations that should be considered.

When configuring a network interface card (NIC) for remote IPL, only the most recently configured protocol (IPv4 or IPv6) is retained. For example, if the network interface card was previously configured with IPv4 information and is now being configured with IPv6 information, the IPv4 configuration information is discarded.

A single network interface card may only be chosen once for the boot device list. In other words, the interface cannot be configured for the IPv6 protocol and for the IPv4 protocol at the same time.

2.2 Concurrent Firmware Updates

Concurrent system firmware update is supported on HMC Managed Systems only.

Ensure that there are no RMC connections issues for any system partitions prior to applying the firmware update.  If there is a RMC connection failure to a partition during the firmware update, the RMC connection will need to be restored and additional recovery actions for that partition will be required to complete partition firmware updates.

2.3 Memory Considerations for Firmware Upgrades

Firmware Release Level upgrades and Service Pack updates may consume additional system memory.
Server firmware requires memory to support the logical partitions on the server. The amount of memory required by the server firmware varies according to several factors.
Factors influencing server firmware memory requirements include the following:
Generally, you can estimate the amount of memory required by server firmware to be approximately 8% of the system installed memory. The actual amount required will generally be less than 8%. However, there are some server models that require an absolute minimum amount of memory for server firmware, regardless of the previously mentioned considerations.

Additional information can be found at:
https://www.ibm.com/support/knowledgecenter/9009-22A/p9hat/p9hat_lparmemory.htm

2.4 SBE Updates

Power 9 servers contain SBEs (Self Boot Engines) and are used to boot the system.  SBE is internal to each of the Power 9 chips and used to "self boot" the chip.  The SBE image is persistent and is only reloaded if there is a system firmware update that contains a SBE change.  If there is a SBE change and system firmware update is concurrent, then the SBE update is delayed to the next IPL of the CEC which will cause an additional 3-5 minutes per processor chip in the system to be added on to the IPL.  If there is a SBE change and the system firmware update is disruptive, then SBE update will cause an additional 3-5 minutes per processor chip in the system to be added on to the IPL.  During the SBE update process, the HMC or op-panel will display service processor code C1C3C213 for each of the SBEs being updated.  This is a normal progress code and system boot should be not be terminated by the user. Additional time estimate can be between 6-10 minutes.

The FW941.00 image is applied at the factory and updates to FW941.00 from a lower level are not allowed, therefore the SBE update delay does not apply to FW941.00

3.0 Firmware Information

Use the following examples as a reference to determine whether your installation will be concurrent or disruptive.

For systems that are not managed by an HMC, the installation of system firmware is always disruptive.

Note: The concurrent levels of system firmware may, on occasion, contain fixes that are known as Deferred and/or Partition-Deferred. Deferred fixes can be installed concurrently, but will not be activated until the next IPL. Partition-Deferred fixes can be installed concurrently, but will not be activated until a partition reactivate is performed. Deferred and/or Partition-Deferred fixes, if any, will be identified in the "Firmware Update Descriptions" table of this document. For these types of fixes (Deferred and/or Partition-Deferred) within a service pack, only the fixes in the service pack which cannot be concurrently activated are deferred.

Note: The file names and service pack levels used in the following examples are for clarification only, and are not necessarily levels that have been, or will be released.

System firmware file naming convention:

01VLxxx_yyy_zzz

NOTE: Values of service pack and last disruptive service pack level (yyy and zzz) are only unique within a release level (xxx). For example, 01VL900_040_040 and 01VL910_040_045 are different service packs.

An installation is disruptive if:

            Example: Currently installed release is 01VL900_040_040, new release is 01VL910_050_050.

            Example: VL910_040_040 is disruptive, no matter what level of VL910 is currently installed on the system.

            Example: Currently installed service pack is VL910_040_040 and new service pack is VL910_050_045.

An installation is concurrent if:

The release level (xxx) is the same, and
The service pack level (yyy) currently installed on the system is the same or higher than the last disruptive service pack level (zzz) of the service pack to be installed.

Example: Currently installed service pack is VL910_040_040, new service pack is VL910_041_040.

3.1 Firmware Information and Description

 
Filename Size Checksum md5sum
01VL941_045_035.rpm  136003130 22668 7f5fb82308dd513ddacce4e42426b137

Note: The Checksum can be found by running the AIX sum command against the rpm file (only the first 5 digits are listed).
ie: sum 01VL941_045_035.rpm

VL941
For Impact, Severity and other Firmware definitions, Please refer to the below 'Glossary of firmware terms' url:
http://www14.software.ibm.com/webapp/set2/sas/f/power5cm/home.html#termdefs

The complete Firmware Fix History for this Release Level can be reviewed at the following url:
http://download.boulder.ibm.com/ibmdl/pub/software/server/firmware/VL941-Firmware-Hist.html
VL941_045_035 / FW941.03

11/05/20		 Impact: Data       Severity:  HIPER

System firmware changes that affect all systems

  • HIPER/Pervasive:DEFERRED: A security problem was fixed for the case where "Speculative execution controls to mitigate user-to-kernel side channel attacks" was selected in ASMI, but the system was instead running with full speculative execution.  When this fix is applied concurrently, ASMI will prematurely show for this case (until a re-IPL is performed) that speculative execution controls are fully mitigated, when actually the system is still running with full speculative execution.  The system must be re-IPLed for this change to take effect such that the new security setting will be "Speculative execution controls to mitigate user-to-kernel and user-to-user side-channel attacks" to provide full speculative execution mitigation.
  • A problem was fixed for certain PCIe adapters and NVMe U.2 devices. The following feature codes are affected and the presence of these features in the 9009-22G, 9009-41G, 9009-42G, and 5105-22E servers can result in higher fan speeds and therefore higher acoustic levels after upgrading the firmware.  Refer to the acoustic levels published in the IBM Knowledge Center currently located at ===> https://www.ibm.com/support/knowledgecenter/9009-41A/p9had/p9had_90x.htm:
    #EC5J witn CCIN 59B4;  #EC5K with CCIN 59B5; #EC5L with CCIN 59B6; #EC5X with CCIN 59B7; #EC7A/#EC7B with CCIN  594A; #EC7C/#EC7D with CCIN 594B; #EC7E/#EC7F with CCIN 594C; #EC7J/#EC7K  with CCIN  594A; #EC7L/#EC7M  with CCIN 594B; #EC7N/#EC7P with CCIN 594C; #ES1E/#ES1F with CCIN 59B8; #ES1G/#ES1H with CCIN 59B9;  #EC5V/#EC5W with CCIN 59BA; #EC5G/#EC5B and #EC6U/#EC6V with CCIN 58FC;  #EC5W/#EC5D and #EC6W/#EC6X with CCIN 58FD; and #EC5G/#EC5B and #EC6Y/#EC6Z with CCIN 58FE.

System firmware changes that affect certain systems

  • HIPER/Pervasive:  A problem was fixed for certain SR-IOV adapters for a condition that may result from frequent resets of adapter Virtual Functions (VFs), or transmission stalls and could lead to potential undetected data corruption.
    The following additional fixes are also included:
    1) The VNIC backing device goes to a powered off state during a VNIC failover or Live Partition Mobility (LPM) migration.  This failure is intermittent and very infrequent.
    2) Adapter time-outs with SRC B400FF01 or B400FF02 logged.
    3) Adapter time-outs related to adapter commands becoming blocked with SRC B400FF01 or B400FF02 logged.
    4) VF function resets occasionally not completing quickly enough resulting in SRC B400FF02 logged.
    This fix updates the adapter firmware to 11.4.415.33 for the following Feature Codes and CCINs:  #EN15/#EN16 with CCIN 2CE3, #EN17/#EN18 with CCIN 2CE4, #EN0H/#EN0J with CCIN 2B93, #EN0M/#EN0N with CCIN 2CC0, #EN0K/#EN0L with CCIN 2CC1, #EL56/#EL38 with CCIN 2B93, and #EL57/#EL3C with CCIN 2CC1.
    The SR-IOV adapter firmware level update for the shared-mode adapters happens under user control to prevent unexpected temporary outages on the adapters.  A system reboot will update all SR-IOV shared-mode adapters with the new firmware level.  In addition, when an adapter is first set to SR-IOV shared mode, the adapter firmware is updated to the latest level available with the system firmware (and it is also updated automatically during maintenance operations, such as when the adapter is stopped or replaced).  And lastly, selective manual updates of the SR-IOV adapters can be performed using the Hardware Management Console (HMC).  To selectively update the adapter firmware, follow the steps given at the IBM Knowledge Center for using HMC to make the updates:  https://www.ibm.com/support/knowledgecenter/en/POWER9/p9efd/p9efd_updating_sriov_firmware.htm.
    Note: Adapters that are capable of running in SR-IOV mode, but are currently running in dedicated mode and assigned to a partition, can be updated concurrently either by the OS that owns the adapter or the managing HMC (if OS is AIX or VIOS and RMC is running).
    This pertains to 9009-xxG models only.
VL941_039_035 / FW941.01

10/02/20		 Impact: Availability       Severity:  SPE

System firmware changes that affect certain systems

  • DEFERRED:  On systems with an IBM i partition, a problem was fixed for the QPRCFEAT IBM System Value showing the 9009-xxG models as 9009-xxA models.  For example, the 9009-22G reports as "EP11" instead of "EP51".  This mismatch can prevent 3rd party software licenses from working.
    The system has to be re-IPLed for the fix to take effect.  This pertains to 9009-xxG models only.
    For IBM i stand-alone systems (no HMC), please contact IBM i Software Support (http://www.ibm.com/mysupport) for help installing this service pack as there is no officially released IBM i  PTF available for it.
VL941_035_035 / FW941.00

07/27/20
 Impact: New        Severity:  New

GA Level with key features included are listed below with new field defects since FW940.10.
All features and fixes from the FW940.10. service pack are included in this release but are not shown.

New features and functions

  • OPAL is supported with skiboot level v6.6-rc1 and petitboot level v1.12. This pertains to model ESS 5105-22E only.
  • Support for NVDIMMs for Linux OS RHEL 8.1 and later with feature code #EM71. This pertains to model ESS 5105-22E only.
  • Support for NVDIMM Backup Power Module (BPM) firmware level v1.07 (0x0107). This pertains to model ESS 5105-22E only.
  • Support for NVDIMM controller firmware level v3.B (0x3B). This pertains to model ESS 5105-22E only.
  • This server firmware level includes the SR-IOV adapter firmware level 11.4.415.28 for the following Feature Codes and CCINs: #EN15/#EN16 with CCIN 2CE3; #EN0H/#EN0J with CCIN 2B93; and #EN0K/#EN0L with CCIN 2CC1. This pertains to 9009-xxG models only.
  • This server firmware includes the SR-IOV adapter firmware level 1x.26.6000 for the following Feature Codes and CCINs: #EC2R/EC2S with CCIN 58FA;  #EC2T/EC2U with CCIN 58FB;  #EC3L/EC3M with CCIN 2CEC; and #EC66/EC67 with CCIN 2CF3. This pertains to 9009-xxG models only.
  • Support for the 6.4TB SSD PCIe4 NVMe U.2 module for AIX/Linux and IBM i with feature codes #EC5V/#EC5W and CCIN 59BA. Feature #EC5V indicates usage by AIX, Linux, or VIOS in which the SSD is formatted in 4096 byte sectors. Feature #EC5W indicates usage by IBM i in which the SSD is formatted in 4160 byte sectors. This pertains to 9009-xxG models only.
  • Support for the 800 GB SSD PCIe3 NVMe U.2 module for AIX/Linux with feature codes #EC5X and CCIN 59B7. Feature #EC5X indicates usage by AIX, Linux, or VIOS in which the SSD is formatted in 4096 byte sectors. This pertains to 9009-xxG models only.
  • Support for the 1.6 TB SSD PCIe4 NVMe U.2 module for AIX/Linux and IBM i with feature codes #ES1E/#ES1F and CCIN 59B8. Feature #ES1E indicates usage by AIX, Linux, or VIOS in which the SSD is formatted in 4096 byte sectors. Feature #ES1F indicates usage by IBM i in which the SSD is formatted in 4160 byte sectors. This pertains to 9009-xxG models only.
  • Support for the 3.2 TB SSD PCIe4 NVMe U.2 module for AIX/Linux and IBM i with feature codes #ES1G/#ES1H and CCIN 59B9. Feature #ES1G indicates usage by AIX, Linux, or VIOS in which the SSD is formatted in 4096 byte sectors. Feature #ES1H indicates usage by IBM i in which the SSD is formatted in 4160 byte sectors. This pertains to 9009-xxG models only.

System firmware changes that affect all systems

  • A problem was fixed for the REST/Redfish interface to change the success return code for object creation from "200" to "201". The "200" status code means that the request was received and understood and is being processed. A "201" status code indicates that a request was successful and, as a result, a resource has been created. The Redfish Ruby Client, "redfish_client" may fail a transaction if a "200" status code is returned when "201" is expected.
  • A problem was fixed to allow quicker recovery of PCIe links for the #EMXO PCIe expansion drawer for a run time fault with B7006A22 logged.  The time for recovery attempts can exceed six minutes on rare occasions which may cause I/O adapter failures and failed nodes. With the fix, the PCIe links will recover or fail faster (in the order of seconds) so that redundancy in a cluster configuration can be used with failure detection and failover processing by other hosts, if available, in the case where the PCIe links fail to recover. This pertains to 9009-xxG models only.
  • A problem was fixed for a concurrent maintenance "Repair and Verify" (R&V) operation for a #EMX0 fanout module that fails with an "Unable to isolate the resource" error message. This should occur only infrequently for cases where a physical hardware failure has occurred which prevents access to slot power controls. This problem can be worked around by bringing up the "PCIe Hardware Topology" screen from either ASMI or the HMC after the hardware failure but before the concurrent repair is attempted. This will avoid the problem with the PCIe slot isolation. These steps can also be used to recover from the error to allow the R&V repair to be attempted again. This pertains to 9009-xxG models only.
  • A problem was fixed for utilization statistics for commands such as HMC lslparutil and third-party lpar2rrd that do not accurately represent CPU utilization. The values are incorrect every time for a partition that is migrated with Live Partition Mobility (LPM).  Power Enterprise Pools 2.0 is not affected by this problem. If this problem has occurred, here are three possible recovery options:
    1) Re-IPL the target system of the migration.
    2) Or delete and recreate the partition on the target system.
    3) Or perform an inactive migration of the partition.  The cycle values get zeroed in this case.
    This pertains to 9009-xxG models only.

4.0 How to Determine The Currently Installed Firmware Level

You can view the server's current firmware level on the Advanced System Management Interface (ASMI) Welcome pane. It appears in the top right corner. Example: VL910_123.

5.0 Downloading the Firmware Package

Follow the instructions on Fix Central. You must read and agree to the license agreement to obtain the firmware packages.

Note: If your HMC is not internet-connected you will need to download the new firmware level to a USB flash memory device or ftp server.

6.0 Installing the Firmware

The method used to install new firmware will depend on the release level of firmware which is currently installed on your server. The release level can be determined by the prefix of the new firmware's filename.

Example: VLxxx_yyy_zzz

Where xxx = release level

HMC Managed Systems:

Instructions for installing firmware updates and upgrades on systems managed by an HMC can be found at:
https://www.ibm.com/support/knowledgecenter/9009-22A/p9eh6/p9eh6_updates_sys.htm


NovaLink Managed Systems:

A NovaLink managed system does not have a HMC attached and is managed either by PowerVM Novalink or PowerVC using PowerVM Novalink.
Instructions for installing firmware updates and upgrades on systems managed by PowerVM NovaLink can be found at:
https://www.ibm.com/support/knowledgecenter/9009-22A/p9eig/p9eig_updating_firmware.htm


HMC and NovaLink Co-Managed Systems  (Disruptive firmware updates only):

 A co-managed system is managed by HMC and NovaLink, with one of the interfaces in the co-management master mode.
Instructions for installing firmware updates and upgrades on systems co-managed by an HMC and Novalink is the same as above for a HMC managed systems since the firmware update must be done by the HMC in the co-management master mode.  Before the firmware update is attempted, one must be sure that HMC is set in the master mode using the steps at the following IBM KnowledgeCenter link for NovaLink co-managed systems:
https://www.ibm.com/support/knowledgecenter/9009-22A/p9eig/p9eig_kickoff.htm

Then the firmware updates can proceed with the same steps as for the HMC managed systems except the system must be powered off because only a disruptive update is allowed.   If a concurrent update is attempted, the following error will occur: " HSCF0180E Operation failed for <system name> (<system mtms>).  The operation failed.  E302F861 is the error code:
https://www.ibm.com/support/knowledgecenter/9009-22A/p9eh6/p9eh6_updates_sys.htm


Systems not Managed by an HMC or NovaLink:

Power Systems:

Instructions for installing firmware on systems that are not managed by an HMC can be found at:
https://www.ibm.com/support/knowledgecenter/9009-22A/p9ha5/fix_serv_firm_kick.htm


Systems running Ubuntu operating system:

If Ubuntu will be used to update the system firmware, please follow these instructions to extract the installable binary and update/upgrade the firmware:

1) Download the .gz (tarball) from Fix Central to your Ubuntu system (ie, to /tmp/fwupdate).

2) Extract the .gz file to /tmp/ on the Ubuntu system:

Example:
    tar -xzf /tmp/fwupdate/01VL910_040_040.tar.gz -C /tmp/fwupdate

3) Use update_flash -v -f <extracted file name> to verify the package.

4) Update your firmware using update_flash:

/usr/sbin/update_flash  -f <extracted file name>

System will reboot during the firmware update. When the system reaches Ubuntu run-time state, you can then commit or reject the firmware update:
Commit: /usr/sbin/update_flash -c
Reject:   /usr/sbin/update_flash -r

IBM i Systems:

For information concerning IBM i Systems, go to the following URL to access Fix Central: 
http://www-933.ibm.com/support/fixcentral/

Choose "Select product", under Product Group specify "System i", under Product specify "IBM i", then Continue and specify the desired firmware PTF accordingly.

7.0 Firmware History

The complete Firmware Fix History (including HIPER descriptions)  for this Release level can be reviewed at the following url:
http://download.boulder.ibm.com/ibmdl/pub/software/server/firmware/VL941-Firmware-Hist.html