Power10 System Firmware

Applies to:   9043-MRX

This document provides information about the installation of Licensed Machine or Licensed Internal Code, which is sometimes referred to generically as microcode or firmware.

Contents

1.0 Systems Affected

This package provides firmware for IBM Power System E1050 (9043-MRX) server only.

The firmware level in this package is:

1.1 Minimum HMC Code Level

This section is intended to describe the "Minimum HMC Code Level" required by the System Firmware to complete the firmware installation process. When installing the System Firmware, the HMC level must be equal to or higher than the "Minimum HMC Code Level" before starting the system firmware update.  If the HMC managing the server targeted for the System Firmware update is running a code level lower than the "Minimum HMC Code Level" the firmware update will not proceed.

The Minimum HMC Code levels for this firmware for HMC x86,  ppc64 or ppc64le are listed below.

x86 -  This term is used to reference the legacy HMC that runs on x86/Intel/AMD hardware for the Virtual HMC that can run on the Intel hypervisors (KVM, XEN, VMWare ESXi).
ppc64 or ppc64le - describes the Linux code that is compiled to run on Power-based servers or LPARS (Logical Partitions)
The Minimum HMC level supports the following HMC models:
HMC models: 7063-CR1 and 7063-CR2
x86 - KVM, XEN, VMWare ESXi (6.0/6.5)
ppc64le - vHMC on PowerVM (POWER8,POWER9, and POWER10 systems)

For information concerning HMC releases and the latest PTFs,  go to the following URL to access Fix Central:
https://www.ibm.com/support/fixcentral/

For specific fix level information on key components of IBM Power Systems running the AIX, IBM i and Linux operating systems, we suggest using the Fix Level Recommendation Tool (FLRT):
https://esupport.ibm.com/customercare/flrt/home


NOTES:
                -You must be logged in as hscroot in order for the firmware installation to complete correctly.
                - Systems Director Management Console (SDMC) does not support this System Firmware level.

2.0 Important Information

FW1030 needs to be installed twice when upgrading from FW1020 firmware levels
When installing the FW1030 firmware to a system that previously was at a FW1020 level,  you need to complete the upgrade then perform the upgrade again consecutively. This to ensure both the T (temporary, also known as current) and P (permanent, also known as backup) are equal. This was previously known as the "Accept" process that was performed automatically. This will be addressed in a newer release/PTF of the HMC to automatically do the accept.
Improvements in FW1030.00 and higher to the eBMC will cause any IPL from a backup side that is at a FW1020 firmware level to fail. An AC power cycle would be required to recover from this condition.

2.2 Concurrent Firmware Updates

Concurrent system firmware update is supported on HMC Managed Systems only.

Ensure that there are no RMC connections issues for any system partitions prior to applying the firmware update.  If there is a RMC connection failure to a partition during the firmware update, the RMC connection will need to be restored and additional recovery actions for that partition will be required to complete partition firmware updates.

2.3 Memory Considerations for Firmware Upgrades

Firmware Release Level upgrades and Service Pack updates may consume additional system memory.
Server firmware requires memory to support the logical partitions on the server. The amount of memory required by the server firmware varies according to several factors.
Factors influencing server firmware memory requirements include the following:
Generally, you can estimate the amount of memory required by server firmware to be approximately 8% of the system installed memory. The actual amount required will generally be less than 8%. However, there are some server models that require an absolute minimum amount of memory for server firmware, regardless of the previously mentioned considerations.

Additional information can be found at:
https://www.ibm.com/docs/en/power10/9043-MRX?topic=resources-memory

2.4 SBE Updates

Power10 servers contain SBEs (Self Boot Engines) and are used to boot the system.  SBE is internal to each of the Power10 chips and used to "self boot" the chip.  The SBE image is persistent and is only reloaded if there is a system firmware update that contains a SBE change.  If there is a SBE change and system firmware update is concurrent, then the SBE update is delayed to the next IPL of the CEC which will cause an additional 3-5 minutes per processor chip in the system to be added on to the IPL.  If there is a SBE change and the system firmware update is disruptive, then SBE update will cause an additional 3-5 minutes per processor chip in the system to be added on to the IPL.  During the SBE update process, the HMC or op-panel will display service processor code C1C3C213 for each of the SBEs being updated.  This is a normal progress code and system boot should be not be terminated by the user. Additional time estimate can be between 12-20 minutes per drawer or up to 48-80 minutes for maximum configuration.

The SBE image is updated with this service pack.

3.0 Firmware Information

Use the following examples as a reference to determine whether your installation will be concurrent or disruptive.

For systems that are not managed by an HMC, the installation of system firmware is always disruptive.

Note: The concurrent levels of system firmware may, on occasion, contain fixes that are known as Deferred and/or Partition-Deferred. Deferred fixes can be installed concurrently, but will not be activated until the next IPL. Partition-Deferred fixes can be installed concurrently, but will not be activated until a partition reactivate is performed. Deferred and/or Partition-Deferred fixes, if any, will be identified in the "Firmware Update Descriptions" table of this document. For these types of fixes (Deferred and/or Partition-Deferred) within a service pack, only the fixes in the service pack which cannot be concurrently activated are deferred.

Note: The file names and service pack levels used in the following examples are for clarification only, and are not necessarily levels that have been, or will be released.

System firmware file naming convention:

01VHxxx_yyy_zzz

NOTE: Values of service pack and last disruptive service pack level (yyy and zzz) are only unique within a release level (xxx). For example, 01MM1010_040_040 and 01MM1010_040_040 are different service packs.

An installation is disruptive if:

            Example: Currently installed release is 01VH900_040_040, new release is 01VH910_050_050.

            Example: VH910_040_040 is disruptive, no matter what level of VH910 is currently installed on the system.

            Example: Currently installed service pack is VH910_040_040 and new service pack is VH910_050_045.

An installation is concurrent if:

The release level (xxx) is the same, and
The service pack level (yyy) currently installed on the system is the same or higher than the last disruptive service pack level (zzz) of the service pack to be installed.

Example: Currently installed service pack is VH910_040_040, new service pack is VH910_041_040.

3.1 Firmware Information and Description

 
Filename Size Checksum md5sum
01MM1030_030_026.img 287854784
 19872
 a4735639e21444e9e6c0a5658a86a4a8 
01MM1030_030_026.tar
 135557120
 43806
 411da1b123cb55b6bfff19a0f8de14ac

Note: The Checksum can be found by running the AIX sum command against the rpm file (only the first 5 digits are listed).
ie: sum     01MM1030_030_026.img

MM1030
For Impact, Severity and other Firmware definitions, Please refer to the below 'Glossary of firmware terms' url:
https://www.ibm.com/support/pages/node/6555136

The complete Firmware Fix History for this Release Level can be reviewed at the following url:
https://public.dhe.ibm.com/software/server/firmware/MM-Firmware-Hist.html
MM1030_030_026 / FW1030.01

12/22/22		 Impact: Availability  Severity:  SPE

System firmware changes that affect all systems
  • A problem was fixed for an error that happens on servers upgrading to or running FW1030.00. Failure symptoms may include any of the following:
    1) The eBMC does not power on the system (instead it is in a quiesced state).
    2) The Operations Panel is a scrolling ball.
    3)  Error log has SRC BD8D3404.
    4) An HMC-managed system will show 'no connection' and possibly Incomplete State on the HMC.
    If this problem is active on the system, the eBMC ASMI can be used to install the fix by updating to the FW1030.01 or later level.
    If your FW1030.00 system is not yet having the problem, an update to the FW1030.01 or later level is still strongly recommended as soon as possible to prevent the problem from occurring.
MM1030_026_026 / FW1030.00

12/09/22		 Impact: New  Severity:  New

GA Level with key features listed below along with security fixes.

New Features and Functions
  • This server firmware includes the SR-IOV adapter firmware level xx.34.1002 for the following Feature Codes and CCINs: #EC2R/EC2S with CCIN 58FA; #EC2T/EC2U with CCIN 58FB; and #EC66/EC67 with CCIN 2CF3.  
    Update instructions:  https://www.ibm.com/docs/en/power10?topic=updates-sr-iov-firmware-update
  • Support was added for Secure Boot for SUSE Linux Enterprise Server (SLES) partitions.  The SUSE Linux level must be SLES 15 SP4 or later.  Without this feature, partitions with SLES 15 SP4 or later and which have the OS Secure Boot partition property set to "Enabled and Enforced" will fail to boot.  A workaround to this is to change the partition's Secure Boot setting in the HMC partition configuration to "Disabled" or "Enabled and Log only".
  • HIPER/Pervasive: For systems with Power Linux partitions, support was added for a new Linux secure boot key.  The support for the new secure boot key for Linux partitions may cause secure boot for Linux to fail if the Linux OS for SUSE or RHEL distributions does not have a secure boot key update.  
    The affected Linux distributions are as follows that need the Linux fix level that includes "Key for secure boot signing grub2 builds ppc64le".
    1) SLES 15 SP4 - The GA for this Linux level includes the secure boot fix.
    2) RHEL 8.5- This Linux level has no fix.  The user must update to RHEL: 8.6 or RHEL 9.0.
    3) RHEL 8.6
    4) RHEL 9.0.  
    The update to a Linux level that supports the new secure boot key also addresses the following security issues in Linux GRUB2 and are the reasons that the change in secure boot key is needed as documented in the following six CVEs:
    1) CVE-2021-3695
    2) CVE-2022-28733
    3) CVE-2022-28734
    4) CVE-2022-28735
    5) CVE-2022-28736
    6) CVE-2022-28737
    Please note that when this firmware level of FW1030.00 is installed, any Linux OS not updated to a secure boot fix level will fail to secure boot.  And any Linux OS partition updated to a fix level for secure boot requires a minimum firmware level of FW1010.30 or later,  FW1020.00 or later, or FW1030.00 or later to be able to do a secure boot.  If lesser firmware levels are active but the Linux fix levels for secure boot are loaded for the Linux partition, the secure boot failure that occurs will have BA540010 logged.  If secure boot verification is enabled, but not enforced (log only mode), then the fixed Linux partition will boot, but a BA540020 informational error will be logged.
  • Support has been dropped for the smaller logical-memory block (LMB) sizes of 16MB, 32MB, and 64MB. 128MB and 256MB are the only LMB sizes that can be selected in the eBMC ASMI.
  • Password quality rules were enhanced on the eBMC for local passwords such that new passwords must have characters from at least two classes: lower-case letters, upper-case letters, digits, and other characters. With this enhancement, you can get a new error message from the `passwd` command:
    "BAD PASSWORD: The password contains less than 2 character classes".
  • Live Partition Mobility (LPM) support for partitions with vPMEM volumes assigned to them.  With this feature, the PowerVM hypervisor manages the migration of the data in the vPMEM volumes as part of its normal LPM operations.
  • Support added to display on the management console (HMC, NovaLink) the physical port MAC address of an SR-IOV shared mode enabled adapter's physical ports.  This allows for verification of an adapter's physical port connection to an external switch without physically tracing cables.
  • Support for concurrent maintenance for the system operator panel.
  • Support for redundant clocks on the system.
  • Advanced Memory Mirroring (AMM) support for the Virtualization Management Interface (VMI).  This feature adds AMM support for mirroring the memory used by VMI.
  • Support for Linux 2 MB I/O mappings (TCEs) for a PCIe slot enabled with Huge Dynamic DMA Window capability (HDDW) using the I/O Adapter Enlarged Capacity setting in ASMI.   This applies to both dedicated PCIe slots as well as SR-IOV virtual functions.
  • Support for PCIe3 4-port 10GbE BaseT RJ45 Adapter with Feature Code #EN2W with CCIN 2F04.   Feature #EN2W has a tailstock for full-height PCIe slots.
  • Support for enablement of the self-encrypting drive (SED) capability of NVMe drives on Power10 systems. This enables data-at-rest encryption on NVMe drives without additional impact to I/O performance or CPU utilization. IBM PowerVM Platform KeyStore (PKS) must be enabled for NVMe SED key management. The new AIX command line utility nvmesed is introduced to provide management of NVMe SED drives.  Booting from the NVMe SED-enabled drive is supported.
    Note: NVMe SED enablement requires a SED-capable NVMe drive and AIX 7,3 TL1 or later.
    Power firmware version FW1030.00 or later is required for this feature.
  • Improvements to Fibre Channel (FC) Non-Volatile Memory Express (FC-NVMe) capability to include N-port ID virtualization (NPIV) client support. This capability requires AIX 7.3 TL1 or later,  IBM PowerVM Virtual I/O Server (VIOS) 3.1.4, an NVMeoF NPIV-capable FC adapter that supports NVMeof; and an NVMeoF storage subsystem.  The FC adapters supported include the PCIe4  2-Port 64 Gb FC Adapter ( feature codes #EN1N and #EN1P); and the PCIe4 4-Port 32 Gb FC Adapter (feature codes #EN1L and #EN1M); or any  any high-bandwidth FC adapters that support NVMeoF protocol in the AIX physical stack.
    NVMe Over Fabric (SAN) Boot is supported.
    Note: Booting from FC-NVMe disk may fail if certain fabric errors are returned, hence a boot disk set up with multiple paths is recommended.  In case there is a failure to boot, the boot process may continue if you exit from the SMS menu. Another potential workaround is to discover boot LUNs from the SMS menu and then retry boot.
    Power firmware version FW1030.00 or later is required for this feature.
  • Support for a PowerVM Watchdog for AIX and Linux using a hypervisor call to set up a watchdog for kernel and userspace use.
  • Support for SR-IOV including NIC, RoCE, and vNIC for a PCIe4 2-port 100Gb No Cryptographic ConnectX-6 DX QFSP56 adapter with Feature Code #EC76 with CCIN 2CFB,  This PCIe Gen4 Ethernet x16 adapter provides two 100 GbE QFSP56 ports. The adapter is based on a Mellanox ConnectX-6 adapter, which uses a ConnectX-6 EN network controller.
    OS support is as follows:
    AIX 7.2 TL5 and later: Dedicated, SR-IOV NIC/RoCE, VIOS, and vNIC.
    Linux RHEL 8.4, RHEL 9, and SLES 15 SP3: Dedicated, SR-IOV NIC/RoCE, VIOS, and vNIC.
  • Support for a PCIe 4.0 8x 2-port 64 Gigabit optical fibre channel (FC) adapter with feature code #EN1N and CCIN 2CFD.  Support includes direct attach configurations.  Feature #EN1N has a tail stock for full height PCIe slots.  Firmware support is for all P10 and later levels.
    OS support  is as follows for AIX and Linux:
    AIX 7.2 TL5 and later.
    Linux RHEL 8 and SLES 15.
  • Support for a PCIe 4.0 8x 4-port 32 Gigabit optical fibre channel (FC) adapter with feature code #EN1L and CCIN 2CFC.  Support includes direct attach configurations.  Feature #EN1N has a tail stock for full height PCIe slots.  Firmware support is for all P10 and later levels.
    OS support is as follows for AIX and Linux:
    AIX 7.2 TL5 and later.
    Linux RHEL 8 and SLES 15.
System firmware changes that affect all systems
  • HIPER/Pervasive: The following problems were fixed for certain SR-IOV adapters in shared mode when the physical port is configured for Virtual Ethernet Port Aggregator (VEPA):
    1) A security problem for CVE-2022-34331 was addressed where switches configured to monitor network traffic for malicious activity are not effective because of errant adapter configuration changes.  The misconfigured adapter can cause network traffic to flow directly between the VFs and not out the physical port hence bypassing any possible monitoring that could be configured in the switch.
    2) Packets may not be forwarded after a firmware update, or after certain error scenarios which require an adapter reset. Users configuring or using VEPA mode should install this update. These fixes pertain to adapters with the following Feature Codes and CCINs:  #EC2R/EC2S with CCIN 58FA; #EC2T/EC2U with CCIN 58FB; and #EC66/EC67 with CCIN 2CF3.
    Update instructions:  https://www.ibm.com/docs/en/power10?topic=updates-sr-iov-firmware-update
  • Security problems were fixed for vTPM 1.2 by updating its OpenSSL library to version 0.9.8zh.  Security vulnerabilities CVE-2022-0778, CVE-2018-5407, CVE-2014-0076, and CVE-2009-3245 were addressed.  These problems only impact a partition if vTPM version 1.2 is enabled for the partition.
  • A security problem was fixed for vTPM 2.0 by updating its libtpms library.  Security vulnerability CVE-2021-3746 was addressed.  This problem only impacts a partition if vTPM version 2.0 is enabled for the partition.  The biggest threat from this vulnerability is system availability.
  • A security problem was fixed for the Virtualization Management Interface (VMI) for vulnerability CVE-2021-45486 that could allow a remote attacker to reveal sensitive information.  This can happen for session connections using IPv4.
  • A security problem was fixed for the eBMC for vulnerability CVE-2022-3435 that could allow a remote attacker to reveal sensitive information from the eBMC.  This can happen for session connections using IPv4.
  • A security problem was fixed for the eBMC HTTPS server where a specially crafted multi-part HTTPS header, on a specific URI only available to admin users, could cause a buffer overflow and lead to a denial of service for the eBMC.  This Common Vulnerabilities and Exposures issue number is CVE-2022-2809.
  • A security problem was fixed for a flaw in OpenSSL certificate parsing that could result in an infinite loop in the hypervisor, causing a hang in a Live Partition Mobility (LPM) target partition.   The trigger for this failure is an LPM migration of a partition with a corrupted physical trusted platform module (pTPM) certificate. This is expected to be a rare problem.  The Common Vulnerability and Exposure number for this problem is CVE-2022-0778.
  • A problem was fixed where the eBMC ASMI user was not informed that changing settings to enable or disable the eBMC's SSH or IPMI service will take about 15 seconds to take effect, after successfully changing the setting.  The operation for changing the setting does not take effect immediately.  With the fix, the eBMC ASMI user is given a message about this delay when performing the operation.

4.0 How to Determine The Currently Installed Firmware Level

You can view the server's current firmware level on the Advanced System Management Interface (ASMI) Overview page under the System Information section in the Firmware Information panel. Example: (MM1020_079)

5.0 Downloading the Firmware Package

Follow the instructions on Fix Central. You must read and agree to the license agreement to obtain the firmware packages.

Note: If your HMC is not internet-connected you will need to download the new firmware level to a USB flash memory device or ftp server.

6.0 Installing the Firmware

The method used to install new firmware will depend on the release level of firmware which is currently installed on your server. The release level can be determined by the prefix of the new firmware's filename.

Example: MHxxx_yyy_zzz

Where xxx = release level

Instructions for installing firmware updates and upgrades can be found at https://www.ibm.com/docs/en/power10/9043-MRX?topic=9043-MRX/p10eh6/p10eh6_updates_sys.htm

IBM i Systems:

For information concerning IBM i Systems, go to the following URL to access Fix Central: 
https://www.ibm.com/support/fixcentral/

Choose "Select product", under Product Group specify "System i", under Product specify "IBM i", then Continue and specify the desired firmware PTF accordingly.

HMC and NovaLink Co-Managed Systems (Disruptive firmware updates only):

A co-managed system is managed by HMC and NovaLink, with one of the interfaces in the co-management master mode.
Instructions for installing firmware updates and upgrades on systems co-managed by an HMC and Novalink is the same as above for a HMC managed systems since the firmware update must be done by the HMC in the co-management master mode.  Before the firmware update is attempted, one must be sure that HMC is set in the master mode using the steps at the following IBM KnowledgeCenter link for NovaLink co-managed systems:
https://www.ibm.com/docs/en/power10/9043-MRX?topic=environment-powervm-novalink

Then the firmware updates can proceed with the same steps as for the HMC managed systems except the system must be powered off because only a disruptive update is allowed.   If a concurrent update is attempted, the following error will occur: " HSCF0180E Operation failed for <system name> (<system mtms>).  The operation failed.  E302F861 is the error code:"
https://www.ibm.com/docs/en/power10/9043-MRX?topic=9043-MRX/p10eh6/p10eh6_updates_sys.htm

7.0 Firmware History

The complete Firmware Fix History (including HIPER descriptions)  for this Release level can be reviewed at the following url:
https://public.dhe.ibm.com/software/server/firmware/MM-Firmware-Hist.html

8.0 Change History

Date
 Description
January 20, 2023
 Updated readme description:
 A problem was fixed for an error that happens on servers upgrading to or running FW1030.00. Failure symptoms may include any of the following:
1) The eBMC does not power on the system (instead it is in a quiesced state).
2) The Operations Panel is a scrolling ball.
3)  Error log has SRC BD8D3404.
4) An HMC-managed system will show 'no connection' and possibly Incomplete State on the HMC.
If this problem is active on the system, the eBMC ASMI can be used to install the fix by updating to the FW1030.01 or later level.
If your FW1030.00 system is not yet having the problem, an update to the FW1030.01 or later level is still strongly recommended as soon as possible to prevent the problem from occurring.