New features and functions

• Support was enabled for eRepair spare lane deployment for fabric and memory buses.
• Support was added for doing soft post package memory row repair (sPPR) on the DDR4 DIMMs during the system IPL. The sPPR feature saves on the use of ECC spares for memory recovery, reducing the number of DIMMs that have to be guarded for memory errors.
• Support was added for Multi-Function clock card failover.
  

System firmware changes that affect all systems

• HIPER/Non-Pervasive: DISRUPTIVE:  Fixes included to address potential scenarios that could result in undetected data corruption, system hangs, or system terminations.
  
• DISRUPTIVE:  A problem was fixed for PCIe and SAS adapters in slots attached to a PLX (PCIe switch) failing to initialize and not being found by the Operating System.  The problem should not occur on the first IPL after an AC power cycle, but subsequent IPLs may experience the problem.
• DEFERRED:  A problem was fixed for a PCIe clock failure in the  PCIe3 I/O expansion drawer (feature #EMX0), causing loss of PCIe slots.   The system must be re-IPLed for the fix to activate.
• DEFERRED:  A problem was fixed for a possible system hang in the early boot stage.  This  could occur during periods of very high activity for memory read operations which deplete all read buffers, hanging an internal process that requires a read buffer,  With the fix, a congested memory controller can stall the read pipeline to make a read buffer available for the internal processes.
• DEFERRED:  A problem was fixed for concurrent maintenance operations for PCIe expansion drawer cable cards and PCI adapters that could cause  loss of system hardware information in the hypervisor with these side effects:  1) partition secure boots could fail with SRC BA540100 logged.; 2) Live Partition Mobility (LPM) migrations could be blocked; 3) SR-IOV adapters could be blocked from going into shared mode; 4) Power Management services could be lost; and 5) warm re-IPLs of the system can fail.  The system can be recovered by powering off and then IPLing again.
• DEFERRED:  A problem was fixed for a transient VRM over-current condition for loads on the USB bus that could fail an IPL with SRC 11002700 00002708 logged.  The frequency of the failure is about 1 in every 5 IPL attempts.  The system can be recovered by doing another IPL.
  
• A problem was fixed for an unhelpful error message of "HSCL1473 Cannot execute atomic operation. Atomic operations are not enabled." that is displayed on the HMC if there are no licensed processors available for the boot of a partition.
• A problem was fixed for a memory channel failure due to a RCD parity error calling out the affected DIMMs correctly, but also falsely calling out either the memory controller or a processor, or both. 
  
• A problem was fixed for adapters in slots attached to a PLX (PCIe switch) failing with SRCs B7006970 and BA188002  when a second and subsequent errors on the PLX failed to initiate PLX recovery.  For this infrequent problem to occur, it requires a second error on the PLX after recovery from the first error.
• A problem was fixed for the system going into Safe Mode after a run-time deconfiguration of a processor core,  resulting in slower performance.  For this problem to occur, there must be a second fault  in the Power Management complex after the processor core has been deconfigured.
• A problem was  fixed for service processor resets confusing the wakeup state of  processor cores, resulting in degraded cores that cannot be managed for power usage.  This will result in the system consuming more power, but also running slower due to the inability to make use of WOF optimizations around the cores.  The degraded  processor cores can be recovered by a re-IPL of the system.
• A problem was fixed for the On-Chip Controller (OCC) MAX memory bandwidth sensor sometimes having values that are too high.
• A problem was fixed for DDR4 memory training in the IPL to improve the DDR4 write margin.  Lesser write margins can potentially cause memory errors.
• A problem was fixed for a system failure with SRC B700F103 that can occur if a shared-mode SR-IOV adapter is moved from a high-performance slot to a lower performance slot.   This problem can be avoided by disabling shared mode on the SR-IOV adapter; moving the adapter;  and then re-enabling shared mode.
• A problem was fixed for the system going to Safe Mode if all the cores of a processor are lost at run-time.
• A problem was fixed for a Core Management Engine (CME) fault causing a system failure with SRC B700F105 if processor cores had been guarded during the IPL.
  
• A problem was fixed for a Core Management Engine (CME) fault that could result in a system checkstop.
  
• A problem was fixed for a missing error log for the case of the TPM card not being detected when it is required for a trusted boot.
• A problem was fixed for a flood of BC130311 SRCs that could occur when changing Energy Scale Power settings, if the Power Management is in a reset loop because of errors.
• A problem was fixed for coherent accelerator processor proxy (CAPP) unit  errors being called out as CEC hardware Subsystem instead of  PROCESSOR_UNIT.
• A problem was fixed for an incorrect processor callout on a memory channel error that causes a CHIFIR[61] checkstop on the processor.
• A problem was fixed for a Logical LAN (l-lan) device failing to boot when there is a UDP packet checksum error.  With the fix, there is a new option when configuring a l-lan port in SMS to enable or disable the UDP checksum validation.  If the adapter is already providing the checksum validation, then the l-lan port needs to have its validation disabled.
• A problem was fixed for missing error logs for hardware faults if the hypervisor terminates before the faults can be processed.  With the fix, the hardware attentions for the bad FRUs will get handled, prior to processing the termination of the hypervisor.
  
• A problem was fixed for the diagnostics for a system boot checkstop failing to isolate to the bad FRU if it  occurred on a non-master processor or a memory chip connected to a non-master processor.  With the fix, the  fault attentions from a non-master processor are properly isolated to the failing chip so it can be guarded or recovered as needed to allow the IPL to continue.
• A problem was fixed for Hostboot error log IDs (EID) getting reused from one IPL to the next, resulting in error logs getting suppressed (missing)  for new problems on the subsequent IPLs if they have a re-used EID that was already present in the service processor error logs.
• A problem was fixed for Live Partition Mobility (LPM) partition migration to preserve the Secure Boot setting on the target partition.  Secure Boot is supported in FW920 and later partitions.  If the Secure Boot setting is non-zero for the partition, it will zero after the migration.
  
• A problem was fixed for an SR-IOV adapter using the wrong Port VLAN ID (PVID) for a logical port  (VF) when its non-zero PVID could be changed following a network install using the logical port.
  This fix updates adapter firmware to 11.2.211.32  for the following Feature Codes: EN15,  EN17, EN0H, EN0J, EN0M, EN0N, EN0K, EN0L, EL38, EL3C, EL56, and EL57.
  The SR-IOV adapter firmware level update for the shared-mode adapters happens under user control to prevent unexpected temporary outages on the adapters.  A system reboot will update all SR-IOV shared-mode adapters with the new firmware level.  In addition, when an adapter is first set to SR-IOV shared mode, the adapter firmware is updated to the latest level available with the system firmware (and it is also updated automatically during maintenance operations, such as when the adapter is stopped or replaced).  And lastly, selective manual updates of the SR-IOV adapters can be performed using the Hardware Management Console (HMC).  To selectively update the adapter firmware, follow the steps given at the IBM Knowledge Center for using HMC to make the updates:   https://www.ibm.com/support/knowledgecenter/en/POWER9/p9efd/p9efd_updating_sriov_firmware.htm.
  Note: Adapters that are capable of running in SR-IOV mode, but are currently running in dedicated mode and assigned to a partition, can be updated concurrently either by the OS that owns the adapter or the managing HMC (if OS is AIX or VIOS and RMC is running).
  
• A problem was fixed for a SMS ping failure for a SR-IOV adapter VF with a non-zero Port VLAN ID (PVID).  This failure may occur after the partition with the adapter has been booted to AIX, and then rebooted back to SMS.  Without the fix, residue information from the AIX boot is retained for the VF that should have been cleared.
• A problem was fixed for a SR-IOV adapter vNIC configuration error that did not provide a proper SRC to help resolve the issue of the boot device not pinging in SMS due to maximum transmission unit (MTU) size mismatch in the configuration.  The use of a vNIC backing device does not allow configuring VFs for jumbo frames when the Partition Firmware configuration for the adapter (as specified on the HMC) does not support jumbo frames.  When this happens, the vNIC adapter will fail to ping in SMS and thus cannot be used as a boot device.  With the fix,  the vNIC driver configuration code is now checking the vNIC login (open) return code so it can issue an SRC when the open fails for a MTU  issue (such as  jumbo frame mismatch) or for some other reason.  A jumbo frame is an Ethernet frame with a payload greater than the standard MTU of 1,500 bytes and can be as large as 9,000 bytes.
• A problem was fixed for three bad lanes causing a memory channel  fail on the DMI interface.  With the fix, the errors on the third lane on the DMI interface will  be recovered and it will continue to be used as long as it functions.
• A problem was fixed for preventing loss of function on an SR-IOV adapter with an 8MB adapter firmware image if it is placed into SR-IOV shared mode.  The 8MB image is not supported at the FW920.20 firmware level.  With the fix, the adapter with the 8MB image is rejected with an error without an attempt to load the older 4MB image on the adapter which could damage it.  This problem affects the following  SR-IOV adapters:  #EC2R/#EC2S with CCIN 58FA; and  #EC2T/#EC2U with CCIN 58FB.
  
• A problem was fixed for incorrect recovery from a service processor mailbox error that was causing the system IPL to fail with the loss of all the PCIe links.  If this occurs, the system will normally re-IPL successfully. 
  
• A problem was fixed for SR-IOV adapter failures when running in shared mode in a Huge Dynamic DMA Window (HDDW) slot.  I/O slots are enabled with HDDW by using the I/O Adapter Enlarged Capacity setting in the Advanced System Management Interface (ASMI).   This problem can be circumvented by moving the SR-IOV adapter to a non-HDDW slot, or alternatively, disabling HDDW on the system.
• A problem was fixed for system termination for a re-IPL with power on with SRC B181E540 logged.  The system can be recovered by powering off and then IPLing.  This problem occurs infrequently and can be avoided by powering off the system between IPLs.
  

System firmware changes that affect certain systems

• For a shared memory partition,  a problem was fixed for Live Partition Mobility (LPM) migration hang after a Mover Service Partition (MSP) failover in the early part of the migration.  To recover from the hang, a migration stop command must be given on the HMC.  Then the migration can be retried.
• For a shared memory partition,  a problem was fixed for Live Partition Mobility (LPM) migration failure to an indeterminate state.  This can occur if the Mover Service Partition (MSP) has a failover that occurs when the migrating partition is in the state of "Suspended."  To recover from this problem, the partition must be shutdown and restarted.
• On a system with a Cloud Management Console and a HMC Cloud Connector, a problem was fixed for memory leaks in the Redfish server causing Out of Memory (OOM) resets of the service processor.
• On a system with a partition with dedicated processors that are set to allow processor sharing with "Allow when partition is active" or "Allow always", a problem was fixed for a potential system hang if the partition is booting or shutting down while Dynamic Platform Optimizer (DPO) is running.  As a work-around to the problem, the processor sharing can be turned off before running DPO, or avoid starting or shutting down dedicated partitions with processor sharing while DPO is active.
• On a system with an AMS partition, a problem was fixed for a Live Partition Mobility (LPM) migration failure when migrating from P9 to a pre-FW860 P8 or P7 system.  This failure can occur if the P9 partition is in dedicated memory mode, and the Physical Page Table (PPT) ratio is explicitly set on the HMC (rather than keeping the default value) and the partition is then transitioned to Active Memory Sharing (AMS) mode prior to the migration to the older system.  This problem can be avoided by using dedicated memory in the partition being migrated back to the older system.
  

New features and functions

• DISRUPTIVE:  Support was added for installing and running mixed levels of P9 processors on the system in compatibility mode.
  
• Support added for PCIe4 2-port 100Gb ROCE RN adapter with feature code #EC66 for AIX and IBM i.  This PCIe Gen4 Ethernet x16 adapter provides two 100 GbE QSFP28 ports.
• Support was added to enable mirrored Hostboot memory.
  

System firmware changes that affect all systems

• HIPER/Non-Pervasive:   A problem was fixed for a potential problem that could result in undetected data corruption.
• DEFERRED:   A problem was fixed for the Input Offset Voltage (VIO) to the processor being set too low, having less margin for PCIe and XBUS errors that could cause a higher than normal rate of processor or PCIe device failures during the IPL or at run time.
• A problem was fixed for truncated firmware assisted dumps (fadump/kdump).  This can happen when the dumps are configured with chunks > 1Gb.
• A problem was fixed for the default gateway in the Advanced System Management Interface (ASMI) IPv4 network configurations showing as 0.0.0.0 which is an invalid gateway IP address.  This problem can occur if ASMI is used to clear the gateway value with blanks.  
  
• A problem was fixed for the Advanced System Management Interface (ASMI) displaying the IPv6 network prefix in decimal instead of hex character values.  The service processor command line "ifconfig" can be used to see the IPv6 network prefix value in hex as a circumvention to the problem.
• A problem was fixed for link speed for PCIe Generation 4 adapters showing as "unknown"  in the Advanced System Management Interface (ASMI) PCIe Hardware Topology menu.
• A problem was fixed for the system crashing on  PCIe errors that result in guard action for the FRU.
• A problem was fixed for an extraneous SRC B7000602 being logged intermittently when is the system is being powered off.  The trigger for the error log is a HMC request for information that does not complete before the system is shut down.  If the HMC sends certain commands to get capacity information (eg, 0x8001/0x0107) while the CEC is shutting down, the SFLPHMCCMD task can fail with this assertion.   This error log may be ignored.
• A problem was fixed for the service processor Thermal Management not being made aware of a Power Management failure that the hypervisor had detected.  This could cause the system to go into Safe Mode with degraded performance if the error does not have recovery done.
  
• A problem was fixed for the On-Chip Controller (OCC) being held in reset after a channel error for the memory.  The system would remain in Safe Mode (with degraded performance) until a re-IPL of the system. The trigger for the problem requires the memory channel checkstop and the OCC not being able to detect the error.  Both of these conditions are rare, making the problem unlikely to occur.
• A problem was fixed for the memory bandwidth sensors for the P9 memory modules being off by a factor of 2.  As a workaround, divide memory sensor values  by 2 to get a corrected value.
• A problem was fixed for known bad DRAM bits having  errors logs being generated repeatedly with each IPL.  With the fix, the error logs only occur one time at the initial failure  and then thereafter the known bad DRAM bits are repaired as  part of the normal memory initialization.
• A problem was fixed for a Hostboot run time memory channel error where the processor could be called out erroneously instead of the memory DIMM.  For this error to happen, there must be a RCD parity error on the memory DIMM with a channel failure attention on the processor side of the bus and no channel failure attention on the memory side of the bus, and the system must recover from the channel failure.
• A problem was fixed for DDR3 DIMM memory training where the ranks not being calibrated had their outputs enabled.  The JEDEC specification requires that the outputs be disabled.  Adding the termination settings on the non-calibrating ranks can improve memory margins ( thereby reduce the rate of memory failures), and it matches the memory training technique used for the DDR4 memory.
• A problem was fixed for a PCIe2 4-port Slot Adapter  with feature code #2E17  that cannot recover from a double EEH error if the second error occurs during the EEH recovery.  Because is a double-error scenario, the problem should be very infrequent.
• A rare problem was fixed for slow downs in a Live Partition Mobility migration of a partition with Active Memory Sharing (AMS).  The AMS partition does not fail but the slower performance could cause time-outs in the workload if there are time constraints on the operations.
• A problem was fixed for isolation of memory channel failure attentions on the processor side of the differential memory interface (DMI) bus.  This only is a problem if there are no attentions from the memory module side of the bus and it could cause the service processor run time diagnostics to get caught in hang condition, or result in a system checkstop with the processor called out.
• A problem was fixed for the memory bandwidth sensors for the P9 memory modules sometimes being zero.
• A problem was fixed for  deconfiguring checkstopped processor cores at run time.  Without the fix, the processor core checkstop error could cause a checkstop of the system and a re-IPL,  or it could force the system into Safe Mode.
• A problem was fixed for a failed TPM card preventing a system IPL, even after the card was replaced.
• A problem was fixed for differential memory interface (DMI) lane sparing to prevent shutting down a good lane on the TX side of the bus when a lane has been spared on the RX side of the bus.  If the  XBUS or DMI bus runs out of spare lanes, it can checkstop the system, so the fix helps use these resources more efficiently.
• A problem was fixed for IPL failures with SRC BC50090F when replacing Xbus FRUs.  The problem occurs if VPD has a stale bad memory lane record and that record does not exist on both ends of the bus.
• A problem was fixed for SR-IOV adapter dumps hanging with low-level EEH events causing failures on VFs of other non-target SR-IOV adapters.
• A problem was fixed for SR-IOV VF configured with a PVID  that fails to function correctly after a virtual function reset.  It will allow receiving untagged frames but not be able to transmit the untagged frames.
• A problem was fixed for SR-IOV VFs, where a VF configured with a PVID  priority may be presented to the OS with an incorrect priority value.
  
• A problem was fixed for a Self Boot Engine (SBE) recoverable error at run time causing the system to go into Safe Mode.
• A problem was fixed for a rare Live Partition Mobility migration hang with the partition left in VPM (Virtual Page Mode) which causes performance concerns.  This error is triggered by a  migration failover operation occurring during the migration state of "Suspended" and there has to be insufficient VASI buffers available to clear all partition state data waiting to be sent to the migration target.  Migration failovers are rare and the migration state of "Suspended" is a migration state lasting only a few seconds for most partitions, so this problem should not be frequent.  On the HMC, there will be an inability to complete either a migration stop or a recovery operation.  The HMC will show the partition as migrating and any attempt to change that will fail.  The system must be re-IPLed to recover from the problem.
• A problem was fixed for Self Boot Engine (SBE) failure data being collected from the wrong processor if the SBE is not running on processor 0.  This can result in the wrong FRU being called out for SBE failures.
  

System firmware changes that affect certain systems

• On systems which do not have a HMC attached,  a problem was fixed for a firmware update initiated from the OS from FW920.00 to FW920.10 that caused a system crash one hour after the code update completed.  This does not fix the case of the OS initiated firmware update back to FW920.00 from FW920.10 which will still result in a crash of the system.  Do not initiate a FW920.10  to FW920.00 code update via the operating system.  Use only HMC or USB methods of code update for this case.  If a HMC or USB code update is not an option,  please contact IBM support.
  
• A problem was fixed for Linux or AIX partitions crashing during a firmware assisted dump or when using Linux kexec to restart with a new kernel.  This problem was more frequent for the Linux OS with kdump failing with "Kernel panic - not syncing: Attempted to kill init" in some cases.
• On a system with an IBM i partition with more than 64 virtual processors assigned to it, a problem was fixed for a possible system crash or other unexpected behavior when doing a partition dump IPL
• On a system with an IBM i partition, a problem was fix for I/O operations timeouts with SRCs B600512D and B6005275 logged and IBM i performance degradation.  This problem only occurs with heavy I/O traffic.

• GA Level