New features and functions

• Support was added for redundant VPD EEPROMs.  If the primary module VPD EEPROM fails, the system will automatically change to the backup module.
• Enable periodic logging of internal component operational data for the PCIe3 expansion drawer paths. The logging of this data does not impact the normal use of the system.
• Support added for SR-IOV Hybrid Network Virtualization (HNV) in a production environment (no longer a Technology Preview) for AIX and IBM i.   This capability allows an AIX or IBM i partition to take advantage of the efficiency and performance benefits of SR-IOV logical ports and participate in mobility operations such as active and inactive Live Partition Mobility (LPM) and Simplified Remote Restart (SRR).  HNV is enabled by selecting a new Migratable option when an SR-IOV logical port is configured. The Migratable option is used to create a backup virtual device.  The backup virtual device can be either a Virtual Ethernet adapter or a virtual Network Interface Controller (vNIC) adapter. In addition to this firmware HNV support in a production environment requires HMC 9.1.941.0 or later, AIX Version 7.2 with the 7200-04 Technology Level and Service Pack 7200-04-02-2015 or AIX Version 7.1 with the 7100-05 Technology Level and Service Pack 7100-05-06-2015, IBM i 7.3 TR8 or IBM i 7.4 TR2, and VIOS 3.1.1.20.
  

System firmware changes that affect all systems

• DEFERRED:  A problem was fixed for a processor core failure with SRCs B150BA3C and BC8A090F logged that deconfigures the entire processor for the current IPL.  A re-IPL of the system will recover the lost processor with only the bad core guarded.
• A problem was fixed for Performance Monitor Unit (PMU) events that had the incorrect Alink address (Xlink data given instead) that could be seen in 24x7 performance reports.  The Alink event data is a recent addition for FW940 and would not have been seen at the earlier firmware levels.
• A problem was fixed for an SR-IOV adapter hang with B400FF02/B400FF04 errors logged during firmware update or error recovery.  The adapter may recover after the error log and dump, but it is possible the adapter VF will remain disabled until the partition using it is rebooted.  This affects the SR-IOV adapters with the following feature codes and CCINs:   #EC2R/EC2S with CCIN 58FA;  #EC2T/EC2U with CCIN 58FB;  #EC3L/EC3M with CCIN 2CEC;  and #EC66/EC67 with CCIN 2CF3.
• A problem was fixed for a failed clock card causing a node to be guarded during the IPL of a multi-node system.  With the fix, the redundant clock card allows all the nodes to IPL in the case of a single clock card failure.
• A problem was fixed for the green power LED on the System Control Unit (SCU) not being lit even though the system is powered on.  Without the fix, the LED is always in the off state.
• A problem was fixed for a loss of service processor redundancy after a failover to the backup on a Hostboot IPL error.  Although the failover is successful to the backup service processor, the original primary service processor may terminate.  The failed service processor can be recovered from termination by using a soft reset from ASMI.
• A problem was fixed for extraneous B400FF01 and B400FF02 SRCs logged when moving cables on SR-IOV adapters.  This is an infrequent error that can occur if the HMC performance monitor is running at the same time the cables are moved.  These SRCs can be ignored when accompanied by cable movement.
• A problem was fixed for certain SR-IOV adapters that can have B400FF02 SRCs logged with LPA dumps during Live Partition Mobility (LPM) migrations or vNIC failovers.  The adapters can have issues with a deadlock on error starts after many resets of the VF and errors in managing memory pages.  In most cases, the operations should recover and complete.  This fix updates the adapter firmware to 1X.25.6100  for the following Feature Codes and CCINs: #EC2R/EC2S with CCIN 58FA; #EC2T/EC2U with CCIN 58FB;  #EC3L/EC3M with CCIN 2CE; and #EC66/EC67 with CCIN 2CF3.
  The SR-IOV adapter firmware level update for the shared-mode adapters happens under user control to prevent unexpected temporary outages on the adapters.  A system reboot will update all SR-IOV shared-mode adapters with the new firmware level.  In addition, when an adapter is first set to SR-IOV shared mode, the adapter firmware is updated to the latest level available with the system firmware (and it is also updated automatically during maintenance operations, such as when the adapter is stopped or replaced).  And lastly, selective manual updates of the SR-IOV adapters can be performed using the Hardware Management Console (HMC).  To selectively update the adapter firmware, follow the steps given at the IBM Knowledge Center for using HMC to make the updates:   https://www.ibm.com/support/knowledgecenter/en/POWER9/p9efd/p9efd_updating_sriov_firmware.htm.
  Note: Adapters that are capable of running in SR-IOV mode, but are currently running in dedicated mode and assigned to a partition, can be updated concurrently either by the OS that owns the adapter or the managing HMC (if OS is AIX or VIOS and RMC is running).
  
• A problem was fixed where SR-IOV adapter VFs occasionally failed to provision successfully on the low-speed ports (1 Gbps) with SRC B400FF04 logged, or SR-IOV adapter VFs occasionally failed to provision successfully with SRC B400FF04 logged when the RoCE option is enabled.
  This affects the adapters with low speed ports (1 Gbps) with the following  Feature Codes and CCINs:  #EN0H/EN0J with CCIN 2B93, #EN0M/EN0N with CCIN 2CC0,  and #EN0K/EN0L with CCIN 2CC1.  And it affects  the adapters with the ROCE option enabled  with the following feature codes and CCINs:   #EC2R/EC2S with CCIN 58FA;  #EC2T/EC2U with CCIN 58FB;  #EC3L/EC3M with CCIN 2CEC;  and #EC66/EC67 with CCIN 2CF3.
  
• A problem was fixed for an expired trial or elastic Capacity on Demand ( CoD) memory not warning of the use of unlicensed memory if the memory is not returned. This lack of warning can occur if the trial memory has been allocated as Virtual Persistent Memory (vPMEM).
• A problem was fixed for a B7006A96 fanout module FPGA corruption error that can occur in unsupported PCIe3 expansion drawer(#EMX0) configurations that mix an enhanced PCIe3 fanout module (#EMXH) in the same drawer with legacy PCIe3 fanout modules (#EMXF, #EMXG, #ELMF, or #ELMG).  This causes the FPGA on the enhanced #EMXH to be updated with the legacy firmware and it becomes a non-working and unusable fanout module.  With the fix, the unsupported #EMX0 configurations are detected and handled gracefully without harm to the FPGA on the enhanced fanout modules.
• A problem was fixed for possible dispatching delays for partitions running in POWER8, POWER9_base or POWER9 processor compatibility mode.
• A problem was fixed for system memory not returned after create and delete of partitions, resulting in slightly less memory available after configuration changes in the systems.  With the fix, an IPL of the system will recover any of the memory that was orphaned by the issue.
• A problem was fixed for failover support for the Mover Service Partition (MSP) where a failover to the MSP partner during an LPM could cause the migration to abort.   This vulnerability is only for a very specific window in the migration process. The recovery is to restart the migration operation.
• A rare problem was fixed for a checkstop during an IPL that fails to isolate and guard the problem core.  An SRC is logged with B1xxE5xx and an extended hex word 8 xxxxDD90.  With the fix, the failing hardware is guarded and a node is possibly deconfigured to allow the subsequent IPLs of the system to be successful.
• A problem was fixed for a hypervisor error during system shutdown where a B7000602 SRC is logged and the system may also briefly go "Incomplete" on the HMC but the shutdown is successful.  The system will power back on with no problems so the SRC can be ignored if it occurred during a shutdown.
  
• A problem was fixed for certain NVRAM corruptions causing a system crash with a bad pointer reference instead of expected Terminate Immediate (TI) with B7005960 logged.
• A problem was fixed for certain SR-IOV adapters that do not support the "Disable Logical Port" option from the HMC but the HMC was allowing the user to select this, causing incorrect operation.  The invalid state of the logical port causes an "Enable Logical Port" to fail in a subsequent operation.  With the fix, the HMC provides the message that the "Disable Logical Port" is not supported for the adapter.  This affects the adapters with the following  Feature Codes and CCINs: #EN15/EN16 with CCIN 2CE3,  #EN17/EN18 with CCIN 2CE4, #EN0H/EN0J with CCIN 2B93, #EN0M/EN0N with CCIN 2CC0, and  #EN0K/EN0L with CCIN 2CC1.
  
• A problem was fixed for clock card errors not being called out in the error log when the primary clock card fails.  This problem makes it more difficult for the system user to be aware that clock card redundancy has been lost, and that service is needed to restore the redundancy.
• A problem was fixed to remove unneeded resets of a VF for SR-IOV adapters, providing for improved performance of the startup or recovery time of the VF.  This performance difference may be noticed during a Live Partition Mobility migration of a partition or during vNIC (Virtual Network Interface Controller) failovers where many resets of VFs are occurring.
• A problem was fixed for SR-IOV adapters having an SRC B400FF04 logged when a VF is reset.  This is an infrequent issue and can occur for a Live Partition Mobility migration of a partition or during vNIC (Virtual Network Interface Controller) failovers where many resets of VFs are occurring.  This error is recovered automatically with no impact on the system.
• A problem was fixed for initial configuration of  SR-IOV adapter VFs with certain configuration settings for the following Feature Codes and CCINs:   #EC2R/EC2S with CCIN 58FA; #EC2T/EC2U with CCIN 58FB;  #EC3L/EC3M with CCIN 2CE;  and #EC66/EC67 with CCIN 2CF3.
  These VFs may then fail following an adapter restart, with other VFs functioning normally.  The error causes the VF to fail with an SRC B400FF04 logged.With the fix, VFs are configured correctly when created. 
  Because the error condition may pre-exist in an incorrectly configured logical port, a concurrent update of this fix may trigger a logical port failure when the VF logical port is restarted during the firmware update.  Existing VFs with the failure condition can be recovered by dynamically removing/adding the failed port and are automatically recovered during a system restart.
• A problem was fixed for TPM hardware failures not causing SRCs to logged with a call out if the system is configured in ASMI to not require TPM for the IPL.  If this error occurs, the user would not find out about it until they needed to run with TPM on the IPL.  With the fix, the error logs and notifications will occur regardless of how the TPM is configured.
  

• On systems with an IBM i partition, a problem was fixed for a D-mode IPL failure when using a USB DVD drive in an IBM 7226 multimedia storage enclosure.  Error logs with SRC BA16010E, B2003110, and/or B200308C can occur.  As a circumvention, an external DVD drive can be used for the D-mode IPL.
• On systems with an IBM i partition, a problem was fixed that occurs after a Live Partition Mobility (LPM) of an IBM i partition that may cause issues including dispatching delays and the inability to do further LPM operations of that partition.  The frequency of this problem is rare.  A partition encountering this error can be recovered with a reboot of the partition.
  
• On systems with Integrated Facility for Linux ( IFL) processors and Linux-only partitions, a problem was fixed for Power Enterprise Pools (PEP) 1.0 not going back into "Compliance" when resources are moved from Server 1 to Server 2, causing an expected "Approaching Out Of Compliance", but not automatically going back into compliance when the resources are no longer used on Server 1.  As a circumvention, the user can do an extra "push" and "pull" of one resource to make the Pool discover it is back in "Compliance".
• On systems with an IBM i partition in POWER9 processor compatibility mode, a problem was fixed for an MSD in IBM i with SRCs B6000105 or B6000305 logged when a PCIe Host Bridge (PHB) or PCIe Expansion Drawer (#EMX0) is added to the partition.  For this to occur, the adapter had to be previously assigned to a partition (any OS) that was in POWER9 processor compatibility mode and then removed through a DLPAR or partition shut down such that the adapter is taken through recovery.
• On systems with an IBM i partition, a problem was fixed for a possibly incorrect number of Memory COD (Capacity On Demand) resources shown when gathering performance data with IBM i Collection Services. Memory resources activated by Power Enterprise Pools (PEP) 1.0 will be missing from the data.  An error was corrected in the IBM i MATMATR option 0X01F6 that retrieves the Memory COD information for the Collection Services.
• For systems with deconfigured cores and using the default performance and power setting of "Dynamic Performance Mode" or "Maximum Performance Mode", a rare problem was fixed for an incorrect voltage/frequency setting for the processors during heavy workloads with high ambient temperature.  This error could impact power usage, expected performance, or system availability if a processor fault occurs.  This problem can be avoided by using ASMI "Power and Performance Mode Setup"  to disable "All modes" when there are cores deconfigured in the system.
  

System firmware changes that affect all systems

• A problem was fixed for an HMC "Incomplete" state for a system after the HMC user password is changed with ASMI on the service processor.  This problem can occur if the HMC password is changed on the service processor but not also on the HMC, and a reset of the service processor happens.  With the fix, the HMC will get the needed "failed authentication" error so that the user knows to update the old password on the HMC.
  

• HIPER/Pervasive:   For systems using PowerVM NovaLink to manage partitions, a problem was fixed for the hypervisor rejecting setting the system to be NovaLink managed.  The following error message is given:   "FATAL pvm_apd[]: Hypervisor encountered an error creating the ibmvmc device. Error number 5."  This always happens in FW940.00 and FW940.01 which prevents a system from transitioning to be NovaLink managed at these firmware levels.  If you were successfully running as NovaLink managed already on FW930 and upgraded to FW940, you would not experience this issue. 
  For more information on PowerVM Novalink, refer to the IBM Knowledge Center article:  https://www.ibm.com/support/knowledgecenter/POWER9/p9eig/p9eig_kickoff.htm.

New features and functions

• Support was added for improved security for the service processor password policy.  For the service processor, the "admin", "hmc" and "general" password must be set on first use for newly manufactured systems and after a factory reset of the system. The REST/Redfish interface will return an error saying the user account is expired in these scenarios.  This policy change helps to enforce the service processor is not left in a state with a well known password.  The user can change from an expired default password to a new password using the Advanced System Management Interface (ASMI).
• Support was added for real-time data capture for PCIe3 expansion drawer (#EMX0) cable card connection data via resource dump selector on the HMC or in ASMI on the service processor.  Using the resource selector string of "xmfr -dumpccdata" will non-disruptively generate an RSCDUMP type of dump file that has the current cable card data, including data from cables and the retimers.
  

System firmware changes that affect all systems

• A problem was fixed for persistent high fan speeds in the system after a service processor failover.  To restore the fans to normal speed without re-IPLing the system requires the following steps:
  1) Use ASMI to perform a soft reset of the backup service processor.
  2) When the backup service processor has completed its reset, use the HMC to do an administrative failover, so that the reset service processor becomes the primary.
  3) Use ASMI to perform a soft reset on the new backup service processor.  When this has completed, system fan speeds should be back to normal.
• A problem was fixed for system hangs or incomplete states displayed by HMC(s) with SRC B182951C logged.  The hang can occur during operations that require a memory relocation for any partition such as Dynamic Platform Optimization (DPO), memory mirroring defragmentation, or memory guarding that happens as part of memory error recovery during normal operations of the system.
• A problem was fixed for possible unexpected interrupt behavior for partitions running in POWER9 processor compatibility mode.  This issue can occur during the boot of a partition running in POWER9 processor compatibility mode with an OS level that supports the External Interrupt Virtualization Engine (XIVE) exploitation mode.  For more information on compatibility modes, see the following two articles in the IBM Knowledge Center:
  1) Processor compatibility mode overview:   https://www.ibm.com/support/knowledgecenter/POWER9/p9hc3/p9hc3_pcm.htm
  2) Processor compatibility mode definitions:  https://www.ibm.com/support/knowledgecenter/POWER9/p9hc3/p9hc3_pcmdefs.htm
• A problem was fixed for an intermittent IPL failure with SRC B181E540 logged with fault signature " ex(n2p1c0) (L2FIR[13]) NCU Powerbus data timeout".  No FRU is called out.  The error may be ignored and the re-IPL is successful.  The error occurs very infrequently.  This is the second iteration of the fix that has been released.  Expedient routing of the Powerbus interrupts did not occur in all cases in the prior fix, so the timeout problem was still occurring.

• On systems running IBM i partitions configured as Restricted I/O partitions that are also running in either P7 or P8 processor compatibility mode, a problem was fixed for a likely hang during boot with BA210000 and BA218000 checkpoints and error logs after migrating to FW940.00 level system firmware.  The trigger for the problem is booting IBMi partitions configured as Restricted I/O partitions in P7 or P8 compatibility mode on FW940.00 system firmware.  Such partitions are usually configured this way so that they can be used for live partition migration (LPM) to and
  from P7/P8 systems. Without the fix, the user can do either of the following as circumventions for the boot failure of the IBM i partition:
  1) Move the partition to P9 compatibility mode
  2) Or remove the 'Restricted I/O Partition' property

• All features and fixes from the FW930.11. service pack (and below) are included in this release.  At the time of the FW940.00 release, the FW930.11 is a future FW930 service pack scheduled for the fourth quarter of 2019.

• User Mode NX Accelerator Enablement for PowerVM.  This enables the access of NX accelerators such as the gzip engine through user mode interfaces.  The IBM Virtual HMC (vHMC) 9.1.940 provides a user interface to this feature.  The LPAR must be running in POWER9 compatibility mode to use this feature.  For more information on compatibility modes, see the following two articles in the IBM Knowledge Center:
  1) Processor compatibility mode overview:    https://www.ibm.com/support/knowledgecenter/POWER9/p9hc3/p9hc3_pcm.htm
  2) Processor compatibility mode definitions:  https://www.ibm.com/support/knowledgecenter/POWER9/p9hc3/p9hc3_pcmdefs.htm
• Support for SR-IOV logical ports in IBM i restricted I/O mode.
• Support for user mode enablement of the External Interrupt Virtualization Engine (XIVE).  This user mode enables the management of interrupts to move from the hypervisor to the operating system for improved efficiency.  Operating systems may also have to be updated to enable this support.  The LPAR must be running in POWER9 compatibility mode to use this feature.  For more information on compatibility modes, see the following two articles in the IBM Knowledge Center:
  1) Processor compatibility mode overview:    https://www.ibm.com/support/knowledgecenter/POWER9/p9hc3/p9hc3_pcm.htm
  2) Processor compatibility mode definitions:  https://www.ibm.com/support/knowledgecenter/POWER9/p9hc3/p9hc3_pcmdefs.htm
• Extended support for PowerVM Firmware Secure Boot.  This feature restricts access to the Open Firmware prompt and validates all adapter boot driver code. Boot adapters, or adapters which may be used as boot adapters in the future, must be updated to the latest microcode from IBM Fix Central.  The latest microcode will ensure the adapters support the Firmware Secure Boot feature of Power Systems. This requirement applies when updating system firmware from a level prior to FW940 to levels FW940 and later.  The latest adapter microcode levels include signed boot driver code.  If a boot-capable PCI adapter is not installed with the latest level of adapter microcode, the partition which owns the adapter will boot, but error logs with SRCs BA5400A5 or BA5400A6 will be posted. Once the adapter(s) are updated, the error logs will no longer be posted.
• Linux OS support was added for PowerVM LPARs for the PCIe4 2x100GbE ConnectX-5 RoCE adapter with feature codes of #EC66/EC67 and CCIN 2CF3.  Linux versions RHEL 7.5 and SLES 12.3 are supported. 
  

• A problem was fixed for incorrect call outs for PowerVM hypervisor terminations with SRC B7000103 logged.  With the fix, the call outs are changed from SVCDOCS, FSPSP04, and FSPSP06 to FSPSP16.  When this type of termination occurs, IBM support requires the dumps be collected to determine the cause of failure.
  
• A problem was fixed for an IPL failure with the following possible SRCs logged:  11007611, 110076x1, 1100D00C, and 110015xx.  The service processor may reset/reload for this intermittent error and end up in the termination state.