1.0 Systems Affected

2. Update BMC to BMC 2.04 or later using pUpdate 2.20 (or BMC Web GUI).

3. After BMC firmware updated, the BMC is automatically reset. Wait for BMC to come back on line

3.1. Being able to connect BMC Web GUI or able to ping to BMC is an indication of BMC back on line

4. Check BMC version to ensure the right BMC version is updated

5. You must reset the nginx configuration for Redfish (REST API) support:

a) ipmitool -I lanplus -H bmc_ip_address -U <ADMIN_userid> -P <ADMIN_password> raw 0x30 0x70 0xb7

b) ipmitool -I lanplus -H bmc_ip_address -U <ADMIN_userid> -P <ADMIN_password> mc reset cold

c) Wait for BMC to come back on line.

6. Update PNOR version 2.xx using pUpdate 2.20 or using BMC Web GUI

6.1. The new PNOR version is not available at this time.

7. Once the PNOR update is complete, power on the system to petitboot menu or OS.

8. Once the system reaches to OS, you can check the PNOR version to ensure it is correct.

9. System is ready to be used.

Note 2: If BMC and PNOR need to be downgraded from 2.xx or later level down to any 1.xx version of the firmware, the system must be powered off before the update and an AC cycle is needed after the update. If you downgrade a BMC or a PNOR to any 1.xx version, you must downgrade both the BMC and PNOR firmware so that each is at the same primary version level such as 1.xx. There are dependencies between BMC and PNOR. For that reason, you must update BOTH BMC and PNOR while system power is off. Here are the steps:

2. Update BMC version 1.xx using pUpdate 2.20 (or BMC Web GUI)

3. After BMC firmware updated, the BMC is automatically reset. Wait for BMC to come back on line

3.1. Being able to connect BMC Web GUI or able to ping to BMC is an indication that BMC is back on line

4. Check the BMC version to ensure the right BMC version is updated

5. Update PNOR version 1.xx using pUpdate 2.20 (or BMC Web GUI)

6. System must be AC power cycled after PNOR update is complete

1. Unplug all power cables to the system

2. Wait for a bit (15-30 seconds)

3. Plug all the power cables to the system.

4. Wait for BMC to come back on line

4.1 Being able to connect to the BMC WEB GUI or able to ping to BMC is an indication that the BMC is back on line.

7 Power on the system to petitboot menu or OS.

8. Once the system reaches to OS, you can check the PNOR version to ensure it is correct.

9 System is ready to be used.

Note 3: After updating BMC firmware to version 2.10 and newer, the current BMC password for "ADMIN" login ID is still in affect until factory reset is issued to the BMC. Once the factory reset is issued to the BMC, the BMC reset the password for the "ADMIN" login ID to the default password, "ADMIN", but expired. No remote command will work except the command to change the password. You can change to a new password via the interfaces below. You must follow the new password rules: 1) Must be at least 8 characters. 2) Must have at least one lower case alphabet, one upper case alphabet and one numeric number.

Note: A factory reset sets all user IDs back to default. That means that User ID 1 expiration and password are set back to defaults, User ID 2 (a.k.a. account002) is set back to a name of "ADMIN" with password of "ADMIN" and no expiration, and User IDs 3-10 are completely deleted.

1. Web GUI

2. SSH

3. In-band IPMI command (host must have root privilege)

4. Redfish

1.1 Minimum ipmitool Code Level

This section specifies the "Minimum ipmitool Code Level" required by the System Firmware to perform firmware installations and managing the system. OpenPOWER requires ipmitool level v1.8.15 to execute correctly on the LC server firmware.

Verify your ipmitool level on your Linux workstation using the following commands:

bash-4.1$ ipmitool -V

ipmitool version 1.8.15

If you need to update or add impitool to your Linux workstation , you can compile ipmitool (current level 1.8.15) for Linux as follows from Sourceforge:

1.1.1 Download ipmitool tar from http://sourceforge.net/projects/ipmitool / to your Linux system

1.1.2 Extract tarball on Linux system

1.1.3 cd to top-level directory

1.1.4 ./configure

1.1.5 make

1.1.6 ipmitool will be under src/ipmitool

1.2 Minimum Browser levels for BMC Web GUI

The BMC Web GUI is a web-based application that works within a browser. Supported browser levels are shown below with Chrome being the preferred browser:

•Google Chrome Version 46.0.2490.71 m
•Mozilla Firefox version 41.0.3

1.3 Fix level Information on IBM OpenPOWER Components and Operating systems

For specific fix level information on key components of IBM Power Systems servers and Linux operating systems, please refer to the documentation in the IBM Knowledge Center.

Here are the links for the LC921 and LC922 servers:

9006-12P: http://www.ibm.com/support/knowledgecenter/POWER9/p9hdx/9006_12 p _landing.htm

9006-22P http://www.ibm.com/support/knowledgecenter/POWER9/p9hdx/9006_22 p _landing.htm

1.4 Enabling Trusted Boots in RHEL7.5-ALT

On Red Hat Enterprise Linux (RHEL) for PPC, RHEL-Alt 7.5, The Trusted Platform Module (TPM) device driver is not loaded automatically at boot time. Without this driver, the TPM device will not be accessible.

This affects any user-space application needing to access the TPM, as well as kernel security functions, such as the Integrity Measurement Architecture subsystem (IMA) in the Linux kernel. Without the TPM driver loaded, IMA will be unable to record trusted measurements to the TPM.

To load the driver manually, as root:

# modprobe tpm_i2c_nuvoton

To load the driver automatically at boot time:

# echo "tpm_i2c_nuvoton" > /etc/modules-load.d/tpm.conf"

The TPM device driver will be integrated as a built-in kernel module in a future release 7 of RHEL-Alt. Once this is done, it will be loaded automatically and this procedure will no longer be necessary

2.0 Important Information

Downgrading firmware from any given release level to an earlier release level is not recommended.

If you feel that it is necessary to downgrade the firmware on your system to an earlier release level, please contact your next level of support.

Concurrent Firmware Updates not available for LC servers.

Concurrent system firmware update is not supported on these LC servers.

2.1 Recovering from an expired password or changing ADMIN login ID’s password

These instructions are for recovering the ADMIN user ID that has expired for the default “ADMIN” password because the system is new or has been factory reset. These instructions assume that the BMC is on the network.

Assumptions:

BMC login ID is "ADMIN".

Default password for "ADMIN" login ID is "ADMIN"

Assumes the user knows how to login to BMC using ssh, Web GUI

Assumes the user knows how to login to OS

Assumes the user know how to use ipmitool command

Assumes BMC network is up.

Note: The instructions below can be used for any login ID. However, the below instructions only target the "ADMIN" login ID.

The Web GUI, secured shell (SSH), ipmitool from the host, or Redfish may be used to change the expired password as shown below:

1. Web GUI:

1.1 When a user is logging into BMC Web GUI with the "ADMIN" user ID after BMC factory reset, the user must use default password (see above) regardless of what the previous "ADMIN" user ID's password was.

1.2. Once successfully logged in with the "ADMIN" user ID, the BMC Web GUI will force the user to change the password for the "ADMIN" user ID. The BMC Web GUI will log out after successfully changing the password.

1.3. Now the user can log in to the BMC using "ADMIN" user ID with the new password.

2. SSH:

2.1. When a user is logging into a BMC using ssh with "ADMIN" user ID after the BMC factory reset, the user must use the default password (see above) regardless of what the previous "ADMIN" login ID's password was.

2.2 Once successfully logged in with "ADMIN" user ID, the ssh will force the user to change the BMC password for the "ADMIN" user ID. Follow the instructions carefully. ssh will log out after successfully changing the password.

2.3. Now the user can log in to the BMC using the "ADMIN" user ID with the new password.

3. In-band IPMI command

3.1. Power on the host system to OS.

3.2. Log in to the OS as root.

3.3. Change BMC "ADMIN" user ID's password:

3.3.1. Command: ipmitool user set password 2 newPW

3.3.2. newPW is the desired new password.

4. Redfish has two methods that can be used.

4.1. Method 1: Run below command from a remote linux system

4.1.1 Cmd: curl -sD - --data '{"UserName":"ADMIN","Password":"curPW"}' -k -X POST https://[bmc_ ip]/redfish/v1/SessionService/Session

a) curPW is the current password. It could be the default password or other password that is currently set on the BMC.

b) Need "X-Auth-Token:...." from the output of this command for the next command

c)[bmc_ip] is the BMC name or BMC IP address

4.1.2. Cmd: curl -H "X-Auth-Token:...." --data '{"Password":"newPW"}' -k -X PATCH https://[bmc_ip]/redfish/v1/AccountService/Accounts/2

a) [bmc_ip] is the BMC name or BMC IP address

b) newPW is new password you want to change to

4.2. Method 2: Run below command from a remote linux system

4.2.1. Cmd: curl -u ADMIN:curPW --data '{"Password":"newPW"}' -k -X PATCH https://[bmc_ip]/redfish/v1/AccountService/Accounts/2

a) curPW is the current password. It could be the default password or other password that is currently set on the BMC.

b). newPW is new password you want to change to

5. Changing BMC password using SMASH-CLP via ssh

5.1. This only works with the ship level BMC firmware (not available for debug version).

5.2. ssh to BMC

5.2.1 Go to user account target (ADMIN is account002)

5.2.1.1. Cmd: cd /map1/account002

5.2.1.2 Run "show" to see account002 setting

5.2.2. Change password for account002

5.2.2.1. Cmd: set password=newPW

5.2.2.1.1. newPW is the desired new password.

5.2.3. Log out

5.2.3.1. Cmd: exit

5.3. Now the user can login into BMC using "ADMIN" login ID with the newPW

3.0 Firmware Information

Use the following examples as a reference to determine whether your installation will be concurrent or disruptive.

For the LC server systems, the installation of system firmware is always disruptive.

3.1 Firmware Information and Description

The xxx.pnor file updates the primary side of the PNOR. The yyy.bin updates the primary side of the BMC only. The golden sides are unchanged.

The pUpdate zip file contains the Linux powerpc, x86(32 bit), x86(64 bit), and pUpdate user’s guide. The version of the tool selected needs to match the hardware where the tool is running.

Filename	Size	Checksum
P9DSU20210329_IBM_prod_sign.pnor	67108992	015fbe430c05076d9b548b7a868cb2f0
SMT_P9_213.bin	33554432	a66b7f6f48c409d118453107160c563c
pUpdate_220.zip	91832	745f57ea76b2bc641dc8005bf960ecb3

Note: The Checksum can be found by running the Linux/Unix/AIX md5sum command against the file (all 32 characters of the checksum are listed), ie: md5sum pUpdate.

After a successful update to the new firmware level, the PNOR components and BMC should be at the following levels. The ipmitool "fru" command can be used to display FRU ID 47 and the ipmitool "mc info" command can be used to display the BMC level.

Note: FRU information for the PNOR level does not show the updated levels via the fru command until the system has been booted once at the updated level.

PNOR firmware levels from FRU ID 47 inventory list for driver:

open-power-SUPERMICRO-P9DSU-V2.18-20210329-prod

op-build-bdd88bc

buildroot-2018.11.3-12-g222837a

skiboot-v6.0.23

hostboot-e008e33-p37e8440

occ-8fa3854

linux-4.19.57-openpower1-pf7816aa

petitboot-v1.7.6-p0187192

machine-xml-734a35e

hostboot-binaries-hw032021a.op920

capp-ucode-p9-dd2-v4

sbe-7d48841

hcode-hw032321a.op920

BMC Level:

Display BMC firmware level using the "ipmitool mc info | grep Firmware" command:

Firmware Revision : 2.13

OP920.00
For Impact, Severity and other Firmware definitions, Please refer to the below 'Glossary of firmware terms' url:
http://www14.software.ibm.com/webapp/set2/sas/f/power5cm/home.html#termdefs

OP920.40 / V2.18-20210329 / BMC V2.13

05/07/2021

Impact: Availability Severity: HIPER

System firmware changes that affect all systems

HIPER/Pervasive: A problem was fixed for a processor core checkstop with SRC BC70E540 logged with Signature Description " ex(n0p1c4) (NCUFIR[11]) NCU no response to snooped TLBIE". This problem is intermittent and random but occurs with relatively high frequency for certain workloads. The trigger for the failure is one core of a fused core pair going into a stopped state while the other core of the pair continues running.

OP920.30 / V2.17-20201110 / BMC

V2.13 (re-publish)

03/12/21

The wrong key was used to sign the PNOR image that was released as the original OP920.30. This resulted in a signature validation failure at code update time and thus the target system was not able to update to the new PNOR image.

This re-publish replaces the PNOR image with one signed with the correct key. No other functional changes are included.

OP920.30 / V2.17-20201110 / BMC V2.13

12/15/2020

Impact: Security Severity: SPE

New features and functions

Support was added for Redfish version 1.9.0. For more information on this Redfish version, refer to https://www.dmtf.org/sites/default/files/standards/documents/DSP0266_1.9.0.pdf.

System firmware changes that affect all systems

A problem was fixed for security vulnerability CVE-2007-1858. With this vulnerability, the default SSL cipher configuration used certain insecure ciphers, including the anonymous cipher, which could allow remote attackers to obtain sensitive information or have other unspecified impacts.

A problem was fixed for security vulnerability CVE-2016-2183. With this vulnerability, the DES and Triple DES ciphers, as used in the TLS, SSH, and IPSec protocols and other protocols and products, have a birthday bound of approximately four billion blocks, which makes it easier for remote attackers to obtain clear text data via a birthday attack against a long-duration encrypted session.

A problem was fixed for mixing memory DIMMs with different timings (different vendors) under the same memory controller that fail with a SRC BC20E504 error and DIMMs deconfigured. This is a "MCBIST_BRODCAST_OUT_OF_SYNC" error. The loss of memory DIMMs can result in a IPL failure. This problem can happen if the memory DIMMs have a certain level of timing differences. If the timings are not compatible, the failure will occur on the IPL during the memory training. To circumvent this problem, each memory controller should have only memory DIMMs from same vendor plugged.

A problem was fixed for a BMC gui "Alerts Modify" to not drop the first character of the e-mail address when re-visiting the page.

A problem was fixed for a web security issue against Cross-Site Request Forgery with an enhanced fix over the previous delivery for OP920.23.

A problem with a slow transfer of data using iKVM Virtual Storage was fixed to improve the performance. The slow down was caused by a change to use a secure tunnel for the data. With the fix, the default secure channel is providing slightly better performance throughput. If further performance is needed, the secure channel can be temporarily disabled using the "Virtual Media->Secure Encryption" option.

OP920.23 / V2.16-20200427 / BMC V2.12

05/20/2020

Impact: Availability Severity: SPE

New features and functions

Support was added for PLC script like function into Redfish to allow easier debugging.

Support was added to the BMC so that it can force a kdump or other OS dump using the ipmitool chassis power diag command

Support was added for the following grub2 enhancements for petitboot:

1) 'source' command is now supported.

2) UUID and label are now supported in the 'search' command.

System firmware changes that affect all systems

A problem was fixed for RHEL8 boot failures when using GPT Disklabels caused by dependencies on unsupported features in petitboot GRUB2. Functionality in the GRUB2 parser was extended to allow RHEL8 to boot correctly.

A problem was fixed for ipmitool pass through commands from the AMESTER tool causing the BMC to become unresponsive. The Automated Measurement of Systems for Temperature and Energy Reporting (AMESTER) is a software tool to remotely collect power, thermal, and performance metrics from IBM servers.

A problem was fixed for the system time moving ahead one day following a month or year change and a power off of the system. Important:: This fix does not update the date and time to correct values if these are wrong in the RTC. In the case where the date and time are wrong, the user must manually reconfigure the BMC date/time under the Configuration menu option prior to starting the operating system.

The following web security problems were addresses for the BMC:

1) Authentication bypass vulnerability that could allow a network attacker to become an admin user.

2) Persistent (or stored) XSS vulnerability that can occur when the data provided by the attacker is saved by the server, and then permanently displayed on "normal" pages returned to other users in the course of regular browsing, without proper HTML escaping.

3) Cross-Site Request Forgery

4) HTML injection

5) Authenticated code injection as root via command injection

A problem was fixed for incorrect eSEL index in the error logs.

A problem was fixed for a rare IPL failure with SRCs BC8A090F and BC702214 logged caused by an overflow of VPD repair data for the processor cores. A re-IPL of the system should recover from this problem.

A problem was fixed for a GPU being reset from an error state remaining fenced and not usable.

A problem was fixed for a hang in the OS reboot caused by a TOD failure

A problem was fixed to increase the severity of abnormal reboot events in the error log.

A problem was fixed in IPMI for a case where a pointer continued to be used after it was freed, causing an intermittent fail if the memory was reused.

A problem was fixed for eSEL PANIC logs not always being sent to the BMC prior to taking the system down with the error, causing the loss of a log.

A problem was fixed for a failure in a GPU that can occur if the GPU is reset at run time.

A problem was fixed to prevent printing a null when parsing and formatting VPD data.

A problem was fixed in the petitboot discover process to check for required parameters on the communication handlers and writing error logs for missing parameters instead of taking a segmentation fault.

A problem was fixed to display a warning if saving a boot configuration fails to provide an early alert of this problem to the user.

A problem was fixed for the petitboot console exiting because of confusion over interrupts from other petitboot processes.

A problem was fixed for truncation of multi-word strings in the petitboot user interface. When this failure happens, the first word of the string is displayed only, stopping at the first "blank" of the string. Plugin names and vendor variables have been affected by this.

OP920.22 / V2.14-20190807 / BMC V2.11

01/08/2020

Impact: Security Severity: SPE

New features and functions

Improved BMC password policy. For the BMC, the ADMIN password must be set on first use for newly-manufactured systems and after a factory reset of the system. This policy change helps to enforce the BMC is not left in a state with a well-known password.

The user can change from the default password to a new password using the interfaces below:

1. Web GUI

2. ssh (secured shell)

3. Redfish

4. In-band IPMI from the host with a user ID that has root privilege.

Examples of using these interfaces for changing an expired password can be found in the following IBM Knowledge Center article for logging in to the BMC: https://www.ibm.com/support/knowledgecenter/POWER9/p9eih/p9eih_bmc_loggingon.htm.

Support was added to the SMASH shell of ssh to allow changing of the ADMIN user ID password using the SMASH CLP command of "set password" while in the /map1/account002 directory.

Support was added to BMC to have a minimum password strength configured for all interfaces such that it must be at least 8 characters long and must include at least one upper case letter, one lower case letter, and one numeric digit.

Support added for Redfish 1.7.0.

The BMC web GUI for the Event Log page was enhanced to show event severity and acknowledgement status.

System firmware changes that affect all systems

A problem was fixed for a Qualys security scan on the BMC reporting security threat QID 11827 “HTTP Security Header Not Detected” on port 443. The following HTTP security header was not detected in the HTTP response:

Strict-Transport-Security (HSTS): This response header is a security feature that lets a web site tell browsers that it should only be communicated with using HTTPS instead of HTTP protocol

A problem was fixed for an incorrect time format in the ProductionDate property of the Redfish API "/redfish/v1/Chassis/Planar/Assembly#/Assemblies/X".

A problem was fixed for SMCIPMITool being unable to mount virtual media (problem was introduced in BMC V2.08 with a security fix for media vulnerabilities). This problem affected customers that use SMCIPMITool to reimage nodes, as one example of problem impact. Along with the BMC V3.16 level, the SMCIPMITool version 2.22.1 build 109023 or later is needed to fix the issue. The new versions of the SMCIPMITool are backward compatible with the previous versions of the tool.

A problem was fixed for an intermittent disable of the field for the IPv6 address in the BMC web GUI.

A security problem was fixed for the NTP service running on the BMC allowing queries of the internal NTP variables. This problem was reported with Qualsys QID 38293.

A security problem was fixed for multiple Virtual Media vulnerabilities reported by Eclypsium.

OP920.21 / V2.14-20190807 / BMC V2.07

10/07/19

Impact: Data Severity: HIPER

New features and functions

The BMC web gui was enhanced so that the LDAP port can be set regardless of the SSL settings.

Support was added for the BMC web gui to be able to enable and disable IPMI over LAN.

Support was added for Redfish 1.6.1 (2018.3) and new Redfish APIs:

1) /redfish/v1/CertificateService

2) /redfish/v1/Managers/BMC/Oem/Supermicro/IPAccessControl

3) /redfish/v1/Managers/BMC/Oem/Supermicro/BackupRestoreService

4) Removed post and delete method on Firmware Inventory and use HttpPushUri instead for file upload.

5) Revised patch array for ManagerNetworkProtocol.

6) Added Active Directory and LDAP.

Security was enhanced for stunnel by allowing SSL Medium Strength and Anonymous Cipher Suites to be disabled. A reset of the stunnel configuration is required to do the disable using the following ipmitool commands one time:

1) ipmitool ... raw 0x30 0x70 0xB9

2) ipmitool ... mc reset cold

Added support for Active Directory to allow the BMC to make connections to LDAP\AD servers.

A change was made to enable OS software checkstops by default. This prevents hangs of multiple hours in failed reboots if the CPUs become stuck at the start of a kdump.

System firmware changes that affect all systems

A change was made to fix an intermittent processor anomaly that may result in issues such as operating system or hypervisor termination, application segmentation fault, hang, or undetected data corruption. The only issues observed to date have been operating system or hypervisor terminations.

A problem was fixed for Logical Volume Manager (LVM) virtual raid drives not being found that could prevent a system boot.. This problem is a regression error introduced at OP920.20 for PNOR version V2.12 when the "mdadm" command was moved from /usr/sbin to /sbin on the system.

A problem was fixed for failed FRUs associated with checkstops in hostboot not being guarded. This is an intermittent timing problem related to error log entries that have been created but not written or flushed yet at the time of the guard processing.

A problem was fixed for an IPL failure caused by an IPMI timeout. This is a rare problem and the reIPL after the failure recovers from the problem. The eSEL for the IPMI timeout may be ignored.

A problem was fixed for an OS boot that fails because the BMC is going through a reboot itself. The OS boot can fail when it needs to use BMC services when accessing the flash memory. This can happen if the BMC is not ready to receive commands. With the fix, the boot waits for the BMC to become ready instead of failing immediately on the errant flash access.

A problem was fixed for a fast reboot of the OS failing if VFs (Virtual Funcitions) were enabled and disabled prior to the reboot.

A problem was fixed for bad flashes caused by data size of memory to flash not being block aligned. This error can intermittently cause partial data to be written to the flash.

A problem was fixed for a hang in the OS reboot caused by a TOD failure.

A problem was fixed for a session timeout when clicking the "System Event Log" web page.

A problem was fixed for Redfish errors in /redfish/v1/UpdateService.

A problem was fixed for CGI aborts when uploading configurations using HTTP.

A problem was fixed for SSL certificate checks that were incorrectly failing on the check of the private key.

A security problem was fixed for the BMC ethernet Network Interface Card (NIC) device driver. The ethernet packet frames were not being padded with null bytes, which can allow remote attackers to obtain information from previous packets or kernel memory by using malformed packets, This fix protects against the Common Vulnerabilities and Exposures issue number CVE-2003-0001.

A problem was fixed for not being able to access the BMC web gui using HTTPS and IPv6. The fix requires an user step to enable it. After putting on the fix, reset the Lighttpd configuration on the BMC. This reset can be done using ipmitool with the following two commands:

1) ipmitool ... raw 0x30 0x70 0xB7

2) ipmitool ... mc reset cold

A problem was fixed in the BMC web gui for an incorrect VLAN ID ranges in the network configuration.

A problem was fixed for SSH security vulnerabilities that were found running the Qualys tool.

A problem was fixed for a failure that can occur when setting time with hwclock. This failure is triggered by a small time drift that can occur if NTP is active.

A problem was fixed in the BMC web gui for incorrect wording on the AD and LDAP pages.

A security problem was fixed for a password being stored in clear text on the BMC.

OP920.20 / V2.12-20190404 / BMC 2.06

04/18/19

Impact: Data Severity: HIPER

New features and functions

Added support in Redfish for configuring RADIUS (Remote Authentication Dial In User Service), a network protocol for remote user authentication and accounting. It is implemented under redfish/v1/Managers/1/RADIUS. Method supported: Get/Patch. [PATCH]: "RadiusEnabled", "RadiusServerIP", "RadiusPortNumber", "RadiusSecret".

Added support in Redfish for configuring Syslog.

Added ipmi raw command 0x3a 0x30 to be able to set the Meltdown/Spectre risk level to 0, 1, or 2. The default is risk level 0 to provide full mitigation but slowest performance.

Here are the risk levels:

Risk Level 0 = "Speculative execution controls to mitigate user-to-kernel and user-to-user side-channel attacks"

Risk Level 1 = " Speculative execution controls to mitigate user-to-kernel side-channel attacks"

Risk Level 2 = "Speculative execution fully enabled"

More information on these levels can be found at https://www.ibm.com/support/knowledgecenter/en/POWER9/p9hby/p9hby_speculative_execution_control.htm?pos=2.

After the risk level setting is changed, the host needs to be powered off and back on again to be running at the new risk level.

The Spectre/Meltdown fixes were included in the OP920.00 GA release and have the Common Vulnerabilities and Exposures issue numbers CVE-2017-5715, CVE-2017-5753 and CVE-2017-5754.

Added support in the SNMP client to allow connections to V2 and V3 servers to be running at the same.

Added support for Active Directory to allow the BMC to make connections to LDAP\AD servers.

System firmware changes that affect all systems

HIPER/Pervasive: A problem was fixed where, under certain conditions, a Power Management Reset (PM Reset) event may result in undetected data corruption. PM Resets occur under various scenarios such as power management mode changes between Dynamic Performance and Maximum Performance, power management controller recovery procedures, or system boot.

A problem was fixed for a rare Nest Memory Management Unit (NMMU) hang calling out processor hardware incorrectly, masking the real cause of the problem which was an NPU failure. The incorrect error messages take this form on the system:

3 | FQPSPPU0093G | 2018-10-01 01:25:40 | Yes | Warning | CPU 1 has exceeded a correctable error threshold

4 | FQPSPPU0093G | 2018-10-01 03:20:55 | Yes | Warning | CPU 0 has exceeded a correctable error threshold

5 | FQPSPAA0008M | 2018-10-01 04:35:40 | Yes | Critical | Hostboot procedure callout

A problem was fixed for an incorrect Redfish access privilege to make it consistent with the BMC gui webpage.

A problem was fixed for Redfish namespace PhysicalContext.v1_3_0 not being found in schema PhysicalContext_v1.xml.

A problem was fixed for Redfish crashing if an attempt is made to convert a string to an integer when the string does not represent a number.

A problem was fixed for the host console losing data.

A problem was fixed for slow SOL console response for long-running commands.

OP920.10 / V2.10-20190208 / BMC 2.04

02/14/19

Impact: Data Severity: HIPER

New features and functions

Redfish support was extended to version 1.6.0 and the FanMode API was added.

Support for the UART3 and UART4 was disabled in the Linux kernel on the BMC.

The BMC GUI was enhanced to show both the PNOR version and the build date.

Support was added for a new BMC gui page to control the power capping of the system.

Support was added to the BMC for new PNOR version partition that has a 4k signed header.

System firmware changes that affect all systems

HIPER/Non-Pervasive: Fixes included to address potential scenarios that could result in undetected data corruption.

A security problem was fixed to prevent host programs from being able to corrupt the BMC using the internal bridges between the host and BMC. The Common Vulnerabilities and Exposures issue number is CVE-2019-6260.

A security problem was fixed to prevent a buffer overflow when loading the boot image that could cause firmware corruption. The firmware mitigation adds additional checking of the initial boot firmware image's load size and terminates the boot if the size is too big. The Common Vulnerabilities and Exposures issue number is CVE-2018-1992.

A problem was fixed for an intermittent IPL failure with BC131705 and BC8A1703 logged with a processor core called out. This is a rare error and does not have a real hardware fault, so the processor core can be unguarded and used again on the next IPL.

A problem was fixed on the BMC for an incorrect "LanDrvinit fails to initial" message. This is not a true error and the message can be ignored.

A problem was fixed for an intermittent opal-prd crash that can happen on the host OS. The fault signature is: " opal-prd[2864]: unhandled signal 11 at 0000000000029320 nip 00000 00102012830 lr 0000000102016890 code 1"

A problem was fixed for a PCI Host Bridge (PHB) configuration write error that caused the incorrect PCIe device to be frozen. The fault will be attributed to the last device to have a memory-mapped I/O operation (MMIO). With this fix, the freeze action for PHB configuration write errors is disabled in order to not impact functional hardware

A problem was fixed for diagnostic code trying to read sensor values for PCI Host Bridge (PHB) entries that are unused, which causes debug output to have incorrect values for the unused entries. With the fix, only the used entries are processed by the diagnostic code.

A problem was fixed for a IPL loop/hang with a fatal MCE exception log caused by a probe of a failed PCI Host Bridge (PHB) that had been guarded. This is an infrequent error because it requires a PHB to have previously failed. The exception log has the following format:

Fatal MCE at 000000003006ecd4 .probe_phb4+0x570

CFAR : 00000000300b98a0

<snip>

Aborting!

CPU 0018 Backtrace:

S: 0000000031cc37e0 R: 000000003001a51c ._abort+0x4c

S: 0000000031cc3860 R: 0000000030028170 .exception_entry+0x180

S: 0000000031cc3a40 R: 0000000000001f10 *

S: 0000000031cc3c20 R: 000000003006ecb0 .probe_phb4+0x54c

S: 0000000031cc3e30 R: 0000000030014ca4 .main_cpu_entry+0x5b0

S: 0000000031cc3f00 R: 0000000030002700 boot_entry+0x1b8

A problem was fixed for memory Over-Temperature (OT) throttling not occurring when a DIMM reaches the throttle temperature. Although the frequency to the memory DIMMs is not reduced, the fan speeds do increase to provide more cooling for the DIMMs.

A problem was fixed for error logs occurring on the IPL following a DIMM error recovery. These logs, related to failed memory scrubbing, have the following "Signature Description": "mba(n0p15c1) () ERROR: command complete analysis failed". These error logs do not indicate a hardware problem and may be ignored.

A problem was fixed for system termination for a re-IPL with power on. The system can be recovered by powering off and then IPLing. This problem occurs infrequently and can be avoided by powering off the system between IPLs.

A problem was fixed for certain system boot failures not propagating to the BMC before the boot firmware shuts down. Some details of the error log may still appear in the console output trace, but the details will not be available with the BMC queries. This problem is timing dependent and intermittently possible depending on the timing of the shutdown path. However, immediate shutdowns exacerbate the problem and increase the chance it can occur.

A problem was fixed for a BMC web gui freeze condition when an error event occurs on the backplane.

A problem was fixed for random "????" characters displayed on the SOL Console during the skiboot boot.

A problem was fixed for Redfish-Service-Validator detected errors in the following two Redfish APIs: "/redfish/v1/Managers/BMC/LogServices/Log/Entries/[ID]" and "/redfish/v1/Chassis/Planar/Assembly".

A problem was fixed for an IPv4 address change not persisting after a BMC reboot. This error can occur if the last octet of the IP address is reduced in characters by the IP address change. For the case where this was observed, the IP address was changed fro 50.6.36.100 to 50.6.36.1. But after the BMC reboot, the IP address again had two trailing zeros on IP as the IP address had reverted to 50.6.36.100.

A problem was fixed for a skiboot hang that could occur rarely for an i2C request if the i2c bus is in error or locked by the On-Chip Controller (OCC).

A problem was fixed for "Unexpected TCE size" error messages when Linux tried the default P9 PHB4 pages size and used the unsupported 2M and 1G page sizes. The TCE page size property is now set correctly with 4K/64K/16M and 256M supported.

Support was added to recognize a port parameter in the URL path for the Preboot eXecution Environment (PXE) in the ethernet adapters. Without the fix, there could be PXE discovery failures if a port was specified in the URL for the PXE.

A problem was fixed for an intermittent rare processor core lock failure that is not a real hardware problem. The erroneous failure looks like this in the logs:

LOCK ERROR: Releasing lock we don't hold depth @0x30493d20 (state: 0x0000000000000001)

[13836.000173140,0] Aborting!

CPU 0000 Backtrace:

S: 0000000031c03930 R: 000000003001d840 ._abort+0x60

S: 0000000031c039c0 R: 000000003001a0c4 .lock_error+0x64

S: 0000000031c03a50 R: 0000000030019c70 .unlock+0x54

S: 0000000031c03af0 R: 000000003001a040 .drop_my_locks+0xf4

OP920.02 / V1.16-20180531/ BMC 1.27

10/15/2018

Impact: Availability Severity: SPE

System firmware changes that affect all systems

A problem was fixed for a false over-temperature reading on the AOC SAS/SATA adapter due to an incorrect low 65C maximum temperature threshold. With the fix, the maximum operating temperature for the device is set to 100C as the upper threshold.

OP920.01 / V1.16-20180531/ BMC 1.23

06/13/2018

Impact: Availability Severity: SPE

System firmware changes that affect all systems

A problem was fixed for a failure in DDR4 RCD (Register Clock Driver) memory initialization that causes half of the DIMM memory to be unusable after an IPL. This is an intermittent problem where the memory can sometimes be recovered by doing another IPL. The error is not a hardware problem with the DIMM but it is an error in the initialization sequence needed get the DIMM ready for normal operations.

A problem was fixed for a failure to IPL if there are DIMM failures that require DIMM chips to be guarded. During the memory reconfiguration, the system would reboot itself before the reconfiguration could complete. The memory failure would persist into the next IPL attempt, with the result that the system might not be able to IPL with bad memory DIMMs. This problem is more prevalent when there are errors in the larger DIMM modules, such as the 128 GB chips, as it takes longer to reconfigure the ranks of the larger memory chips. There is a possibility that an IPL watchdog timeout and system reboot can occur if a rank of memory takes over two minutes to reconfigure to guard out the bad memory.

A problem was fixed for a failure to isolate to an errant FRU for a system checkstop. This is an intermittent error related to the On-Chip Controller (OCC) not waiting long enough to collect the failure information for a checkstop that occurs on a busy system. When this error happens, it prevents checkstop diagnosis procedures from identifying the cause of the checkstop fault. For this error, no active error bits are found and the checkstop analysis failure error log is mapped to a SEL which directs the customer to contact support as shown below:

1 | 05/17/2018 | 20:49:27 | System Firmware Progress Boot Progress | Motherboard initialization () | Asserted

2 | 05/17/2018 | 20:50:13 | System Firmware Progress Boot Progress | System boot initiated () | Asserted

3 | 05/18/2018 | 09:27:07 | OEM record c0 | 040020 | ceff6fffffff ==> Checkstop Signal , check other serviceable SELs and resolve them

4 | 05/18/2018 | 09:27:44 | System Firmware Progress Boot Progress | Motherboard initialization () | Asserted

5 | 05/18/2018 | 09:27:56 | OEM record df | 040020 | 12046faa0000

6 | 05/18/2018 | 09:28:05 | OEM record de | 000000 | 100000000005 ==> Procedure callout, decodes to contact next level of suppoort of assistance

7 | 05/18/2018 | 09:28:31 | System Firmware Progress Boot Progress | System boot initiated () | Asserted

The following steps can be used to identify the signature for this problem. Look for the SEL indicating checkstop signal as shown below:

3 | 05/18/2018 | 09:27:07 | OEM record c0 | 040020 | ceff6fffffff ==> Checkstop Signal , check other serviceable SELs and resolve them

If found, look for a "OEM de" SEL that is logged a few minutes after the checsktop signal SEL, decoding into "contact next level of support for assistance":

5 | 05/18/2018 | 09:27:56 | OEM record df | 040020 | 12046faa0000

6 | 05/18/2018 | 09:28:05 | OEM record de | 000000 | 100000000005 ==> Procedure callout, decodes to contact next level of suppoort of assistance

If found, collect plc.pl output and inspect the decoded eSEL (PEL) associated with the checkstop analysis. If the signature description in the PRD log reads "No active error bits found", the checkstop analysis failure is confirmed.

| Reference Code : BC70E550 |

| Hex Words 2 - 5 : 000000E0 00000B00 00000000 00200000 |

| Hex Words 6 - 9 : 000B0004 00000103 BC4ADD02 00000000 |

| |

| ModuleId : 0x0B |

| Reason Code : 0xE550 |

| Code Location : 0x0103 |

| |

| PRD SRC Type : PRD Detected Hardware Indication |

| PRD SRC Class : Software likely caused a hardware error condition,|

| : smaller possibility of a hardware cause. |

| |

| PRD Signature : 0x000B0004 0xBC4ADD02 |

| Signature Description : mcs(n0p1c0) No active error bits found |

OP920.00 / V1.16-20180518

05/25/2018

Impact: New Severity: New

New features and functions for MTMs 9006-12P and 9006-22P:

GA Level

Support for trusted boot and secure boot with TPM2.0 Nuvoton NPCT650ABAWX through an I2C controller.

Dedicated 1 GB IPMI port

Integrated MicroSemi PM8069 SAS/SATA 16-port Internal Storage Controller PCIe3.0 x8 with RAID 0, 1, 5, and 10 support (no write cache)

Integrated Intel XL710 Quad Port 10GBase-T PCIe3.0 x8 UIO built-in LAN (one shared management port)

Supermicro BMC support for Redfish:

Get System/Chassis inventory info

Manage user accounts and privileges

BMC configuration (AD, LDAP, SNMP, SMTP, RADIUS, Fanmode, Mousemode, NTP, Snooping etc.)

BIOS configuration

Boot order change

RAID configuration (For 3108)

Storage Management

Get NIC MAC info (NIC asset info)

BMC/BIOS Firmware updates

Get thermal/power/sensor info

Get system memory info

Get hostname

Launch iKVM/HTML5 using redfish

Update SSL certificate and key

Perform computer system reset

BMC reset

BMC configuration reset

Get health event log/Advanced system event log

Acknowledge warning/critical severity events

Virtual Media ISO image mounting

System firmware changes that affect all systems

HIPER/Pervasive: A firmware change was made to address a rare case where a memory correctable error on POWER9 servers may result in an undetected corruption of data.

4.0 Operating System Information

OS levels supported by the LC921 and LC922 servers:

- Red Hat Enterprise Linux (RHEL) 7.5 little endian (LE) (POWER9), or later

- Ubuntu Server 18.04 LTS

- Virtualization Engine

Feature (#EC16) -OpenPOWER non-virtualized configuration (also known as bare-metal install)

IBM Power LC921 and LC922 servers support Linux which provides a UNIX like implementation across many computer architectures. Linux supports almost all of the Power System I/O and the configurator verifies support on order. For more information about the software that is available on IBM Power Systems, see the Linux on IBM Power Systems website:

http://www.ibm.com/systems/power/software/linux/index.html

4.1 Linux Operating System

The Linux operating system is an open source, cross-platform OS. It is supported on every Power Systems server IBM sells. Linux on Power Systems is the only Linux infrastructure that offers both scale-out and scale-up choices. One supported version of Linux on the IBM Power LC921 and LC922 servers is Ubuntu Server 18.04 LTS for IBM POWER9. For more information about Ubuntu Server for Ubuntu for POWER9 see the following website:

https://wiki.ubuntu.com/ppc64el

Another supported version of Linux on the LC921 and LC922 servers is Red Hat Enterprise Linux 7.5 LE. For additional questions about the availability of this release and supported Power servers, consult the Red Hat Hardware Catalog at https://access.redhat.com/products/red-hat-enterprise-linux/#addl-arch

For more information about Linux on Power, see the Linux on Power developer center at https://developer.ibm.com/linuxonpower/

For information about the features and external devices that are supported by Linux, see this website:

http://www.ibm.com/systems/power/software/linux/index.html

4.2 How to Determine the Level of a Linux Operating System

Use one of the following commands at the Linux command prompt to determine the current Linux level:

•cat /proc/version
•uname -a

The output string from the command will provide the Linux version level.

4.3 How to Determine if the opal-prd (Processor Recovery Diagnostics) package is installed

The opal-prd package on the Linux system collects the OPAL Processor Recovery Diagnostics messages to log file /var/log/syslog. It is recommended that this package be installed if it is not already present as it will help with maintaining the system processors by alerting the users to processor maintenance when needed.

On Red Hat Linux, perform command "rpm -qa | grep -i opal-prd ". The command output indicates the package is installed on your system if the rpm for opal-prd is found and displayed. This package provides a daemon to load and run the OpenPower firmware's Processor Recovery Diagnostics binary. This is responsible for run-time maintenance of Power hardware. If the package is not installed on your system, the following command can be run on Red Hat to install it:

sudo yum update opal-prd

5.0 How to Determine The Currently Installed Firmware Level

The system firmware is a combination of BMC and PNOR firmware levels.

Use the ipmtool "fru" command or the BMC Web GUI FRU option to look at product details of FRU 47.

ipmitool -I lanplus -H <bmc host IP address> -U <ADMIN_userid> -P <ADMIN_password> fru print 47

Use the ipmitool "mc info" command or the BMC Web GUI System tab to look at the BMC level.

ipmitool -I lanplus -H <bmc host IP address> -U <ADMIN_userid> -P <ADMIN_password> mc info|grep "Firmware"

6.0 Downloading the Firmware Package

Follow the instructions on Fix Central. You must read and agree to the license agreement to obtain the firmware packages.

7.0 Installing the Firmware

Note 1: For upgrade firmware from firmware version 1.xx to version 2.xx: There are dependencies between BMC and PNOR. For that reason, you must update BOTH BMC and PNOR while system power is off.

2. Update BMC to BMC 2.04 or later using pUpdate 2.20 (or BMC Web GUI).

3. After BMC firmware updated, the BMC is automatically reset. Wait for BMC to come back on line

3.1. Being able to connect BMC Web GUI or able to ping to BMC is an indication of BMC back on line

4. Check BMC version to ensure the right BMC version is updated

5. You must reset the nginx configuration for Redfish (REST API) support:

a) ipmitool -I lanplus -H bmc_ip_address -U <ADMIN_userid> -P <ADMIN_password> raw 0x30 0x70 0xb7

b) ipmitool -I lanplus -H bmc_ip_address -U <ADMIN_userid> -P <ADMIN_password> mc reset cold

c) Wait for BMC to come back on line.

6. Update PNOR version 2.xx using pUpdate 2.20 or using BMC Web GUI

6.1. The new PNOR version is not available at this time.

7. Once the PNOR update is complete, power on the system to petitboot menu or OS.

8. Once the system reaches to OS, you can check the PNOR version to ensure it is correct.

9. System is ready to be used.

2. Update BMC version 1.xx using pUpdate 2.20 (or BMC Web GUI)

3. After BMC firmware updated, the BMC is automatically reset. Wait for BMC to come back on line

3.1. Being able to connect BMC Web GUI or able to ping to BMC is an indication that BMC is back on line

4. Check the BMC version to ensure the right BMC version is updated