IBM Power Systems Hyperconverged CS Server Firmware
Applies to: CS821 (8005-12N) and CS822 (8005-22N)
This document provides information about the installation of Licensed Machine or Licensed Internal Code, which is sometimes referred to generically as microcode or firmware.
This package provides firmware for the IBM Power Hyperconverged Systems powered by Nutanix CS servers of CS821(8005-12N) and CS822 (8005-22N).
The firmware level in this package is:
•PNOR V3.11 and SMC BMC V3.26
There are the following images:
.pnor (Host image)
.bin (BMC image)
pUpdate version 2.20 Utility : pUpdate_220.zip
Details on the package binaries are included in section 3.1.
When downgrading to a BMC version that is older than BMC 3.25, port 80 is not disabled even though the gui states it is.
If the BMC was previously configured to disable port 80 (Web port), take the following steps after the downgrade from BMC 3.25 to confirm the permanent disablement of port 80:
1) Access the BMC UI
2) Navigate to "Configure -> Ports"
3) Re-enable the "Web port" option by checking the box next to it
4) Click "Save"
5) Disable the "Web port" option by un-checking the box next to it
6) Click "Save"
7) Port 80 should now be disabled and in sync with the setting displayed in the BMC user interface.
For all firmware updates where the starting level is less than OP825.41 (BMC version is V3.16 or older) to a newer level, there is a problem where the system date and time may be incorrect after the firmware update process has completed. To correct this error, the user must manually reconfigure the BMC date/time under the Configuration menu option prior to starting the operating system. Updating time of day is required to ensure system operations are working correctly.
One function that you may see that is not working correctly is the remote console. The connection to the remote console will fail if the time of day is not correct.
Note 1: Before proceeding with an update to this service pack level, update the OS to the latest fix level for its release as there is an important fix for the AST driver that, if not applied, could prevent the system from IPLing after the service pack is installed. For Ubuntu level 16.04.1, there is no fix for the ast driver, so the OS level must be upgraded to an approved OS level (see section “4.0 Operating System Information“) or the AST driver can be disabled using the below command. The disable of the AST driver will cause loss of output to the VGA console, so this must be considered before choosing this option.
Ensure the "modprobe.blacklist=ast" parameter is set in your boot loader (grub).
Note 2:
1. When upgrading firmware from firmware version 1.xx to version 3.xx:, there are dependencies between BMC and PNOR. For that reason, you must update BOTH BMC and PNOR while system power is off.
1.1 Power off the system
1.2 Update BMC version 3.xx using pUpdate 2.20 or using BMC Web GUI.
1.3 After the BMC firmware is updated, the BMC is automatically reset. Wait for the BMC to come back on line.
1.3.1 You can ping the BMC to see if it is active on the network or you connect to the BMC Web GUI.
1.4 Check the BMC version to ensure the correct BMC version is updated.
1.4.1 Check Web GUI display for new BMC version, or
1.4.2 Use the ipmitool to check the BMC version
1.5 Update PNOR version 3.xx using pUpdate 2.20 or using BMC Web GUI.
1.5.1 The new PNOR version is not yet available until a power on is completed.
1.6 Once the PNOR update is complete, power on the system to petitboot menu or OS.
1.6.1 When the system IPL reaches petitboot, SBE validation will be started automatically. This will take about 2 minutes. Please allow the system to complete the SBE validation before attempting to shutdown. If SBE validation fails for any reason, the system power will be turned off by firmware. Follow repair action given in the SEL to correct the problem.
1.6.2 Once the system reaches to OS, you can check the PNOR version to ensure it is correct.
1.7 The system is ready to be used.
2a. Downgrade of firmware from version 3.xx or later to version 1.xx is NOT supported.
2b. BMC 3.23 and newer contains the fix for CVE-2019-11072, which requires change to the web config file. There is no requirement to change the web config file when upgrade to BMC 3.23 or newer as it is done automatically during the firmware update.
If a user needs to downgrade the BMC 3.23 or newer firmware to a level BMC 3.20 or older, the below steps must be performed. Otherwise, the user will not be able to connect to the BMC web interface.
2b.1 Ensure system is powered down
2b.2 Downgrade PNOR (to match target PNOR) but do not IPL ( Do this step if changing the PNOR level)
2b.3 Reset nginx with ipmitool raw command but do not reset BMC
ipmitool -H <<bmcHostname_or_bmcIPaddress>> -I lanplus -U ADMIN -P <<bmc_password>> raw 0x30 0x70 0xB7
2b.4 Downgrade BMC ( it will automatically reset)
2b.5 Power on the system to IPL (finish PNOR install/SBE updates) ( Do this step if PNOR level was changed).
3. Fast Reboot is not supported (fast reboot can be disabled with below command):
3.1 'nvram -p ibm,skiboot --update-config fast-reset=0'
3.2 Fast reboot is disabled by default.
4. After updating BMC firmware to version 3.15 or newer, the current BMC password for "ADMIN" user ID is still in effect until a factory reset is issued to the BMC. Once the factory reset is issued to the BMC, the BMC will reset the password for the "ADMIN" user ID to the default password, "ADMIN", but it will be expired. No remote command will work except the command to change the password. The "ADMIN" user ID's password can be changed with the interfaces listed below. New password strength rules (at least 8 characters with at least one lowercase letter, one uppercase letter and one numeric number) must be followed.
4.1 Web GUI
4.2 SSH
4.3 In-band IPMI command (host user must have root privilege)
This section specifies the "Minimum ipmitool Code Level" required by the System Firmware to perform firmware installations and managing the system. Open Power requires ipmitool level v1.8.15 to execute correctly on the V1.00 firmware.
Verify your ipmitool level on your linux workstation using the following commands:
bash-4.1$ ipmitool -V
ipmitool version 1.8.15
If you need to update or add impitool to your Linux workstation , you can compile ipmitool (current level 1.8.15) for Linux as follows from the Sourceforge:
1.1.1 Download impitool tar from http://sourceforge.net/projects/ipmitool/ to your linux system
1.1.2 Extract tarball on linux system
1.1.3 cd to top-level directory
1.1.4 ./configure
1.1.5 make
1.1.6 ipmitool will be under src/ipmitool
You may also get the ipmitool package directly from your workstation linux packages such as Ubuntu 14.04.3:
sudo apt-get install ipmitool
The BMC Web GUI is a web-based application that works within a browser. Supported browser levels are shown below with Chrome being the preferred browser:
•Google Chrome Version 46.0.2490.71 m
•Mozilla Firefox version 41.0.3
For specific fix level information on key components of IBM Power Systems servers and Linux operating systems, please refer to the documentation in the IBM Knowledge Center.
Here are the links for the Hyperconverged CS 8005 servers powered by Nutanix:
http://www.ibm.com/support/knowledgecenter/POWER8/p8hdx/8005_12n_landing.htm
http://www.ibm.com/support/knowledgecenter/POWER8/p8hdx/8005_22n_landing.htm.
Due to an incompatibility with the VGA driver, the Nutanix’s AHV OS local console is only available through IPMI.
If the starting level of the firmware is less than OP825.41 prior to the update to this level of firmware, then manually setting the TOD after installation is required. A message will be presented to the user after the update as a reminder. The TOD must be set prior to initiating the boot of the host OS.
Downgrading firmware from any given release level to an earlier release level is not recommended.
If you feel that it is necessary to downgrade the firmware on your system to an earlier release level, please contact your next level of support.
Concurrent Firmware Updates not available for CS servers.
Concurrent system firmware update is not supported on CS 8005 servers.
If the system hangs when doing a reboot from the host OS, a disable of NVRAM fast-reset may resolve this problem. Run the following command on the BMC:
nvram -p ibm.skiboot -update-config fast-reset=0
The Nutanix node re-image will fail if levels of SMCIPMITool less than 2.22.1 are used.
These instructions are for recovering the ADMIN user ID that has expired for the default “ADMIN” password because the system is new or has been factory reset. These instructions assume that the BMC is on the network. The Web GUI, secured shell (SSH), or ipmitool from the host may be used to change the expired password as shown below:
1. Web GUI:
1.1 When a user is logging into BMC Web GUI with the "ADMIN" user ID after BMC factory reset, the user must use default password (see above) regardless of what the previous "ADMIN" user ID's password was.
1.2. Once successfully logged in with the "ADMIN" user ID, the BMC Web GUI will force the user to change the password for the "ADMIN" user ID. The BMC Web GUI will log out after successfully changing the password.
1.3. Now the user can log in to the BMC using "ADMIN" user ID with the new password.
2. SSH:
2.1. When a user is logging into a BMC using ssh with "ADMIN" user ID after the BMC factory reset, the user must use the default password (see above) regardless of what the previous "ADMIN" user ID's password was.
2.2 Once successfully logged in with "ADMIN" user ID, the ssh will force the user to change the BMC password for the "ADMIN" user ID. Follow the instructions carefully. ssh will log out after successfully changing the password.
2.3. Now the user can log in to the BMC using the "ADMIN" user ID with the new password.
3. In-band IPMI command
3.1. Power on the host system to OS.
3.2. Log in to the OS as root.
3.3. Change BMC "ADMIN" user ID's password:
3.3.1. Command: ipmitool user set password 2 newPW
3.3.2. newPW is the desired new password.
The V2.00 and later firmware for CS Servers is now less verbose during boot - only error-level messages are printed during Petitboot bootloader initialization. This means that there will be fewer messages printed as the system boots.
Additionally, the Petitboot user interface is started earlier in the boot process. This means that you will be presented with the user interface sooner, but it may still take time, potentially up to 30 seconds, for the user interface to be populated with boot options as storage and network hardware is being initialized. During this time, Petitboot will show the status message "Info: Waiting for device discovery". When Petitboot device discovery is completed, the following status message will be shown "Info: Connected to pb-discover!".
Use the following examples as a reference to determine whether your installation will be concurrent or disruptive.
For the CS 8005 server systems, the installation of system firmware is always disruptive.
The xxx.pnor file updates the primary side of the PNOR. The yyy.bin updates the primary side of the BMC only. The golden sides are unchanged.
The pUpdate zip file contains the Linux powerpc, x86(32 bit), x86(64 bit), and pUpdate user’s guide. The version of the tool selected needs to match the hardware where the tool is running.
Filename | Size | Checksum |
P8DTU20200610_IBM_sign.pnor | 33554560 | c5a4b94094a6d92a2f35b74eb0b6fe0d |
SMT_P8_326.bin | 33554432 | d59f58112dc62d46159a86953f6a289c |
pUpdate_220.zip | 91832 | 745f57ea76b2bc641dc8005bf960ecb3 |
Note: The Checksum can be found by running the Linux/Unix/AIX md5sum command against the file (all 32 characters of the checksum are listed), ie: md5sum xxx.pnor.
After a successful update to this firmware level, the PNOR components and BMC should be at the following levels. The ipmitool "fru" command can be used to display FRU ID 47 and the ipmitool "mc info" command can be used to display the BMC level.
Note: FRU information for the PNOR level does not show the updated levels via the fru command until the system has been booted once at the updated level.
PNOR firmware levels from FRU ID 47 inventory list for driver:
FRU Device Description : System Firmware (ID 47)
System Firmware:
open-power-SUPERMICRO-P8DTU-V3.11-20200610
op-build-8593887
buildroot-2019.02.3-21-gb1a7f41
skiboot-v6.3.5
hostboot-p8-c893515-p0ae181e
occ-p8-a2856b7
linux-5.1.15-openpower1-p84b2ad1
petitboot-v1.10.4-pb97a401
BMC Level:
Display BMC firmware level using the "ipmitool mc info | grep Firmware" command:
Firmware Revision : 3.26
OP825 | |
V3.11 with SMC BMC V3.26 (OP825.52)
09/01/2021
| Impact: Function Severity: SPE
System firmware changes that affect all systems
A problem was fixed for an incorrect threshold value for the monitoring of the memory buffer temperature. Whenever the temperature of the memory buffer exceeds a pre-defined threshold, system cooling fan speeds are increased to prevent the system from overheating.
A problem was fixed for a Remote Authentication Dial-in User Service (RADIUS) time out. The RADIUS time-out value has been increased to fix the problem. RADIUS is a protocol for carrying authentication, authorization, and configuration information.
|
V3.11 with SMC BMC V3.25 (OP825.51)
07/14/2021 | Impact: Security Severity: SPE
New features and functions
Support was added for enabling and disabling the http and https ports for the web server, and to set the associated port addresses for the web service. The https, https, and redirect SSL settings are synchronize in the lighttpd configuration. If preservation is unchecked during upgrading to this version of firmware, the default setting of http, https, and SSL redirect is used.
System firmware changes that affect all systems
A problem was fixed for the default value for the Virtual Media service port being missing. The default is port 6230.
A problem was fixed for security vulnerabilities in the Virtual Media function. BMC/IPMI Virtual Media is a feature of the Virtual Console that enables users to attach a CD/DVD image to the server as a virtual CD/DVD drive. These vulnerabilities include plain text authentication, weak encryption, and authentication bypass within the Virtual Media capabilities. These vulnerabilities have been documented with CVE-2019-16649 and CVE-2019-16650.
|
V3.11 with SMC BMC V3.23 (OP825.50)
04/21/2021 | Impact: Security Severity: SPE
New features and functions
Support was added for Chain Certification.
Support was added for the following grub2 enhancements for petitboot: 1) 'source' command is now supported. 2) UUID and label are now supported in the 'search' command.
System firmware changes that affect all systems
A problem was fixed for security vulnerability CVE-2019-11072. With this vulnerability, a signed integer overflow can be triggered, which could allow remote attackers to cause a denial of service (BMC crash) or possibly have unspecified other impacts via a malicious HTTP GET request. With this fix, a change is made to the BMC web configuration file. There is no need to manually change the web configuration file when upgrading to BMC 3.23 as it is done automatically during the firmware update. However, if a user needs to downgrade the BMC firmware to a level less than BMC 3.23, the below steps must be performed. Otherwise, the user will not be able to connect to the BMC web interface. 1. Ensure system is powered down 2. Downgrade PNOR (to match target PNOR) but do not IPL (do this step if changing the PNOR level). 3. Reset nginx with ipmitool raw command but do not reset BMC ipmitool -H <<bmcHostname_or_bmcIPaddress>> -I lanplus -U ADMIN -P <<bmc_password>> raw 0x30 0x70 0xB7 4. Downgrade BMC ( it will automatically reset) 5. Power on the system to IPL (finish PNOR install/SBE updates) ( Do this step if PNOR level was changed).
A problem was fixed for RHEL8 boot failures when using GPT Disklabels caused by dependencies on unsupported features in petitboot GRUB2. Functionality in the GRUB2 parser was extended to allow RHEL8 to boot correctly.
A problem was fixed for security vulnerability CVE-2007-1858. With this vulnerability, the default SSL cipher configuration used certain insecure ciphers, including the anonymous cipher, which could allow remote attackers to obtain sensitive information or have other, unspecified impacts.
A problem was fixed for security vulnerability CVE-2016-2183. With this vulnerability, the DES and Triple DES ciphers, as used in the TLS, SSH, and IPSec protocols and other protocols and products, have a birthday bound of approximately four billion blocks, which makes it easier for remote attackers to obtain clear text data via a birthday attack against a long-duration encrypted session,
A problem was fixed for an unresponsive system after an unscheduled reset of the BMC causes the FSI bus to become locked. A restart of the system via BMC or power button, is necessary to recover.
A problem with a slow transfer of data using iKVM Virtual Storage was fixed to improve the performance. The slow down was caused by a change to use a secure tunnel for the data. With the fix, the default secure channel is providing slightly better performance throughput. If further performance is needed, the secure channel can be temporarily disabled using the "Virtual Media->Secure Encryption" option. A problem was fixed for a BMC gui "Alerts Modify" to not drop the first character of the e-mail address when re-visiting the page.
A problem was fixed for a web security issue against Cross-Site Request Forgery with an enhanced fix over the previous delivery for OP825.41.
A problem was fixed for an unresponsive system after a reset of the BMC by dropping console write data if the BMC becomes unresponsive. This prevents a deadlock condition.
A problem was fixed for intermittent ipmi aborts caused by the use of invalid pointers that had been freed.
A problem was fixed for truncation of multi-word strings in the petitboot user interface. When this failure happens, the first word of the string is displayed only, stopping at the first "blank" of the string. Plugin names and vendor variables have been affected by this.
|
V3.09 with SMC BMC V3.17 (OP825.41)
07/06/2020
| Impact: Security Severity: SPE
System firmware changes that affect all systems with user action needed
A problem was fixed for the system time moving ahead one day following a month or year change and a power off of the system. Important:: This fix does not update the date and time to correct values if these are wrong in the RTC. The user must manually reconfigure the BMC date/time under the Configuration menu option prior to starting the operating system.
New features and functions
Improved BMC password policy. For the BMC, the ADMIN password must be set on first use for newly manufactured systems and after a factory reset of the system. This policy change helps to enforce the BMC is not left in a state with a well known password. The user can change from the default password to a new password using the interfaces below: 1. Web GUI 2. ssh (secured shell) 3. In-band IPMI from the host with a user ID that has root privilege. Examples of using these interfaces for expired password recovery can be found in section "2.3 Recovering from an expired password" of the readme.
Support was added to the SMASH shell of ssh to allow changing of the ADMIN user ID password using the SMASH CLP command of "set password" while in the /map1/account002 directory.
Support was added to BMC to have a minimum password strength configured for all interfaces such that it must be at least 8 characters long and must include at least one upper case letter, one lower case letter, and one numeric digit.
In response to recently reported security vulnerabilities, this firmware update is being released to address Common Vulnerabilities and Exposures issue numbers CVE-2017-5715, CVE-2017-5753 and CVE-2017-5754. Operating System updates are required in conjunction with this FW level for CVE-2017-5753 and CVE-2017-5754. This replaces an earlier firmware update for the same problem which was found to not be effective.
Support was added for node re-imaging using SMCIPMITool 2.22.1 build 190923 or above. If lower levels of SMCIPMITool are used, the node re-imaging will fail.
The BMC web gui was enhanced so that the LDAP port can be set regardless of the SSL settings.
The BMC web gui for the Event Log page was enhanced to show event severity and acknowledgment status.
Support was added for a Self-Boot Engine (SBE) validation during the IPL to verify that the firmware images are the shipped versions.
Added BMC support to be able to detect Self Boot Engine (SBE) SEEPROM corruption
Added BMC support for new PNOR version partition that has a 4k signed header.
Security was enhanced for stunnel by allowing SSL Medium Strength and Anonymous Cipher Suites to be disabled. A reset of the stunnel configuration is required to do the disable using the following ipmitool commands one time: 1) ipmitool ... raw 0x30 0x70 0xB9 2) ipmitool ... mc reset cold
Added support in the SNMP client to allow connections to V2 and V3 servers to be running at the same.
Support was added for the BMC web gui to be able to enable and disable IPMI over LAN.
Added support for Active Directory to allow the BMC to make connections to LDAP\AD servers.
The BMC web gui was enhanced to display the IP address of the BMC.
For the IPMI DCMI configuration, the DCMI Discovery Configuratiuon (Option 12) is now enabled by default.
Sensor polling was enhanced to continue to work even after On-Chip Controller (OCC) goes into an error state.
Support for VUART/MBOX/FSI was enabled.
The BMC web gui was enhanced to be able to change the community string for the SNMP PET trap.
Support was added for a new BMC gui page to control the power capping of the system.
The BMC web gui was enhanced to show the PNOR version and build date. If the PNOR version is not found, only the PNOR build date is shown.
Support for tftp was added for BMC brick recovery (the old method of recovery has been deprecated).
Support has been removed from XIVE interrupt controller for the store EOI operation. Hardware has limitations which would require a sync after each store EOI to make sure the MMIO operations that change the ESB state are ordered. This would be performance prohibitive and the PCI Host Bridges (PHBs) do not support the synchronization.
Support was added to recognize a port parameter in the URL path for the Preboot eXecution Environment (PXE) in the ethernet adapters. Without the fix, there could be PXE discovery failures if a port was specified in the URL for the PXE.
System firmware changes that affect all systems
The following web security problems were addresses for the BMC: 1) Authentication bypass vulnerability that could allow a network attacker to become an admin user. 2) Persistent (or stored) XSS vulnerability that can occur when the data provided by the attacker is saved by the server, and then permanently displayed on "normal" pages returned to other users in the course of regular browsing, without proper HTML escaping. 3) Cross-Site Request Forgery 4) HTML injection 5) Authenticated code injection as root via command injection
A problem was fixed for incorrect eSEL index in the error logs.
A problem was fixed for a Qualys security scan on the BMC reporting security threat QID 11827 “HTTP Security Header Not Detected” on port 443. The following HTTP security header was not detected in the HTTP response: Strict-Transport-Security (HSTS): This response header is a security feature that lets a web site tell browsers that it should only be communicated with using HTTPS instead of HTTP protocol.
A problem was fixed for SMCIPMITool being unable to mount virtual media (problem was introduced in BMC V2.08 with a security fix for media vulnerabilities). This problem affected customers that use SMCIPMITool to reimage nodes, as one example of problem impact. Along with the BMC V3.16 level, the SMCIPMITool version 2.22.1 build 109023 or later is needed to fix the issue. The new versions of the SMCIPMITool are backward compatible with the previous versions of the tool.
A problem was fixed for an intermittent disable of the field for the IPv6 address in the BMC web gui.
A problem was fixed for fast-reboot inadvertently always being enabled when it should have been disabled for the Power8 systems. This caused intermittent host reboot failures. A re-IPL of the system can be done to fix the reboot error if it happens. With the fix, the fast-reboot setting is disabled, and the reboots of the host OS no longer have the intermittent failures.
A problem was fixed for an incorrect interpretation for an OEM SEL.
A problem was fixed in the BMC web gui for an incorrect VLAN ID ranges in the network configuration.
A problem was fixed for SSH security vulnerabilities that were found running the Qualys tool.
A problem was fixed for a failure that can occur when setting time with hwclock. This failure is triggered by a small time drift that can occur if NTP is active.
A problem was fixed in the BMC web gui for incorrect wording on the AD and LDAP pages.
A problem was fixed for not being able to access the BMC web gui using HTTPS and IPv6. The fix requires a user step to enable it. After putting on the fix, reset the Lighttpd configuration on the BMC. This reset can be done using ipmitool with the following two commands: 1) ipmitool ... raw 0x30 0x70 0xB7 2) ipmitool ... mc reset cold
A security problem was fixed for a password being stored in clear text on the BMC.
A security problem was fixed for the NTP service running on the BMC allowing queries of the internal NTP variables. This problem was reported with Qualsys QID#38293.
A security problem was fixed for multiple Virtual Media vulnerabilities reported by Eclypsium.
A problem was fixed for cleaning up cache memory when a GPU is in reset to make more cache available for the reset of the system.
A problem was fixed for opal-prd messages being truncated and causing the following error message: "opal-prd: FW: error reading from firmware: alloc 32 rc -1: Invalid argument". This intermittent problem is caused by the allocated buffer for the opal-prd message being too small.
A problem was fixed for possible OPAL partition cosmetic damage when using the "opal-gard clear" command. With the problem, the guarded FRUs are cleared correctly but the adjoining partition will also have some bytes cleared at the beginning of the partition. The inadvertently cleared bytes in the partition have not caused an issue for the system.
A security problem was fixed to prevent host programs from being able to corrupt the BMC using the internal software bridges between the host and BMC. The Common Vulnerabilities and Exposures issue number is CVE-2019-6260.
A security problem was fixed to detect and prevent Self Boot Engine (SBE) SEEPROM corruption. The Common Vulnerabilities and Exposures issue number is CVE-2018-8931.
A security problem was fixed to prevent a firmware update causing an unsigned image to be activated. The Common Vulnerabilities and Exposures issue number is CVE-2018-13787.
A security problem was fixed for the BMC ethernet Network Interface Card (NIC) device driver. The ethernet packet frames were not being padded with null bytes, which can allow remote attackers to obtain information from previous packets or kernel memory by using malformed packets, This fix protects against the Common Vulnerabilities and Exposures issue number CVE-2003-0001.
A problem was fixed for system hangs for early fails that occur in Hostboot. With the fix, the early fails are handled and recovery attempted to allow the IPL to succeed.
A problem was fixed for CGI aborts when uploading configurations using HTTP.
A problem was fixed for SSL certificate checks that were incorrectly failing on the check of the private key.
A problem was fixed for BMC remote console not conforming to security standards by not being digitally signed. The function has been updated to be cryptographically signed,
A problem was fixed for the host console losing data.
A problem was fixed for slow SOL console response for long-running commands.
A problem was fixed for an intermittent failure to IPL.
A problem was fixed for an intermittent system hang during IPL with "Power Status Error" in the SEL.
A problem was fixed for the BMC not showing all the sensor readings (except for OCC Active sensors which are always present) after an IPL.
A problem was fixed for an incorrect power cap setting when BMC sometimes incorrectly did not use the power cap setting from the host On-Chip Controller (OCC).
A problem was fixed for a failed SBE side not triggering a retry boot from the other SBE side.
A problem was fixed for an overrun of serial data from the host side to the BMC that caused loss of output character on the IPMI SOL console.
A problem was fixed for a "mc reset cold" being allowed during a IPL, causing a reboot of the BMC and an IPL failure. With the fix, the BMC cold reset is not allowed during the system IPL.
A problem was fixed for the BMC gui not being able to show all the System Event Log entries . "BMC gui -> Maintenance -> System Event Log " failed to display entire log with no scroll bar present to see new log entries.
A problem was fixed for pUpdate having troubles recovering from retry attempts and not be able to complete the firmware updates in a timely manner.
A problem was fixed for a IPv4 address change not persisting after a BMC reboot . This error can occur if the last octet of the IP address is reduced in characters by the IP address change. For the case where this was observed, the IP address was changed fro 50.6.36.100 to 50.6.36.1. But after the BMC reboot, the IP address again had two trailing zeros on IP as the IP address had reverted to 50.6.36.100.
A problem was fixed for an abnormal Java SOL display in Petitboot.
A problem was fixed for a missing failed operation description in the Session Audit SEL.
A problem was fixed in the BMC weg gui for logical volume color status.
A problem was fixed in the BMC web gui for not being able to mount an iso volume.
A problem was fixed for a BMC web gui freeze condition when an error event occurs on the backplane.
A problem was fixed for a "LanDrvinit fails to initial" message that was logging during the boot. There was no error in the boot as a wrong status value was returned to cause the message.
A problem was fixed for the power capping range allowed for the user. OCC provides two limits for minimum powercap. One being hard powercap minimum which is guaranteed by OCC and the other one is a soft powercap minimum which is lesser than hard-min and may or may not be asserted due to various power-thermal reasons. So to allow the users to access the entire powercap range, this fix exports soft powercap minimum as the “powercap-min” DT property. And it also adds a new DT property called “powercap-hard-min” to export the hard-min powercap limit.
A problem was fixed for lost output on the console when the OS is stopping or rebooting. With the fix, the console output is always flushed before stopping the system.
A problem was fixed for the AST VGA device which could sometimes fail to initialize when the vendor ID for the device was parsed incorrectly.
A problem was fixed for a system hang that could occur while printing with system debug options and having a active user on the console.
A problem was fixed for an intermittent opal-prd crash that can happen on the host OS. This is the fault signature: " opal-prd[2864]: unhandled signal 11 at 0000000000029320 nip 00000 00102012830 lr 0000000102016890 code 1"
A problem was fixed for diagnostic code trying to read sensor values for PCI Host Bridge (PHB) entries that are unused, which causes debug output to have incorrect values for the unused entries. With the fix, only the used entries are processed by the diagnostic code.
A problem was fixed for Petitboot exiting to the shell with xCAT genesis in the menu when trying to do a network boot. Petitboot was timing out when trying to access the ftpserver but it was not doing the network re-queries necessary for a proper retry. If this error happens on a system, it can be made to boot with the following two steps: 1) Type the word "exit" and press enter key. This brings it back to petitboot menu. 2) Press the enter key again to start the boot of the xCAT image.
A problem has been fixed for a slow start up of a process that can occur when the system had been previously in an idle state.
A problem has been fixed for a TOD error that can cause a soft lockup of the kernel. A 'soft lockup' is defined as a bug that causes the kernel to loop in kernel mode for more than 20 seconds, without giving other tasks a chance to run. The current stack trace is displayed upon detection and, by default, the system will stay locked up.
A problem has been fixed to add part and serial numbers to the processors when accessed through the device tree.
A problem has been fixed to make the OS aware of the DARN random number generator at 0x00200000 PPC_FEATURE2_DARN) and the SCV syscall at 0x00100000 (PPC_FEATURE2_SCV). Without this fix, these service constants are not defined in the OS userspace.
A problem was fixed for Coherent Accelerator Processor Proxy (CAPP) mode for the PCI Host Bridge (PHB) to improve DMA write performance by enabling channel tag streaming for the PHB. With this enabled, the DMA write does not have to wait for a response before sending a new write command on the bus.
A problem was fixed for the Open-Power Flash tool "pflash" failing with a blocklevel_smart_erase error during a pflash. This problem is infrequent and is triggered if pflash detects a smart erase fits entirely within one erase block.
A problem was fixed in the Petitboot user interface to handle cursor mode arrow keys for the VT100 'application' cursor to prevent mis-interpreting an arrow key as an escape key in some situations. For more information on the VT100 cursor keys, see http://www.tldp.org/HOWTO/Keyboard-and-Console-HOWTO-21.html.
A problem was fixed in the Petitboot user interface to cancel the autoboot if the user has exited the Petitboot user interface. This prevents the user dropping to the shell and then having the machine boot on them instead of waiting until the user is ready for the boot.
A problem was fixed in the Petitboot parsing of manually-specified configuration files that caused the parser to create file paths relative to the downloaded file's path, not the original remote path.
A problem was fixed for a flood of OPAL error messages that can occur for a processor fault. The message "CPU ATTEMPT TO RE-ENTER FIRMWARE" appears as a large group of messages and precede the relevant error messages for the processor fault. A reboot of the system is needed to recover from this error.
A problem was fixed for a skiboot hang that could occur rarely for a i2C request if the i2c bus is in error or locked by the On-Chip Controller (OCC).
A problem was fixed for an OS reboot after a shutdown that intermittently fails after the shutdown. This can happen if the BMC is not ready to receive commands. With the fix, the messages to the BMC are validated and retried as needed. To recover from this error, the system can be rebooted from the BMC interface.
A problem was fixed for a kernel hard lock up that could occur if IPMI synchronous messages were sent from the OS to BMC while the BMC was rebooting. For these type of messages, a processor thread remains waiting in OPAL until a response is returned from the BMC.
|
V2.20 with SMC BMC V1.27 (OP825.20) 03/07/18 | Impact: Availability Severity: SPE
The PNOR for this service pack has the firmware fixes for Spectre/Meltdown enabled (CVE-2017-5715, CVE-2017-5753, and CVE-2017-5754). If you need to run without the Spectre/Meltdown mitigation enabled, the 20170512 PNOR provided in the V2.10 level on Fix Central can be installed in conjunction with the BMC V1.27 level from this service pack.
New features and functions
Support was added for the user to be able to create a BMC group name via LDAP. This provides a BMC login privilege by group name.
Support was added for a new alert message that occurs on the BMC web gui when a disk, power supply, or a fan is unplugged. When an event is logged against one of the associated sensors, the alert message is sent.
Support for a 2 MB IPMI SOL console data capture and download.
Support for "Product Extra Information" for FRU reading on the BMC GUI web.
System firmware changes that affect all systems
A problem was fixed for RHEL7.3 GUI mode having a large square cursor via the BMC iKVM console. This over-sized cursor can interfere with normal operations on the screen by blocking the view of task options on pull-down menus.
A problem was fixed for a duplicate sensor id for occ_active and boot_count which shared a device id of "0x09". This caused one of the sensors to be absent from the sensor device tree and also caused an OPAL error log: "[ 84.176082494,3] DT: dt_attach_root failed, duplicate sensor@9". There was no other adverse impact for this problem other than the missing sensor information. With the fix, the boot count sensor has a new device code of "0x0A".
A problem was fixed for an IPMI SOL Console drop during a PNOR firmware update. There is a log message "SOL session closed by BMC", but there is no impact on the system.
A problem was fixed for an IPMI DCMI "Get Power Reading" command average reading being inaccurate.
A problem was fixed for the FRU information for MfgDatetime which was not displayed correctly.
A problem was fixed for "SENSOR_ID" not being consistent with the sensor name in email alerts.
A problem was fixed for opening the Java iKVM program on a Mac OS X client system.
A problem was fixed in the BMC NTP support to protect against a possible denial-of-service attack in the NTP processing of a query. This is for protection against the Common Vulnerabilities and Exposures (CVE) number CVE-2016-7434.
A problem was fixed for an erroneous voltage reading at an empty CPU socket.
A problem was fixed for iKVM not showing the Hostboot and OPAL boot progress.
A problem was fixed for the BMC not requesting an OCC reset if the OCC is in Safe mode but the "OCC Active" sensor is still enabled. With the fix, after sixty seconds of the OCC being in Safe mode, the OCC reset will be requested regardless of the state of the "OCC Active" sensor.
A problem was fixed for the On-Chip Controller (OCC) dropping to Safe mode during BMC communication failures when an OCC reset was requested.
A problem was fixed for the OpenPOWER logo being pixelated (a very low-resolution image) in the Java iKVM viewer.
A problem was fixed for a SMASH console hang from the BMC SSH login: Start the SMASH console by typing "start /system1/sol1". Note that the command is unresponsive, and does not show a login prompt. With the fix, this sequence is responsive and shows the same console as the IPMI SOL console. |
V2.11
01/12/18 | Impact: New Severity: Available
New features and functions In response to recently reported security vulnerabilities, this firmware update is being released to address Common Vulnerabilities and Exposures issue numbers, CVE-2017-5715, CVE-2017-5753 and CVE-2017-5754. Operating System updates are required in conjunction with this FW level for CVE-2017-5753 and CVE-2017-5754.
|
V2.10 | Impact: New Severity: Available New features and functions Support for resonant clocking was enabled to reduce power consumption of the processors for an approximate 10W per processor socket savings. The Preboot Execution Environment (PXE) installation process was enhanced to provide more status log information and to show percentage-based download progress information. Support was added for a On-Chip Controller (OCC) Soft User Power Cap. Previously, if the User Power Cap was being exceeded and could not be maintained, the OCC placed the system into Safe mode (a power mode with reduced processor frequencies). With the Soft User Power Cap feature, the Safe mode is not used when the User Power Cap is exceeded. Instead, the OCC will log an error and continue to try to maintain the User Power Cap. System firmware changes that affect all systems A problem was fixed for a missing device discovery message and overly verbose output messages during the boot. It is now less verbose during the boot - only error-level messages are printed during Petitboot bootloader initialization. This means that there will be fewer messages printed as the system boots. Additionally, the Petitboot user interface is started earlier in the boot process. This means that the user will be presented with the user interface sooner, but it may still take time, potentially up to 30 seconds, for the user interface to be populated with boot options as storage and network hardware is being initialized. During this time, Petitboot will show the status message "Info: Waiting for device discovery". When Petitboot device discovery is completed, the following status message will be shown "Info: Connected to pb-discover!". System firmware changes that affect certain systems On systems with maximum memory configurations (where every DIMM slot is populated - size of DIMM does not matter), a problem has been fixed for systems losing performance and going into Safe mode (a power mode with reduced processor frequencies intended to protect the system from over-heating and excessive power consumption). This happened because of On-Chip Controller (OCC) time out errors when collecting Analog Power Subsystem Sweep (APSS) data, used by the OCC to tune the processor frequency. This problem occurs more frequently on systems that are running heavy workloads. Recovery from Safe mode back to normal performance can be done with a reboot of the system.
|
V2.00
12/16/16 | Impact: New Severity: Available
New features and functions
Support for the Red Hat Enterprise Linux 7.3 OS as a OPAL bare-metal install. For more information on the features delivered with RHEL7.3, see the Red Hat information portal: https://access.redhat.com/documentation/en/red-hat-enterprise-linux/.
Support for a OPAL raw console to receive output from the PowerPC boot EPAPR (Embedded Power Architecture Platform Requirements) wrapper. This allows decompression failures inside the wrapper caused by data corruption to be reported to the user.
Support for logging the following user activities on the BMC: 1) user add/delete 2) user password changed 3) Login web 4) Logout web 5) Launch iKVM 6) Close iKVM 7) Reset iKVM 8) BIOS FW is updated 9) BMC FW is updated 10) Event log is cleared
Support for BMC password management for password expiration date and user password lockout with the following variable controls: 1) Bad Password Threshold - integer 2) Invalid password disable - yes or no 3) Attempt Count Reset Interval (seconds) - integer 4) User Lockout Interval (seconds) - integer
Support for power-saving nap mode while in Skiboot, reducing power consumption during the boot.
Support for filtering the OCC Active Device Disabled Asserted SELs asserted during power off and OCC Active Device Enabled SELs asserted during power on as these do not represent a problem in the system but are part of normal operations. Here are the OCC SELs that are no longer reported during power off: 1 | 10/13/2016 | 13:53:23 | Processor OCC Active 1 | Device Enabled | Deasserted 2 | 10/13/2016 | 13:53:23 | Processor OCC Active 1 | Device Disabled | Asserted 3 | 10/13/2016 | 13:53:23 | Processor OCC Active 2 | Device Enabled | Deasserted 4 | 10/13/2016 | 13:53:23 | Processor OCC Active 2 | Device Disabled | Asserted
And here are the OCC SELs that are no longer reported during the power on: 5 | 10/13/2016 | 13:54:45 | Processor OCC Active 1 | Device Disabled | Deasserted 6 | 10/13/2016 | 13:54:45 | Processor OCC Active 1 | Device Enabled | Asserted 7 | 10/13/2016 | 13:54:45 | Processor OCC Active 2 | Device Disabled | Deasserted 8 | 10/13/2016 | 13:54:45 | Processor OCC Active 2 | Device Enabled | Asserted
Support for a new parameter in the pUpdate procedure to specify that SDR settings should be preserved on the PNOR firmware update. The default for the preservation of SDR settings in pUpdate was also changed to "not preserve" from "preserve" as this value is needed for the firmware update to V2.0 to work correctly.
System firmware changes that affect all systems
A problem was fixed for a critical hardware guard causing an infinite reboot loop and a flood of SELs. With the fix, the number of reboot attempts in the retry loop is limited to 10, assuring that the failing SELs are available for debugging the boot problem.
A problem was fixed for the IPMI Serial Over LAN (SOL) console to the Petitboot user interface for the left and right arrow movements. When editing the command line for the kernel, the user could not go to the start of the line and then go forward one character at a time.
A problem was fixed for a kexec-hardboot reboot of the system that caused USB devices to be lost. A system power cycle is needed to recover the USB devices when this error occurs.
A problem was fixed for the shutdown of PCI devices that was causing spurious reboots of the system for a power off. The logical PCI devices are now removed during the shutdown.
A problem was fixed for failures that happen when multiple Hypervisor Virtual Console (HVC) are active at the same time. On machines with more than one HVC console, any console after the first failed to register an interrupt handler since all consoles shared the same IRQ number.
A problem was fixed for fundamental PCI resets at boot time causing the PCI adapters to not be usable in the Linux OS. No errors occur in the skiboot but the adapters are configurable once the OS is reached.
A problem was fixed for time-out errors during the power off of PCI slots with " Timeout powering off slot ... FIRENZE-PCI: Wrong state 00000000 on slot" error message during a power off of the system.
A problem was fixed for the system remaining in "safe" mode after an On-Chip Controller (OCC) reset. In "safe" mode, the system is running at reduced processor frequencies, affecting system performance. The OCC reset is an error recovery command that can be requested by the BMC or OPAL for certain OCC errors.
A problem was fixed for the error recovery on a DIMM fault occurring newly installed DIMMs that can cause unnecessary DIIMM callouts and deconfiguration during the IPL or cause a IPL hang loop at step 13.10 with no bad DIMMs identified. With the fix, only the bad DIMM is called out and the IPL is successful.
A problem was fixed for a Non-recoverable processor error on the I2C primary with SEL "Processor #0x0e | Transition to Non-recoverable | Asserted" during the IPL.
A problem was fixed for a security vulnerability in the DropBear SSH server on the BMC to prevent an unauthenticated remote user from executing arbitrary code with root privileges. This was done by exploiting improper handling of string format specifiers in user names and host arguments. This Common Vulnerabilities and Exposures (CVE) number is CVE-2016-7406.
A problem was fixed for a security vulnerability in the DropBear SSH server on the BMC for an improper handling of a OpenSSH key file that could allow an unauthorized to execute arbitrary code with root privileges. This Common Vulnerabilities and Exposures (CVE) number is CVE-2016-7407.
A problem was fixed for a security vulnerability in the DropBear SSH dbclient on the BBC when handling the "-m" or "-c" arguments in scripts. An unauthenticated, remote user can exploit this, via a specially crafted script, to execute arbitrary code with root privileges. This Common Vulnerabilities and Exposures (CVE) number is CVE-2016-7408.
A problem was fixed for a security vulnerability in the DropBear SSH server on the BMC when compiled with the DEBUG_TRACE option and then run with the -v switch. A local user can exploit this to disclose process memory. This Common Vulnerabilities and Exposures (CVE) number is CVE-2016-7409.
|
V1.10
10/04/16 | Impact: New Severity: Available
New features and functions
Support for populating the host device tree with tracking information from the VPD. This includes, but is not limited to: system serial/part/manufacturer number, processor serial/part/manufacturer number, and DIMM serial/part/manufacturer number. This feature also populates the location code set in the system XML.
System firmware changes that affect all systems
A problem was fixed for the reporting of DIMM failures when two DIMMs have failed. With the fix, the second DIMM is able to be called out along with the first DIMM failure, instead of just the first DIMM failure.
A problem was fixed for a system IPL failure due to DRAM training issues for a faulty DIMM that does not log any SEL or eSEL records for the failure. The only signature of the fault is that it fails at ISTEP 13.10.
A problem was fixed for Petitboot adding unwanted kernel command-line options if no kernel arguments are provided on the OS boot. The unwanted options added were "console=hvc0 console=tty1" which breaks a SOL boot of the OS installer images.
A problem was fixed in the BMC login for a possible security issue.
Security problems were fixed in the BMC web interface for OWASP ZAP reported problems in the X-FRAME-OPTIONS; the X-CONTENT-TYPE_OPTIONS; and in the Cache control for the HTTP header. |
V1.00
09/09/16 | Impact: New Severity: New
New features and functions for MTM 8001-12C and 8001-22C
GA Level |
OS levels supported:
RHEL 7.2, any subsequent RHEL 7.x releases
RHEL 8.0, any subsequent RHEL 8.x releases
Ubuntu 16.04.1, any subsequent 16.x updates and releases
Ubuntu 18.04.1, and any subsequent Ubuntu 18.x updates and releases
Ubuntu 16.04.1 is the only supported version that does not have an important AST driver fix. If the Linux kernel level is lower than 4.11 and the AST fix is missing, the system will hang on the IPL. To circumvent this hang, the AST driver can be disabled. This will allow the IPL to complete but there will be no output on the VGA console:
Ensure the "modprobe.blacklist=ast" parameter is set in your boot loader (grub)
IBM Power CS 8005 servers supports Linux which provides a UNIX like implementation across many computer architectures. Linux supports almost all of the Power System I/O and the configurator verifies support on order. For more information about the software that is available on IBM Power Systems, see the Linux on IBM Power Systems website:
http://www.ibm.com/systems/power/software/linux/index.html
For the Hyperconverged CS 8005 servers, the OS hypervisor is Nutanix AHV running on Nutanix Acropolis with Prism Starter (includes Prism Central). Nutanix Acropolis is an operating system (AOS) for the Nutanix hyper-converged infrastructure platform at level 5.2 or later. It contains a number of data services and features for data protection, space efficiency, scalability, automated data tiering and security. Acropolis comes with its own built-in hypervisor called AHV; the name was derived from the words Acropolis Hypervisor. Nutanix Prism is an end-to-end management solution for virtualized data center environments that streamlines and automates common work flows, eliminating the need for multiple management solutions across data center operations.
The Linux operating system is an open source, cross-platform OS. It is supported on every Power Systems server IBM sells. Linux on Power Systems is the only Linux infrastructure that offers both scale-out and scale-up choices. One supported version of Linux on the IBM Power CS 8005 is Ubuntu Server 16.04.5 for IBM POWER8. For more information about Ubuntu Server for Ubuntu for POWER8 see the following website:
http://www.ubuntu.com/download/server/power8
Another supported version of Linux on the Power CS 8005 is Red Hat Enterprise Linux 7.4 and later releases, in the BE and LE modes. For additional questions about the availability of this release and supported Power servers, consult the Red Hat Hardware Catalog at https://hardware.redhat.com.
For information about the PowerLinux Community, see the following website:
https://www.ibm.com/developerworks/group/tpl
For information about the features and external devices that are supported by Linux, see this website:
http://www.ibm.com/systems/power/software/linux/index.html
Use one of the following commands at the Linux command prompt to determine the current Linux level:
•cat /proc/version
•uname -a
The output string from the command will provide the Linux version level.
The opal-prd package on the Linux system collects the OPAL Processor Recovery Diagnostics messages to log file /var/log/syslog. It is recommended that this package be installed if it is not already present as it will help with maintaining the system processors by alerting the users to processor maintenance when needed.
On Ubuntu Linux, perform command dpkg -l "opal-prd". The output shows whether the package is installed on your system by marking it with ii (installed) and un (not installed).
This package provides a daemon to load and run the OpenPower firmware's Processor Recovery Diagnostics binary. This is responsible for run-time maintenance of Power hardware.
If the package is not installed on your system, the following command can be run on Ubuntu to install it:
sudo apt-get install opal-prd
On Red Hat Linux, perform command "rpm -qa | grep -i opal-prd ". The command output indicates the package is installed on your system if the rpm for opal-prd is found and displayed. This package provides a daemon to load and run the OpenPower firmware's Processor Recovery Diagnostics binary. This is responsible for run-time maintenance of Power hardware. If the package is not installed on your system, the following command can be run on Red Hat to install it:
sudo yum update opal-prd
Use the ipmtool "fru" command or the BMC Web GUI FRU option to look at product details of FRU 47.
ipmitool -I lanplus -H <bmc host IP address> -U ADMIN -P <ADMIN password> fru print 47
Follow the instructions on Fix Central. You must read and agree to the license agreement to obtain the firmware packages.
When downgrading to a BMC version that is older than BMC 3.25, port 80 is not disabled even though the gui states it is.
If the BMC was previously configured to disable port 80 (Web port), take the following steps after the downgrade from BMC 3.25 to confirm the permanent disablement of port 80:
1) Access the BMC UI
2) Navigate to "Configure -> Ports"
3) Re-enable the "Web port" option by checking the box next to it
4) Click "Save"
5) Disable the "Web port" option by un-checking the box next to it
6) Click "Save"
7) Port 80 should now be disabled and in sync with the setting displayed in the BMC user interface.
For all firmware updates where the starting level is less than OP825.41 (BMC version is V3.16 or older) to a newer level, there is a problem where the system date and time may be incorrect after the firmware update process has completed. To correct this error, the user must manually reconfigure the BMC date/time under the Configuration menu option prior to starting the operating system. Updating time of day is required to ensure system operations are working correctly.
One function that you may see that is not working correctly is the remote console. The connection to the remote console will fail if the time of day is not correct.
Note 1: Before proceeding with a update to this service pack level, update the OS to the latest fix level for its release as there is an important fix for the AST driver that, if not applied, could prevent the system from IPLing after the service pack is installed. For Ubuntu level 16.04.1, there is no fix for the ast driver, so the OS level must be upgraded to an approved OS level (see section “4.0 Operating System Information“) or the AST driver can be disabled using the below command. The disable of the AST driver will cause loss of output to the VGA console, so this must be considered before choosing this option.
Ensure the "modprobe.blacklist=ast" parameter is set in your boot loader (grub).
Note 2:
1. When upgrading firmware from firmware version 1.xx to version 3.xx:, there are dependencies between BMC and PNOR. For that reason, you must update BOTH BMC and PNOR while system power is off.
1.1 Power off the system
1.2 Update BMC version 3.xx using pUpdate 2.20 or using BMC Web GUI.
1.3 After the BMC firmware is updated, the BMC is automatically reset. Wait for the BMC to come back on line.
1.3.1 You can ping the BMC to see if it is active on the network or you connect to the BMC Web GUI.
1.4 Check the BMC version to ensure the correct BMC version is updated.
1.4.1 Check Web GUI display for new BMC version, or
1.4.2 Use the ipmitool to check the BMC version
1.5 Update PNOR version 3.xx using pUpdate 2.20 or using BMC Web GUI.
1.5.1 The new PNOR version is not yet available until a power on is completed.
1.6 Once the PNOR update is complete, power on the system to petitboot menu or OS.
1.6.1 When the system IPL reaches petitboot, SBE validation will be started automatically. This will take about 2 minutes. Please allow the system to complete the SBE validation before attempting to shutdown. If SBE validation fails for any reason, the system power will be turned off by firmware. Follow repair action given in the SEL to correct the problem.
1.6.2 Once the system reaches to OS, you can check the PNOR version to ensure it is correct.
1.7 The system is ready to be used.
2a. Downgrade of firmware from version 3.xx or later to version 1.xx is NOT supported.
2b. BMC 3.23 and newer contains the fix for CVE-2019-11072, which requires change to the web config file. There is no requirement to change the web config file when upgrade to BMC 3.23 or newer as it is done automatically during the firmware update.
If a user needs to downgrade the BMC 3.23 or newer firmware to a level BMC 3.20 or older, the below steps must be performed. Otherwise, the user will not be able to connect to the BMC web interface.
2b.1 Ensure system is powered down
2b.2 Downgrade PNOR (to match target PNOR) but do not IPL ( Do this step if changing the PNOR level)
2b.3 Reset nginx with ipmitool raw command but do not reset BMC
ipmitool -H <<bmcHostname_or_bmcIPaddress>> -I lanplus -U ADMIN -P <<bmc_password>> raw 0x30 0x70 0xB7
2b.4 Downgrade BMC ( it will automatically reset)
2b.5 Power on the system to IPL (finish PNOR install/SBE updates) ( Do this step if PNOR level was changed).
3. Fast Reboot is not supported (fast reboot can be disabled with below command):
3.1 'nvram -p ibm,skiboot --update-config fast-reset=0'
3.2 Fast reboot is disabled by default.
4. After updating BMC firmware to version 3.15 or newer, the current BMC password for "ADMIN" user ID is still in effect until a factory reset is issued to the BMC. Once the factory reset is issued to the BMC, the BMC will reset the password for the "ADMIN" user ID to the default password, "ADMIN", but it will be expired. No remote command will work except the command to change the password. The "ADMIN" user ID's password can be changed with the interfaces listed below. New password strength rules (at least 8 characters with at least one lowercase letter, one uppercase letter and one numeric number) must be followed.
4.1 Web GUI
4.2 SSH
4.3 In-band IPMI command (host user must have root privilege)
The updating and upgrading of system firmware depends on several factors, such as the current firmware that is installed, and what operating systems is running on the system.
These scenarios and the associated installation instructions are comprehensively outlined in the firmware section of Fix Central, found at the following website:
http://www.ibm.com/support/fixcentral/
Any hardware failures should be resolved before proceeding with the firmware updates to help insure the system will not be running degraded after the updates.
The pUpdate utility is provided with the firmware update files from IBM Fix Central. It can be used to perform in-band (from the host OS), in-band update recovery, and out-of-band updates by selecting either the "-i usb" , "-i bt" or "-i lan" parameters, respectively on the command invocation. The code update needs to be done in two steps: 1) Update the BMC firmware and 2) Update the CEC PNOR for the hostboot and the OPAL components. It is recommended that the BMC be updated first unless otherwise specified in the firmware install instructions.
Before using the pUpdate command on the host, make sure that the ipmi driver is loaded in the kernel and the ipmi service is started.
Note: For updates that use the "usb" or "bt" pUpdate option, you must use the root user ID and password to log in to the host operating system. After you log in to the host operating system, ensure that the IPMI service is activated.
# chkconfig ipmi on
# service ipmi start
For more information about activating the IPMI service, see the OpenIPMI Driver: https://www.ibm.com/support/knowledgecenter/POWER8/p8eih/p8eih_ipmi_open_driver.htm
For in-band update, use the following "-i usb" invocation of pUpdate:
BMC update: "pUpdate -f bmc.bin -i usb", where bmc.bin is the name and location of the BMC image file.
PNOR update: "pUpdate -pnor pnor.bin -i usb", where pnor.bin is the name and location of the PNOR image file.
If the in-band update fails on the BMC, use the recovery option with the Block Transfer (bt) invocation of pUpdate:
BMC update: "pUpdate -f bmc.bin -i bt -r y" where bmc.bin is the name and location of the BMC image file.
PNOR update:" pUpdate -pnor pnor.bin -i bt " where pnor.bin is the name and location of the PNOR image file.
For more information on BMC recovery steps, refer to the following link in the IBM Knowledge Center:
https://www.ibm.com/support/knowledgecenter/POWER8/p8eis/p8eis_console_problem.htm
If the host is not booted, a network connection can be made to the BMC and an out-of-band update done with the following LAN invocation from a Linux companion system:
BMC update: " pUpdate -f bmc.bin -i lan -h xx.xx.xx.xx -u ADMIN -p <ADMIN password> -r y" where bmc.bin is the name and location of the BMC image file, xx.xx.xx.xx is the IP address of the BMC.
PNOR update: "pUpdate -pnor pnor.bin -i lan -h xx.xx.xx.xx -u ADMIN -p <ADMIN password>" where pnor.bin is the name and location of the PNOR image file and xx.xx.xx.xx is the IP address of the BMC.
For more details on how to use the pUpdate utility, refer to the following link:
https://www.ibm.com/support/knowledgecenter/POWER8/p8eit/p8eit_update_firmware_pupdate.htm
You can use diagnostic utilities to diagnose adapter problems.
For more details on how to use the diagnostic utilities, refer to the following link:
https://www.ibm.com/support/knowledgecenter/POWER8/p8eit/p8eit_diags_kickoff.htm
Open Power requires Source Forge ipmitool level v1.8.15 to execute correctly on the V1.00, V2.00 and later firmware.
Another method to update the system firmware is by using the baseboard management controller (BMC).
The system firmware is a combination of the BMC firmware and the PNOR firmware. To update the system firmware, update both the BMC firmware and the PNOR firmware by using the BMC.
Note 1: System firmware update from the BMC Web GUI is only supported on Google Chrome and Mozilla Firefox browsers.
Note 2: Use the firmware update procedure in this section when there is no dependency between BMC and PNOR. When there is a dependency between BMC and PNOR, the procedure in this section may not work. And possibly lead to unexpected system behavior. When there is a dependency between BMC and PNOR that requires both BMC and PNOR to be updated, the user will be alerted to this by the special instructions for the firmware update that can be found in either "Section 1.0 Affected Systems" or in "Section 7.0 Installing the Firmware".
Complete the following steps to update the BMC firmware:
1. Log in to the BMC by entering the user name and password. Then, press Enter.
2. From the Maintenance list on the BMC dashboard, select BMC Update.
3. In the BMC Update window, select Enter Update Mode. Click OK.
4. In the BMC Upload window, choose the .bin file from your local system folder and click Upload Firmware. Wait for the file to be uploaded. Then, click OK.
5. The existing and new versions of the BMC firmware are displayed. Ensure that the Preserve Configuration check box is selected and the Preserve SDR check box is not selected. Click Start Upgrade.
Note: You cannot perform other activities by using the BMC interface until the BMC firmware update is complete.
6. The upgrade progress of the BMC firmware update is displayed. After the BMC firmware update is complete, the system is restarted.
7. After the restart of the system is complete, verify the firmware revision level in the System menu of the BMC dashboard.
Complete the following steps to update the PNOR firmware:
1. Log in to the BMC by entering the user name and password. Then, press Enter.
2. From the Maintenance list on the dashboard, select PNOR Update.
3. In the PNOR Upload window, choose the .pnor file from your local system folder and click Upload PNOR. Wait for the file to be uploaded. Then, click OK.
4. The existing and new dates of the PNOR firmware are displayed. Click Start Upgrade.
Note: You cannot perform other activities by using the BMC interface until the PNOR update is complete.
5. The progress of the PNOR update is displayed. After the PNOR update is completed, the system must be restarted to finish installation of the new PNOR firmware.
For more information on updating the firmware using the BMC, refer to the following link:
https://www.ibm.com/support/knowledgecenter/POWER8/p8eit/p8eit_update_firmware_bmc.htm
System I/O devices have firmware that can be updated. Please see the tables below for minimum level of firmware.
Updated firmware, when released, will include firmware update instructions.
Details regarding the I/O firmware levels for the 8005-12N, and 8005-22N IBM Power systems can be found in the link below.
https://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/W51a7ffcf4dfd_4b40_9d82_446ebc23c550/page/IBM%20Power%208001-12C%20and%208001-22C%20IO%20Firmware
Here are FAQs for OS related dependencies, limitations, and usage for the LC servers and I/O devices:
https://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/W51a7ffcf4dfd_4b40_9d82_446ebc23c550/page/IBM%20Power%20LC%20readme
All references in the linked documents to 8001-12C apply equally to 8005-12N and all references to 8001-22C apply equally to 8005-22N.
The latest firmware, tools and how-to notes are available here:
ftp://ftp.supermicro.com/driver/P8DTU/
Firmware level for these devices can be checked using OS (Linux) utility “ethtool” as follows:
ethtool -i <device>
System | Part# | Description | Minimum FW level |
8001-12C | AOC-UR-i4XTF | Intel x710/XL710 quad port 10G ( Riser card in system) | 5.03 |
8001-22C | AOC-2UR66-i4XTF | Intel x710/XL710 quad port 10G ( Riser card in system) | 5.03 |
To check the firmware version on this device, use the Linux mvcli tool. The command "mvcli info -o hba" will show information of all 9230 adapters present on machine. Examine the "Firmware version" information to see the current level on the desired adapter.
System | Part# | Description | Minimum FW level |
8001-12C | Marvell 88SE9230 | On-board 8P SATA3 Controller (88SE9230) | 2.3.12.1074 |
8001-22C | Marvell 88SE9230 | On-board 8P SATA3 Controller (88SE9230) | 2.3.12.1074 |
The service processor, or baseboard management controller (BMC), provides a hypervisor and operating system-independent layer that uses the robust error detection and self-healing functions that are built into the POWER8 processor and memory buffer modules. Open power application layer (OPAL) is the system firmware in the stack of POWER8 processor-based Linux-only servers.
The service processor, or baseboard management controller (BMC), is the primary control for autonomous sensor monitoring and event logging features on the CS server.
The BMC supports the Intelligent Platform Management Interface (IPMI) for system monitoring and management. The BMC monitors the operation of the firmware during the boot process and also monitors the OPAL hypervisor for termination. The firmware code update is supported through the BMC and Intelligent Platform Monitoring Interface (IPMI) and the BMC Web GUI The GUI console is accessed using a web browser with a "http:" connection to port. See section 1.2 for the supported browsers that can be used with BMC Web GUI.
The Open Power Abstraction Layer (OPAL) provides hardware abstraction and run time services to the running host Operating System.
A KVM can be used on top of the installed OS to run Linux virtual guest OS machines.
Find out more about OPAL skiboot here:
https://github.com/open-power/skiboot
The Intelligent Platform Management Interface (IPMI) is an open standard for monitoring, logging, recovery, inventory, and control of hardware that is implemented independent of the main CPU, BIOS, and OS. It is the default console to use when you configure PowerKVM. The CS 8005 servers provides one 10M/100M baseT IPMI port.
The ipmitool is a utility for managing and configuring devices that support IPMI. It provides a simple command-line interface to the service processor. You can install the ipmitool from the Linux distribution packages in your workstation, sourceforge.net, or another server (preferably on the same network as the installed server). For example, in Ubuntu, use this command:
$ sudo apt-get install ipmitool
For installing ipmitool from sourceforge, please see section 1.1 "Minimum ipmitool Code Level".
For more information about ipmitool, there are several good references for ipmitool commands:
1.The man page
2.The built-in command line help provides a list of IPMItool commands:
# ipmitool help
3.You can also get help for many specific IPMItool commands by adding the word help after the command:
# ipmitool channel help
4.For a list of common ipmitool commands and help on each, you may use the following link:
www.ibm.com/support/knowledgecenter/linuxonibm/liabp/liabpcommonipmi.htm
To connect to your host system with IPMI, you need to know the IP address of the server and have
a valid password. To power on the server with the ipmitool, follow these steps:
1. Open a terminal program.
2. Power on your server with the ipmitool:
ipmitool -I lanplus -H bmc_ip_address -U ADMIN -P ADMIN_password power on
3. Activate your IPMI console:
ipmitool -I lanplus -H bmc_ip_address -U ADMIN -P ADMIN_password sol activate
Petitboot is a kexec based bootloader used by IBM POWER8 systems for doing the bare-metal installs on the 8005 servers.
After the POWER8 system powers on, the petitboot bootloader scans local boot devices and network interfaces to find boot options that are available to the system. Petitboot returns a list of boot options that are available to the system. If you are using a static IP or if you did not provide boot arguments in your network boot server, you must provide the details to petitboot. You can configure petitboot to find your boot with the following instructions:
https://www.ibm.com/support/knowledgecenter/linuxonibm/liabp/liabppetitbootadvanced.htm
You can edit petitboot configuration options, change the amount of time before Petitboot automatically boots, etc. with these instructions:
https://www.ibm.com/support/knowledgecenter/linuxonibm/liabp/liabppetitbootconfig.htm
After you select to boot the ISO media for the Linux distribution of your choice, the installer wizard for that Linux distribution walks you through the steps to set up disk options, your root password, time zones, and so on.
You can read more about the petitboot bootloader program here:
https://www.kernel.org/pub/linux/kernel/people/geoff/petitboot/petitboot.html
Kernel-based Virtual Machine (KVM) is a cross-platform, open source hypervisor that provides enterprise-class performance, scalability and security to run Linux and other workloads on a range of processor architectures. For the 8005 servers, the KVM function is provided by the Nutanix Acropolis Hypervisor (AHV) to create and manage partitions.
Nutanix AHV is an enterprise-class virtualization solution included with the Acropolis hyperconverged infrastructure stack on all Nutanix appliances, with no additional software components to install and manage.
IBM PowerKVM is not supported on the 8005 servers.
Note: Power CS821 and CS822 8005 servers and AHV do not support AIX or IBM i guest VMs and cannot be managed by a HMC.
This guide helps you install Ubuntu on a Linux on a Power Systems server.
Overview
Use the information found in http://www.ibm.com/support/knowledgecenter/linuxonibm/liabw/liabwkickoff.htm to install Linux, in this case Ubuntu, on a non-virtualized (bare metal) IBM Power LC server. Note that the choice of PowerKVM is offered in the link but that is not a supported OS for the 8005 servers.
The Ubuntu installer is available for download for specific Linux levels on Power Systems. For information about which systems support Ubuntu, see Supported Linux distributions for POWER8 Linux on Power systems at https://ibm.biz/BdEH5t.
Date | Description |
09/01/2021 |
|
07/13/2021 |
|
04/21/2021 |
|
07/29/2020 |
|
07/06/2020 |
|
12/02/2019 | OP825.40 with PNOR V3.09 and SMC BMC V3.16 |
09/23/2019 | OP825.31 with PNOR V3.08 and SMC BMC V3.13 |
03/07/2018 | V2.20 release (BMC update only – no change to the PNOR) |
01/12/2018 | V2.11 release (PNOR only – no change to BMC code) |
09/18/2017 | V2.10 release - Adding servers 8005-12N and 8005-22N (No changes to PNOR or BMC code) for Nutanix 5.2 GA |
06/22/2017 | V2.10 release (PNOR only - no change to BMC code) |
05/25/2017 | V2.00 release - updated the readme ONLY. No changes to delivered binaries |
12/16/2016 | V2.00 for LC server 8001-12C and 8001-22C |
10/04/2016 | V1.10 release - Added Marvell SATA3 Controller |
09/09/2016 | New for LC server 8001-12C and 8001-22C |