Fix (APAR): JR47613 Status: Fix Product: WebSphere Process Server Release: 7.0.0.5 Operating System: All operating systems Supersedes Fixes: Pre-requisite Fixes: JR44124;JR45467 Exclusive-requisite Fixes: Defect: JR47613 Byte size of APAR: 167700 Date: 2013-09-13 Abstract: Resilient connection pool for event sequencing. Problem Description: This fix improves the resiliency of the connection pool of event sequencing. I.e. if certain exception happens in the messaging layer, the connection pool will be refreshed. Problem Summary: Event sequencing enabled components can stop processing messages when a connection from the event sequencing pool receives a SIConnectionDroppedException, as listed below. Normally, to overcome the situation an application restart is required. Which is not always acceptable for customers. CNTR0020E: EJB threw an unexpected (non-declared) exception during invocation of method "processMessageInBifurcatedSession" on bean "BeanId(persistentLkMgr#persistentLkMgrEJB.jar#ESWork, null)". Exception data: com.ibm.wbiserver.sequencing.exceptions.SequencingRuntimeExcepti on: CWRES0015E: The message with msgId=7E3935DE7A2DC62D_2500245 failed during processing. The error is CWSIV0155E: An internal error occurred. The exception javax.resource.spi.ResourceAllocationException: CWSIV0353E: An internal error occurred. The exception com.ibm.wsspi.sib.core.exception.SIConnectionDroppedException: CWSIJ0044E: An operation was attempted on a connection that is already closed. was thrown while attempting to clone a connection to the messaging engine. was thrown while attempting to re-associate with a managed connection.. at com.ibm.wbiserver.sequencing.work.ESWorkBean.processMessage(ESWorkBean.java:177) at com.ibm.wbiserver.sequencing.work.ESWorkBean.processMessageInBifurcatedSession(ESWorkBean.java:101) at com.ibm.wbiserver.sequencing.work.EJSLocalStatelessESWork_ddc8f6c2.processMessageInBifurcatedSession(Unknown Source) at com.ibm.wbiserver.sequencing.work.EsWorkImpl.run(EsWorkImpl.java:251) at com.ibm.ws.asynchbeans.J2EEContext$RunProxy.run(J2EEContext.java:268) at java.security.AccessController.doPrivileged(AccessController.java:250) at javax.security.auth.Subject.doAs(Subject.java:495) at com.ibm.websphere.security.auth.WSSubject.doAs(WSSubject.java:131) at com.ibm.websphere.security.auth.WSSubject.doAs(WSSubject.java:89) at com.ibm.ws.asynchbeans.J2EEContext$DoAsProxy.run(J2EEContext.java:335) at java.security.AccessController.doPrivileged(AccessController.java:277) at com.ibm.ws.asynchbeans.J2EEContext.run(J2EEContext.java:777) at com.ibm.ws.asynchbeans.WorkWithExecutionContextImpl.go(WorkWithExecutionContextImpl.java:222) at com.ibm.ws.asynchbeans.ABWorkItemImpl.run(ABWorkItemImpl.java:159) at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1656) Caused by: com.ibm.websphere.sib.exception.SIResourceException: CWSIV0155E: An internal error occurred. The exception javax.resource.spi.ResourceAllocationException: CWSIV0353E: An internal error occurred. The exception com.ibm.wsspi.sib.core.exception.SIConnectionDroppedException: CWSIJ0044E: An operation was attempted on a connection that is already closed. was thrown while attempting to clone a connection to the messaging engine. was thrown while attempting to re-associate with a managed connection. at com.ibm.ws.sib.ra.impl.SibRaConnection.getAssociatedManagedConnection(SibRaConnection.java:1846) at com.ibm.ws.sib.ra.impl.SibRaConnection.getContainerTransaction(SibRaConnection.java:1939) at com.ibm.ws.sib.ra.impl.SibRaConnection.mapTransaction(SibRaConnection.java:2046) at com.ibm.ws.sib.ra.impl.SibRaBifurcatedConsumerSession.deleteSet(SibRaBifurcatedConsumerSession.java:172) at com.ibm.wbiserver.sequencing.work.ESWorkBean.processMessage(ESWorkBean.java:141) ... 14 more Caused by: javax.resource.spi.ResourceAllocationException: CWSIV0353E: An internal error occurred. The exception com.ibm.wsspi.sib.core.exception.SIConnectionDroppedException: CWSIJ0044E: An operation was attempted on a connection that is already closed. was thrown while attempting to clone a connection to the messaging engine. at com.ibm.ejs.j2c.FreePool.createManagedConnectionWithMCWrapper(FreePool.java:2413) at com.ibm.ejs.j2c.FreePool.createOrWaitForConnection(FreePool.java:1759) at com.ibm.ejs.j2c.PoolManager.reserve(PoolManager.java:2631) at com.ibm.ejs.j2c.ConnectionManager.allocateMCWrapper(ConnectionManager.java:1064) at com.ibm.ejs.j2c.ConnectionManager.associateConnection(ConnectionManager.java:2382) at com.ibm.ejs.j2c.ConnectionManager.associateConnection(ConnectionManager.java:2315) at com.ibm.ws.sib.ra.impl.SibRaConnection.getAssociatedManagedConnection(SibRaConnection.java:1813) ... 18 more Caused by: com.ibm.wsspi.sib.core.exception.SIConnectionDroppedException: CWSIJ0044E: An operation was attempted on a connection that is already closed. at com.ibm.ws.sib.jfapchannel.impl.Connection.preSendProcessing(Connection.java:529) at com.ibm.ws.sib.jfapchannel.impl.Connection.send(Connection.java:672) at com.ibm.ws.sib.jfapchannel.impl.ConversationImpl.sendInternal(ConversationImpl.java:1157) at com.ibm.ws.sib.jfapchannel.impl.ConversationImpl.exchange(ConversationImpl.java:978) at com.ibm.ws.sib.comms.common.JFAPCommunicator.jfapExchange(JFAPCommunicator.java:386) at com.ibm.ws.sib.comms.client.ConnectionProxy.cloneConnection(ConnectionProxy.java:2997) at com.ibm.ws.sib.ra.impl.SibRaManagedConnectionFactory.createManagedConnection(SibRaManagedConnectionFactory.java:248) at com.ibm.ejs.j2c.FreePool.createManagedConnectionWithMCWrapper(FreePool.java:2082) ... 24 more Problem Conclusion: This fix enhances error resiliency by explicitly checking if a SequencingRuntimeException was caused by an SIConnectionDroppedException. If this happens the event sequencing connection pool for this particular application will be refreshed. NOTE: IBM Installation Manager is required to install this interim.fix (1.4.1 or later recommended). For further details see: http://publib.boulder.ibm.com/infocenter/dmndhelp/v7r0mx/index.jsp?topic=/com.ibm.websphere.wps.doc/doc/tins_installingfix_manually.html NOTE: This interim fix is applicable to a full WPS installation only, and should not be applied to a WPS client installation. Directions to apply fix: 1) Shutdown WebSphere. It is important that you perform a controlled and complete shutdown of the server to ensure that all transactions have completed, before installing the fix. 2) Extract the .zip file to a directory. For example, extract to C:\temp. 3) Add the fix pack repository location in IBM Installation Manager: a. Start IBM Installation Manager. b. In Installation Manager, click File > Preferences > Repositories. c. Click Add Repository. d. Browse to or enter the file path to the repository.config file. The repository.config file is located in the directory where you extracted the compressed files. For example, enter C:\temp\repository.config, and then click OK. 4) On the main page of the Installation Manager, click Update. 5) Restart WebSphere Directions to remove fix: NOTE: FIXES MUST BE REMOVED IN THE ORDER THEY WERE APPLIED. DO NOT REMOVE A FIX UNLESS ALL FIXES APPLIED AFTER IT HAVE FIRST BEEN REMOVED. YOU MAY REAPPLY ANY REMOVED FIX. Example: If your system has fix1, fix2, and fix3 applied in that order and fix2 is to be removed, fix3 must be removed first, then fix2 may be removed 1) Shutdown WebSphere. It is important that you perform a controlled and complete shutdown of the server to ensure that all transactions have completed, before installing the fix. 2) Start IBM Installation Manager. 3) On the main page of the Installation Manager, click Unistall. 4) Select the package to uninstall and follow the prompts on screen. 5) Restart WebSphere Directions to re-apply fix: 1) Shutdown WebSphere. It is important that you perform a controlled and complete shutdown of the server to ensure that all transactions have completed, before installing the fix. 2) Extract the .zip file to a directory. For example, extract to C:\temp. 3) Add the fix pack repository location in IBM Installation Manager: a. Start IBM Installation Manager. b. In Installation Manager, click File > Preferences > Repositories. c. Click Add Repository. d. Browse to or enter the file path to the repository.config file. The repository.config file is located in the directory where you extracted the compressed files. For example, enter C:\temp\repository.config, and then click OK. 4) On the main page of the Installation Manager, click Update. 5) Restart WebSphere. Additional Information: