Bug 1493558

Summary: [Scale][Infra] Memory leak related to IllegalStateException: Attempted to close chain with outstanding transaction PingPongTransaction
Product: Red Hat OpenStack Reporter: Sridhar Gaddam <sgaddam>
Component: opendaylightAssignee: Michael Vorburger <vorburger>
Status: CLOSED ERRATA QA Contact: Sai Sindhur Malleni <smalleni>
Severity: urgent Docs Contact:
Priority: urgent    
Version: 12.0 (Pike)CC: aadam, knylande, mkolesni, nyechiel, smalleni, trozet, tvignaud, vorburger
Target Milestone: betaKeywords: Triaged
Target Release: 13.0 (Queens)   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard: scale_lab, odl_infra
Fixed In Version: opendaylight-8.0.0-1.el7ost Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
N/A
Last Closed: 2018-06-27 13:37:30 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1451401, 1577975    

Description Sridhar Gaddam 2017-09-20 13:37:13 UTC 
Description of problem:
In a clustered ODL setup with 3 Controllers and 3 Computes when we trigger browbeat/rally with concurrencies of 8, 16, 32 and times set to 500, we observed the following exception in Karaf.


2017-08-22 17:00:50,640 | ERROR | CommitFutures-2  | ExecutionList                    | 65 - com.google.guava - 18.0.0 | RuntimeException while executing runnable com.google.common.util.concurrent.Futures$6@7d061282 with executor INSTANCE
java.lang.IllegalStateException: Attempted to close chain with outstanding transaction PingPongTransaction{delegate=org.opendaylight.controller.cluster.databroker.DOMBrokerReadWriteTransaction@1947c741}
        at com.google.common.base.Preconditions.checkState(Preconditions.java:197)[65:com.google.guava:18.0.0]
        at org.opendaylight.controller.md.sal.dom.broker.impl.PingPongTransactionChain.close(PingPongTransactionChain.java:373)[171:org.opendaylight.controller.sal-broker-impl:1.5.2.Carbon]
        at org.opendaylight.controller.md.sal.binding.impl.BindingDOMTransactionChainAdapter.close(BindingDOMTransactionChainAdapter.java:115)[173:org.opendaylight.controller.sal-binding-broker-impl:1.5.2.Carbon]
        at java.util.Optional.ifPresent(Optional.java:159)[:1.8.0_141]
        at org.opendaylight.openflowplugin.impl.device.TransactionChainManager.createTxChain(TransactionChainManager.java:84)[307:org.opendaylight.openflowplugin.impl:0.4.2.Carbon]
        at org.opendaylight.openflowplugin.impl.device.TransactionChainManager.onTransactionChainFailed(TransactionChainManager.java:242)[307:org.opendaylight.openflowplugin.impl:0.4.2.Carbon]
        at org.opendaylight.controller.md.sal.binding.impl.BindingDOMTransactionChainAdapter.failTransactionChain(BindingDOMTransactionChainAdapter.java:110)[173:org.opendaylight.controller.sal-binding-broker-impl:1.5.2.Carbon]
        at org.opendaylight.controller.md.sal.binding.impl.BindingDOMTransactionChainAdapter.access$200(BindingDOMTransactionChainAdapter.java:31)[173:org.opendaylight.controller.sal-binding-broker-impl:1.5.2.Carbon]
        at org.opendaylight.controller.md.sal.binding.impl.BindingDOMTransactionChainAdapter$3.onFailure(BindingDOMTransactionChainAdapter.java:91)[173:org.opendaylight.controller.sal-binding-broker-impl:1.5.2.Carbon]
        at com.google.common.util.concurrent.Futures$6.run(Futures.java:1310)[65:com.google.guava:18.0.0]
        at com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:457)[65:com.google.guava:18.0.0]
        at com.google.common.util.concurrent.ExecutionList.executeListener(ExecutionList.java:156)[65:com.google.guava:18.0.0]
        at com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:145)[65:com.google.guava:18.0.0]
        at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:202)[65:com.google.guava:18.0.0]
        at com.google.common.util.concurrent.SettableFuture.setException(SettableFuture.java:68)[65:com.google.guava:18.0.0]
        at org.opendaylight.controller.md.sal.dom.broker.impl.PingPongTransaction.onFailure(PingPongTransaction.java:65)[171:org.opendaylight.controller.sal-broker-impl:1.5.2.Carbon]
        at org.opendaylight.controller.md.sal.dom.broker.impl.PingPongTransactionChain.transactionFailed(PingPongTransactionChain.java:293)[171:org.opendaylight.controller.sal-broker-impl:1.5.2.Carbon]
        at org.opendaylight.controller.md.sal.dom.broker.impl.PingPongTransactionChain$2.onFailure(PingPongTransactionChain.java:243)[171:org.opendaylight.controller.sal-broker-impl:1.5.2.Carbon]
        at com.google.common.util.concurrent.Futures$6.run(Futures.java:1310)[65:com.google.guava:18.0.0]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)[:1.8.0_141]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)[:1.8.0_141]
        at java.lang.Thread.run(Thread.java:748)[:1.8.0_141]
Comment 1 Sridhar Gaddam 2017-09-20 13:41:04 UTC 
Upstream Nitrogen: https://git.opendaylight.org/gerrit/#/c/63236
Stable/Carbon: https://git.opendaylight.org/gerrit/#/c/62674/

For info, upstream patch is now merged.
Comment 8 Sai Sindhur Malleni 2018-05-25 03:09:37 UTC 
Tested with puddle 2018-05-23.1 and opendaylight rpm opendaylight-8.0.0-10.el7ost.noarch
. Could not reproduce. Verified that it has been fixed.
Comment 11 errata-xmlrpc 2018-06-27 13:37:30 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2018:2086