Red Hat Bugzilla – Bug 1493558
[Scale][Infra] Memory leak related to IllegalStateException: Attempted to close chain with outstanding transaction PingPongTransaction
Last modified: 2018-10-18 03:22:46 EDT
Description of problem: In a clustered ODL setup with 3 Controllers and 3 Computes when we trigger browbeat/rally with concurrencies of 8, 16, 32 and times set to 500, we observed the following exception in Karaf. 2017-08-22 17:00:50,640 | ERROR | CommitFutures-2 | ExecutionList | 65 - com.google.guava - 18.0.0 | RuntimeException while executing runnable com.google.common.util.concurrent.Futures$6@7d061282 with executor INSTANCE java.lang.IllegalStateException: Attempted to close chain with outstanding transaction PingPongTransaction{delegate=org.opendaylight.controller.cluster.databroker.DOMBrokerReadWriteTransaction@1947c741} at com.google.common.base.Preconditions.checkState(Preconditions.java:197)[65:com.google.guava:18.0.0] at org.opendaylight.controller.md.sal.dom.broker.impl.PingPongTransactionChain.close(PingPongTransactionChain.java:373)[171:org.opendaylight.controller.sal-broker-impl:1.5.2.Carbon] at org.opendaylight.controller.md.sal.binding.impl.BindingDOMTransactionChainAdapter.close(BindingDOMTransactionChainAdapter.java:115)[173:org.opendaylight.controller.sal-binding-broker-impl:1.5.2.Carbon] at java.util.Optional.ifPresent(Optional.java:159)[:1.8.0_141] at org.opendaylight.openflowplugin.impl.device.TransactionChainManager.createTxChain(TransactionChainManager.java:84)[307:org.opendaylight.openflowplugin.impl:0.4.2.Carbon] at org.opendaylight.openflowplugin.impl.device.TransactionChainManager.onTransactionChainFailed(TransactionChainManager.java:242)[307:org.opendaylight.openflowplugin.impl:0.4.2.Carbon] at org.opendaylight.controller.md.sal.binding.impl.BindingDOMTransactionChainAdapter.failTransactionChain(BindingDOMTransactionChainAdapter.java:110)[173:org.opendaylight.controller.sal-binding-broker-impl:1.5.2.Carbon] at org.opendaylight.controller.md.sal.binding.impl.BindingDOMTransactionChainAdapter.access$200(BindingDOMTransactionChainAdapter.java:31)[173:org.opendaylight.controller.sal-binding-broker-impl:1.5.2.Carbon] at org.opendaylight.controller.md.sal.binding.impl.BindingDOMTransactionChainAdapter$3.onFailure(BindingDOMTransactionChainAdapter.java:91)[173:org.opendaylight.controller.sal-binding-broker-impl:1.5.2.Carbon] at com.google.common.util.concurrent.Futures$6.run(Futures.java:1310)[65:com.google.guava:18.0.0] at com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:457)[65:com.google.guava:18.0.0] at com.google.common.util.concurrent.ExecutionList.executeListener(ExecutionList.java:156)[65:com.google.guava:18.0.0] at com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:145)[65:com.google.guava:18.0.0] at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:202)[65:com.google.guava:18.0.0] at com.google.common.util.concurrent.SettableFuture.setException(SettableFuture.java:68)[65:com.google.guava:18.0.0] at org.opendaylight.controller.md.sal.dom.broker.impl.PingPongTransaction.onFailure(PingPongTransaction.java:65)[171:org.opendaylight.controller.sal-broker-impl:1.5.2.Carbon] at org.opendaylight.controller.md.sal.dom.broker.impl.PingPongTransactionChain.transactionFailed(PingPongTransactionChain.java:293)[171:org.opendaylight.controller.sal-broker-impl:1.5.2.Carbon] at org.opendaylight.controller.md.sal.dom.broker.impl.PingPongTransactionChain$2.onFailure(PingPongTransactionChain.java:243)[171:org.opendaylight.controller.sal-broker-impl:1.5.2.Carbon] at com.google.common.util.concurrent.Futures$6.run(Futures.java:1310)[65:com.google.guava:18.0.0] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)[:1.8.0_141] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)[:1.8.0_141] at java.lang.Thread.run(Thread.java:748)[:1.8.0_141]
Upstream Nitrogen: https://git.opendaylight.org/gerrit/#/c/63236 Stable/Carbon: https://git.opendaylight.org/gerrit/#/c/62674/ For info, upstream patch is now merged.
Tested with puddle 2018-05-23.1 and opendaylight rpm opendaylight-8.0.0-10.el7ost.noarch . Could not reproduce. Verified that it has been fixed.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2018:2086