Bug 1493558 - [Scale][Infra] Memory leak related to IllegalStateException: Attempted to close chain with outstanding transaction PingPongTransaction
Summary: [Scale][Infra] Memory leak related to IllegalStateException: Attempted to clo...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: opendaylight
Version: 12.0 (Pike)
Hardware: Unspecified
OS: Unspecified
urgent
urgent
Target Milestone: beta
: 13.0 (Queens)
Assignee: Michael Vorburger
QA Contact: Sai Sindhur Malleni
URL:
Whiteboard: scale_lab, odl_infra
Depends On:
Blocks: 1451401 1577975
TreeView+ depends on / blocked
 
Reported: 2017-09-20 13:37 UTC by Sridhar Gaddam
Modified: 2018-10-18 07:22 UTC (History)
8 users (show)

Fixed In Version: opendaylight-8.0.0-1.el7ost
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
N/A
Last Closed: 2018-06-27 13:37:30 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
OpenDaylight Bug 9038 None None None 2017-09-20 13:37:58 UTC
Red Hat Product Errata RHEA-2018:2086 None None None 2018-06-27 13:38:54 UTC

Description Sridhar Gaddam 2017-09-20 13:37:13 UTC
Description of problem:
In a clustered ODL setup with 3 Controllers and 3 Computes when we trigger browbeat/rally with concurrencies of 8, 16, 32 and times set to 500, we observed the following exception in Karaf.


2017-08-22 17:00:50,640 | ERROR | CommitFutures-2  | ExecutionList                    | 65 - com.google.guava - 18.0.0 | RuntimeException while executing runnable com.google.common.util.concurrent.Futures$6@7d061282 with executor INSTANCE
java.lang.IllegalStateException: Attempted to close chain with outstanding transaction PingPongTransaction{delegate=org.opendaylight.controller.cluster.databroker.DOMBrokerReadWriteTransaction@1947c741}
        at com.google.common.base.Preconditions.checkState(Preconditions.java:197)[65:com.google.guava:18.0.0]
        at org.opendaylight.controller.md.sal.dom.broker.impl.PingPongTransactionChain.close(PingPongTransactionChain.java:373)[171:org.opendaylight.controller.sal-broker-impl:1.5.2.Carbon]
        at org.opendaylight.controller.md.sal.binding.impl.BindingDOMTransactionChainAdapter.close(BindingDOMTransactionChainAdapter.java:115)[173:org.opendaylight.controller.sal-binding-broker-impl:1.5.2.Carbon]
        at java.util.Optional.ifPresent(Optional.java:159)[:1.8.0_141]
        at org.opendaylight.openflowplugin.impl.device.TransactionChainManager.createTxChain(TransactionChainManager.java:84)[307:org.opendaylight.openflowplugin.impl:0.4.2.Carbon]
        at org.opendaylight.openflowplugin.impl.device.TransactionChainManager.onTransactionChainFailed(TransactionChainManager.java:242)[307:org.opendaylight.openflowplugin.impl:0.4.2.Carbon]
        at org.opendaylight.controller.md.sal.binding.impl.BindingDOMTransactionChainAdapter.failTransactionChain(BindingDOMTransactionChainAdapter.java:110)[173:org.opendaylight.controller.sal-binding-broker-impl:1.5.2.Carbon]
        at org.opendaylight.controller.md.sal.binding.impl.BindingDOMTransactionChainAdapter.access$200(BindingDOMTransactionChainAdapter.java:31)[173:org.opendaylight.controller.sal-binding-broker-impl:1.5.2.Carbon]
        at org.opendaylight.controller.md.sal.binding.impl.BindingDOMTransactionChainAdapter$3.onFailure(BindingDOMTransactionChainAdapter.java:91)[173:org.opendaylight.controller.sal-binding-broker-impl:1.5.2.Carbon]
        at com.google.common.util.concurrent.Futures$6.run(Futures.java:1310)[65:com.google.guava:18.0.0]
        at com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:457)[65:com.google.guava:18.0.0]
        at com.google.common.util.concurrent.ExecutionList.executeListener(ExecutionList.java:156)[65:com.google.guava:18.0.0]
        at com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:145)[65:com.google.guava:18.0.0]
        at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:202)[65:com.google.guava:18.0.0]
        at com.google.common.util.concurrent.SettableFuture.setException(SettableFuture.java:68)[65:com.google.guava:18.0.0]
        at org.opendaylight.controller.md.sal.dom.broker.impl.PingPongTransaction.onFailure(PingPongTransaction.java:65)[171:org.opendaylight.controller.sal-broker-impl:1.5.2.Carbon]
        at org.opendaylight.controller.md.sal.dom.broker.impl.PingPongTransactionChain.transactionFailed(PingPongTransactionChain.java:293)[171:org.opendaylight.controller.sal-broker-impl:1.5.2.Carbon]
        at org.opendaylight.controller.md.sal.dom.broker.impl.PingPongTransactionChain$2.onFailure(PingPongTransactionChain.java:243)[171:org.opendaylight.controller.sal-broker-impl:1.5.2.Carbon]
        at com.google.common.util.concurrent.Futures$6.run(Futures.java:1310)[65:com.google.guava:18.0.0]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)[:1.8.0_141]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)[:1.8.0_141]
        at java.lang.Thread.run(Thread.java:748)[:1.8.0_141]

Comment 1 Sridhar Gaddam 2017-09-20 13:41:04 UTC
Upstream Nitrogen: https://git.opendaylight.org/gerrit/#/c/63236
Stable/Carbon: https://git.opendaylight.org/gerrit/#/c/62674/

For info, upstream patch is now merged.

Comment 8 Sai Sindhur Malleni 2018-05-25 03:09:37 UTC
Tested with puddle 2018-05-23.1 and opendaylight rpm opendaylight-8.0.0-10.el7ost.noarch
. Could not reproduce. Verified that it has been fixed.

Comment 11 errata-xmlrpc 2018-06-27 13:37:30 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2018:2086


Note You need to log in before you can comment on or make changes to this bug.