[reply] [−] Private Description Sai Sindhur Malleni 2018-07-17 21:19:32 EDT Description of problem: Running performance and scale tests that create 100s of neutron resources like networks, subnets, ports and routers and waiting for the environment to quiesce, we observe some new transaction leaks based on Michael Vorburger's mdsal-trace tool. 132x transactions opened here, which are not closed: (...) org.opendaylight.controller.md.sal.binding.impl.BindingDOMDataBrokerAdapter.newWriteOnlyTransaction(BindingDOMDataBrokerAdapter.java:69) (...) org.opendaylight.netvirt.vpnmanager.VpnInterfaceManager.lambda$addVpnInterfaceToVpn$0(VpnInterfaceManager.java:268) org.opendaylight.infrautils.jobcoordinator.internal.JobCoordinatorImpl$MainTask.runWithUncheckedExceptionLogging(JobCoordinatorImpl.java:416) org.opendaylight.infrautils.utils.concurrent.LoggingUncaughtThreadDeathContextRunnable.run(LoggingUncaughtThreadDeathContextRunnable.java:60) java.util.concurrent.ForkJoinTask$RunnableExecuteAction.exec(ForkJoinTask.java:1402) java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157) Version-Release number of selected component (if applicable): OSP13 Puddle: 2018-07-13.1 How reproducible: 100% Steps to Reproduce: 1. Run Browbeat+Rally neutron test suite 2. Get open transactions 3. Actual results: We see several open transactions after the tests finished running. Expected results: No open transactions Additional info:
https://gist.github.com/smalleni/63d486b66572199a55d1c29167c5e455 shows a (bigger; 1283x) leak elsewhere in VpnInterfaceManager (from lines 1523 *AND* 1524) which should also be looked into as part of this issue.
Added that to a comment here: https://bugzilla.redhat.com/show_bug.cgi?id=1602188
Stephen, is this one a full duplicate of Bug 1602188 and already fixed (sure?) upstream and thus coming with Oxygen SR3 (so just close as dupe?), or a separate issue that needs to be looked into?
(In reply to Michael Vorburger from comment #3) > Stephen, is this one a full duplicate of Bug 1602188 and already fixed > (sure?) upstream and thus coming with Oxygen SR3 (so just close as dupe?), > or a separate issue that needs to be looked into? No, it’s not a dupe, and it’s not fixed in Oxygen (but fixed in master, https://git.opendaylight.org/gerrit/71799).
Tested with rpm opendaylight-8.3.0-3.el7ost.noarch Still found open1 open transaction 1x transactions opened here, which are not closed: (...) org.opendaylight.controller.md.sal.binding.impl.BindingDOMDataBrokerAdapter.newWriteOnlyTransaction(BindingDOMDataBrokerAdapter.java:69) (...) org.opendaylight.netvirt.vpnmanager.VpnInterfaceManager.lambda$addVpnInterfaceToVpn$0(VpnInterfaceManager.java:269) org.opendaylight.infrautils.jobcoordinator.internal.JobCoordinatorImpl$MainTask.runWithUncheckedExceptionLogging(JobCoordinatorImpl.java:416) org.opendaylight.infrautils.utils.concurrent.LoggingUncaughtThreadDeathContextRunnable.run(LoggingUncaughtThreadDeathContextRunnable.java:60) java.util.concurrent.ForkJoinTask$RunnableExecuteAction.exec(ForkJoinTask.java:1402) java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157)
> Still found open transaction > 1x transactions opened here, which are not closed: we can probably ignore that, if it's just "1x" - we're really only concerned if you see hundreds of still open transactions after it should have closed them after a scale test.
According to our records, this should be resolved by opendaylight-8.3.0-3.el7ost. This build is available now.
Sai, Can you please conclude with Michael Vorburger if this is fixed or not?
I tested this with opendaylight-8.3.0-3.el7ost.noarch and on controller-1 and controller-2 I did not find this open transaction. However, Opendaylight seems to have died on controller-0.
(In reply to Sai Sindhur Malleni from comment #11) > I tested this with opendaylight-8.3.0-3.el7ost.noarch and on controller-1 > and controller-2 I did not find this open transaction. However, Opendaylight > seems to have died on controller-0. I suggest to move to verified since it seems the original problem has been solved.
Verified this with opendaylight-8.3.0-7.el7ost.noarch
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2018:3614