Bug 1586169 - [Infra] NPE seen at org.opendaylight.netvirt.fibmanager.FibRpcServiceImpl.createFibEntry
Summary: [Infra] NPE seen at org.opendaylight.netvirt.fibmanager.FibRpcServiceImpl.cre...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: opendaylight
Version: 13.0 (Queens)
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: z1
: 13.0 (Queens)
Assignee: Josh Hershberg
QA Contact: Tomas Jamrisko
URL:
Whiteboard: Infra
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-06-05 16:01 UTC by Tim Rozet
Modified: 2018-10-18 07:23 UTC (History)
10 users (show)

Fixed In Version: opendaylight-8.3.0-1.el7ost
Doc Type: Bug Fix
Doc Text:
Missing parameters from createFibEntry generate a Null Pointer Exception (NPE) during NAT setup. This bug may result in missing FIB entries from the routing table, causing NAT or routing to fail. This update adds the proper parameters to the RPC call. NPE is no longer seen in the OpenDaylight log, and NAT and routing function correctly.
Clone Of:
Environment:
N/A
Last Closed: 2018-07-19 13:53:05 UTC
Target Upstream Version:
Embargoed:
trozet: needinfo-

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
OpenDaylight Bug NETVIRT-1133 0 None None None 2018-06-14 11:01:51 UTC
OpenDaylight gerrit 73293 0 None None None 2018-06-21 08:11:30 UTC
OpenDaylight gerrit 73294 0 None None None 2018-06-21 08:17:06 UTC
Red Hat Product Errata RHBA-2018:2215 0 None None None 2018-07-19 13:53:47 UTC

Description Tim Rozet 2018-06-05 16:01:10 UTC 
Jun 05 13:28:22 controller-0 dockerd-current[19119]: java.lang.NullPointerException: null
Jun 05 13:28:22 controller-0 dockerd-current[19119]:         at org.opendaylight.netvirt.fibmanager.FibRpcServiceImpl.createFibEntry(FibRpcServiceImpl.java:91) [349:org.opendaylight.netvirt.fibmanager-impl:0.6.0.redhat-9]
Jun 05 13:28:22 controller-0 dockerd-current[19119]:         at org.opendaylight.yangtools.yang.binding.util.RpcMethodInvokerWithInput.invokeOn(RpcMethodInvokerWithInput.java:32) [326:org.opendaylight.mdsal.yang-binding:0.12.0.redhat-9]
Jun 05 13:28:22 controller-0 dockerd-current[19119]:         at org.opendaylight.yangtools.yang.binding.util.AbstractMappedRpcInvoker.invokeRpc(AbstractMappedRpcInvoker.java:53) [326:org.opendaylight.mdsal.yang-binding:0.12.0.redhat-9]
Jun 05 13:28:22 controller-0 dockerd-current[19119]:         at org.opendaylight.controller.md.sal.binding.impl.BindingDOMRpcImplementationAdapter.invoke(BindingDOMRpcImplementationAdapter.java:83) [216:org.opendaylight.controller.sal-binding-broker-impl:1.7.0.redhat-9]
Jun 05 13:28:22 controller-0 dockerd-current[19119]:         at org.opendaylight.controller.md.sal.binding.impl.BindingDOMRpcImplementationAdapter.invokeRpc(BindingDOMRpcImplementationAdapter.java:70) [216:org.opendaylight.controller.sal-binding-broker-impl:1.7.0.redhat-9]
Jun 05 13:28:22 controller-0 dockerd-current[19119]:         at org.opendaylight.controller.md.sal.dom.broker.impl.GlobalDOMRpcRoutingTableEntry.invokeRpc(GlobalDOMRpcRoutingTableEntry.java:41) [219:org.opendaylight.controller.sal-broker-impl:1.7.0.redhat-9]
Jun 05 13:28:22 controller-0 dockerd-current[19119]:         at org.opendaylight.controller.md.sal.dom.broker.impl.DOMRpcRoutingTable.invokeRpc(DOMRpcRoutingTable.java:178) [219:org.opendaylight.controller.sal-broker-impl:1.7.0.redhat-9]
Jun 05 13:28:22 controller-0 dockerd-current[19119]:         at org.opendaylight.controller.md.sal.dom.broker.impl.DOMRpcRouter.invokeRpc(DOMRpcRouter.java:102) [219:org.opendaylight.controller.sal-broker-impl:1.7.0.redhat-9]
Jun 05 13:28:22 controller-0 dockerd-current[19119]:         at Proxydd0e6907_5e3b_4eba_a05a_3f239c44a90a.invokeRpc(Unknown Source) [?:?]
Jun 05 13:28:22 controller-0 dockerd-current[19119]:         at Proxyfae6851b_24a0_448f_a524_4b1638c5d6c5.invokeRpc(Unknown Source) [?:?]
Jun 05 13:28:22 controller-0 dockerd-current[19119]:         at org.opendaylight.controller.md.sal.binding.impl.RpcServiceAdapter.invoke0(RpcServiceAdapter.java:65) [216:org.opendaylight.controller.sal-binding-broker-impl:1.7.0.redhat-9]
Jun 05 13:28:22 controller-0 dockerd-current[19119]:         at org.opendaylight.controller.md.sal.binding.impl.RpcServiceAdapter.access$000(RpcServiceAdapter.java:43) [216:org.opendaylight.controller.sal-binding-broker-impl:1.7.0.redhat-9]
Jun 05 13:28:22 controller-0 dockerd-current[19119]:         at org.opendaylight.controller.md.sal.binding.impl.RpcServiceAdapter$RpcInvocationStrategy.invoke(RpcServiceAdapter.java:156) [216:org.opendaylight.controller.sal-binding-broker-impl:1.7.0.redhat-9]
Jun 05 13:28:22 controller-0 dockerd-current[19119]:         at org.opendaylight.controller.md.sal.binding.impl.RpcServiceAdapter.invoke(RpcServiceAdapter.java:96) [216:org.opendaylight.controller.sal-binding-broker-impl:1.7.0.redhat-9]
Jun 05 13:28:22 controller-0 dockerd-current[19119]:         at com.sun.proxy.$Proxy316.createFibEntry(Unknown Source) [348:org.opendaylight.netvirt.fibmanager-api:0.6.0.redhat-9]
Jun 05 13:28:22 controller-0 dockerd-current[19119]:         at org.opendaylight.netvirt.natservice.internal.NatTunnelInterfaceStateListener.hndlTepAddForDnatInEachRtr(NatTunnelInterfaceStateListener.java:898) [356:org.opendaylight.netvirt.natservice-impl:0.6.0.redhat-9]
Jun 05 13:28:22 controller-0 dockerd-current[19119]:         at org.opendaylight.netvirt.natservice.internal.NatTunnelInterfaceStateListener.hndlTepAddForAllRtrs(NatTunnelInterfaceStateListener.java:412) [356:org.opendaylight.netvirt.natservice-impl:0.6.0.redhat-9]
Jun 05 13:28:22 controller-0 dockerd-current[19119]:         at org.opendaylight.netvirt.natservice.internal.NatTunnelInterfaceStateListener.hndlTepEvntsForDpn(NatTunnelInterfaceStateListener.java:348) [356:org.opendaylight.netvirt.natservice-impl:0.6.0.redhat-9]
Jun 05 13:28:22 controller-0 dockerd-current[19119]:         at org.opendaylight.netvirt.natservice.internal.NatTunnelInterfaceStateListener.add(NatTunnelInterfaceStateListener.java:186) [356:org.opendaylight.netvirt.natservice-impl:0.6.0.redhat-9]
Jun 05 13:28:22 controller-0 dockerd-current[19119]:         at org.opendaylight.netvirt.natservice.internal.NatTunnelInterfaceStateListener.add(NatTunnelInterfaceStateListener.java:82) [356:org.opendaylight.netvirt.natservice-impl:0.6.0.redhat-9]
Jun 05 13:28:22 controller-0 dockerd-current[19119]:         at org.opendaylight.genius.datastoreutils.AsyncDataTreeChangeListenerBase$DataTreeChangeHandler.run(AsyncDataTreeChangeListenerBase.java:167) [259:org.opendaylight.genius.mdsalutil-api:0.4.0.redhat-9]
Jun 05 13:28:22 controller-0 dockerd-current[19119]:         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:?]
Jun 05 13:28:22 controller-0 dockerd-current[19119]:         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:?]
Jun 05 13:28:22 controller-0 dockerd-current[19119]:         at java.lang.Thread.run(Thread.java:748) [?:?]



Jun 05 13:31:13 controller-0 dockerd-current[19119]: java.util.concurrent.TimeoutException: Delivery of system messages timed out and they were dropped.
Jun 05 13:31:13 controller-0 dockerd-current[19119]:         at akka.remote.ReliableDeliverySupervisor$$anonfun$gated$1.applyOrElse(Endpoint.scala:346) [45:com.typesafe.akka.remote:2.5.4]
Jun 05 13:31:13 controller-0 dockerd-current[19119]:         at akka.actor.Actor.aroundReceive(Actor.scala:514) [40:com.typesafe.akka.actor:2.5.4]
Jun 05 13:31:13 controller-0 dockerd-current[19119]:         at akka.actor.Actor.aroundReceive$(Actor.scala:512) [40:com.typesafe.akka.actor:2.5.4]
Jun 05 13:31:13 controller-0 dockerd-current[19119]:         at akka.remote.ReliableDeliverySupervisor.aroundReceive(Endpoint.scala:203) [45:com.typesafe.akka.remote:2.5.4]
Jun 05 13:31:13 controller-0 dockerd-current[19119]:         at akka.actor.ActorCell.receiveMessage(ActorCell.scala:527) [40:com.typesafe.akka.actor:2.5.4]
Jun 05 13:31:13 controller-0 dockerd-current[19119]:         at akka.actor.ActorCell.invoke(ActorCell.scala:496) [40:com.typesafe.akka.actor:2.5.4]
Jun 05 13:31:13 controller-0 dockerd-current[19119]:         at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257) [40:com.typesafe.akka.actor:2.5.4]
Jun 05 13:31:13 controller-0 dockerd-current[19119]:         at akka.dispatch.Mailbox.run(Mailbox.scala:224) [40:com.typesafe.akka.actor:2.5.4]
Jun 05 13:31:13 controller-0 dockerd-current[19119]:         at akka.dispatch.Mailbox.exec(Mailbox.scala:234) [40:com.typesafe.akka.actor:2.5.4]
Jun 05 13:31:13 controller-0 dockerd-current[19119]:         at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) [40:com.typesafe.akka.actor:2.5.4]
Jun 05 13:31:13 controller-0 dockerd-current[19119]:         at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) [40:com.typesafe.akka.actor:2.5.4]
Jun 05 13:31:13 controller-0 dockerd-current[19119]:         at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) [40:com.typesafe.akka.actor:2.5.4]
Jun 05 13:31:13 controller-0 dockerd-current[19119]:         at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) [40:com.typesafe.akka.actor:2.5.4]
Comment 4 Michael Vorburger 2018-06-14 11:01:52 UTC 
Upstream NETVIRT-1133 seems to be the same, but was closed with Cannot Reproduce. Let's open that one if we have a reliable way to reproduce it here?
Comment 5 Tim Rozet 2018-06-14 14:27:08 UTC 
Yeah I saw it again last night after running OOO update.
Comment 6 Aswin Suryanarayanan 2018-06-14 16:36:04 UTC 
What OOO update do? Does it delete neutron resources ?
Comment 7 Tim Rozet 2018-06-15 18:09:14 UTC 
No, it restarts each ODL sequentially and updates it to a newer version.  I'm not 100% certain it is the same NPE, but I saw an NPE in Fib and figured it was the same one was I was glancing through the logs post update. I think there were 2 nova instances at the time which had FIP associated with them before update. We can try to reproduce.
Comment 8 Tim Rozet 2018-06-15 18:09:14 UTC 
No, it restarts each ODL sequentially and updates it to a newer version.  I'm not 100% certain it is the same NPE, but I saw an NPE in Fib and figured it was the same one was I was glancing through the logs post update. I think there were 2 nova instances at the time which had FIP associated with them before update. We can try to reproduce.
Comment 9 Josh Hershberg 2018-06-21 08:17:40 UTC 
Gerrit patches posted upstream. see attachments
Comment 21 Janki 2018-07-17 04:17:17 UTC 
I don't see this exception in karaf logs with opendaylight-8.3.0-1.el7ost.noarch.
Comment 23 errata-xmlrpc 2018-07-19 13:53:05 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:2215
Note You need to log in before you can comment on or make changes to this bug.