Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1586169

Summary: [Infra] NPE seen at org.opendaylight.netvirt.fibmanager.FibRpcServiceImpl.createFibEntry
Product: Red Hat OpenStack Reporter: Tim Rozet <trozet>
Component: opendaylightAssignee: Josh Hershberg <jhershbe>
Status: CLOSED ERRATA QA Contact: Tomas Jamrisko <tjamrisk>
Severity: high Docs Contact:
Priority: high    
Version: 13.0 (Queens)CC: aadam, jchhatba, jhershbe, lmarsh, mkolesni, nmanos, nyechiel, tjamrisk, trozet, vorburger
Target Milestone: z1Keywords: Triaged, ZStream
Target Release: 13.0 (Queens)Flags: trozet: needinfo-
Hardware: Unspecified   
OS: Unspecified   
Whiteboard: Infra
Fixed In Version: opendaylight-8.3.0-1.el7ost Doc Type: Bug Fix
Doc Text:
Missing parameters from createFibEntry generate a Null Pointer Exception (NPE) during NAT setup. This bug may result in missing FIB entries from the routing table, causing NAT or routing to fail. This update adds the proper parameters to the RPC call. NPE is no longer seen in the OpenDaylight log, and NAT and routing function correctly.
Story Points: ---
Clone Of: Environment:
N/A
Last Closed: 2018-07-19 13:53:05 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Tim Rozet 2018-06-05 16:01:10 UTC
Jun 05 13:28:22 controller-0 dockerd-current[19119]: java.lang.NullPointerException: null
Jun 05 13:28:22 controller-0 dockerd-current[19119]:         at org.opendaylight.netvirt.fibmanager.FibRpcServiceImpl.createFibEntry(FibRpcServiceImpl.java:91) [349:org.opendaylight.netvirt.fibmanager-impl:0.6.0.redhat-9]
Jun 05 13:28:22 controller-0 dockerd-current[19119]:         at org.opendaylight.yangtools.yang.binding.util.RpcMethodInvokerWithInput.invokeOn(RpcMethodInvokerWithInput.java:32) [326:org.opendaylight.mdsal.yang-binding:0.12.0.redhat-9]
Jun 05 13:28:22 controller-0 dockerd-current[19119]:         at org.opendaylight.yangtools.yang.binding.util.AbstractMappedRpcInvoker.invokeRpc(AbstractMappedRpcInvoker.java:53) [326:org.opendaylight.mdsal.yang-binding:0.12.0.redhat-9]
Jun 05 13:28:22 controller-0 dockerd-current[19119]:         at org.opendaylight.controller.md.sal.binding.impl.BindingDOMRpcImplementationAdapter.invoke(BindingDOMRpcImplementationAdapter.java:83) [216:org.opendaylight.controller.sal-binding-broker-impl:1.7.0.redhat-9]
Jun 05 13:28:22 controller-0 dockerd-current[19119]:         at org.opendaylight.controller.md.sal.binding.impl.BindingDOMRpcImplementationAdapter.invokeRpc(BindingDOMRpcImplementationAdapter.java:70) [216:org.opendaylight.controller.sal-binding-broker-impl:1.7.0.redhat-9]
Jun 05 13:28:22 controller-0 dockerd-current[19119]:         at org.opendaylight.controller.md.sal.dom.broker.impl.GlobalDOMRpcRoutingTableEntry.invokeRpc(GlobalDOMRpcRoutingTableEntry.java:41) [219:org.opendaylight.controller.sal-broker-impl:1.7.0.redhat-9]
Jun 05 13:28:22 controller-0 dockerd-current[19119]:         at org.opendaylight.controller.md.sal.dom.broker.impl.DOMRpcRoutingTable.invokeRpc(DOMRpcRoutingTable.java:178) [219:org.opendaylight.controller.sal-broker-impl:1.7.0.redhat-9]
Jun 05 13:28:22 controller-0 dockerd-current[19119]:         at org.opendaylight.controller.md.sal.dom.broker.impl.DOMRpcRouter.invokeRpc(DOMRpcRouter.java:102) [219:org.opendaylight.controller.sal-broker-impl:1.7.0.redhat-9]
Jun 05 13:28:22 controller-0 dockerd-current[19119]:         at Proxydd0e6907_5e3b_4eba_a05a_3f239c44a90a.invokeRpc(Unknown Source) [?:?]
Jun 05 13:28:22 controller-0 dockerd-current[19119]:         at Proxyfae6851b_24a0_448f_a524_4b1638c5d6c5.invokeRpc(Unknown Source) [?:?]
Jun 05 13:28:22 controller-0 dockerd-current[19119]:         at org.opendaylight.controller.md.sal.binding.impl.RpcServiceAdapter.invoke0(RpcServiceAdapter.java:65) [216:org.opendaylight.controller.sal-binding-broker-impl:1.7.0.redhat-9]
Jun 05 13:28:22 controller-0 dockerd-current[19119]:         at org.opendaylight.controller.md.sal.binding.impl.RpcServiceAdapter.access$000(RpcServiceAdapter.java:43) [216:org.opendaylight.controller.sal-binding-broker-impl:1.7.0.redhat-9]
Jun 05 13:28:22 controller-0 dockerd-current[19119]:         at org.opendaylight.controller.md.sal.binding.impl.RpcServiceAdapter$RpcInvocationStrategy.invoke(RpcServiceAdapter.java:156) [216:org.opendaylight.controller.sal-binding-broker-impl:1.7.0.redhat-9]
Jun 05 13:28:22 controller-0 dockerd-current[19119]:         at org.opendaylight.controller.md.sal.binding.impl.RpcServiceAdapter.invoke(RpcServiceAdapter.java:96) [216:org.opendaylight.controller.sal-binding-broker-impl:1.7.0.redhat-9]
Jun 05 13:28:22 controller-0 dockerd-current[19119]:         at com.sun.proxy.$Proxy316.createFibEntry(Unknown Source) [348:org.opendaylight.netvirt.fibmanager-api:0.6.0.redhat-9]
Jun 05 13:28:22 controller-0 dockerd-current[19119]:         at org.opendaylight.netvirt.natservice.internal.NatTunnelInterfaceStateListener.hndlTepAddForDnatInEachRtr(NatTunnelInterfaceStateListener.java:898) [356:org.opendaylight.netvirt.natservice-impl:0.6.0.redhat-9]
Jun 05 13:28:22 controller-0 dockerd-current[19119]:         at org.opendaylight.netvirt.natservice.internal.NatTunnelInterfaceStateListener.hndlTepAddForAllRtrs(NatTunnelInterfaceStateListener.java:412) [356:org.opendaylight.netvirt.natservice-impl:0.6.0.redhat-9]
Jun 05 13:28:22 controller-0 dockerd-current[19119]:         at org.opendaylight.netvirt.natservice.internal.NatTunnelInterfaceStateListener.hndlTepEvntsForDpn(NatTunnelInterfaceStateListener.java:348) [356:org.opendaylight.netvirt.natservice-impl:0.6.0.redhat-9]
Jun 05 13:28:22 controller-0 dockerd-current[19119]:         at org.opendaylight.netvirt.natservice.internal.NatTunnelInterfaceStateListener.add(NatTunnelInterfaceStateListener.java:186) [356:org.opendaylight.netvirt.natservice-impl:0.6.0.redhat-9]
Jun 05 13:28:22 controller-0 dockerd-current[19119]:         at org.opendaylight.netvirt.natservice.internal.NatTunnelInterfaceStateListener.add(NatTunnelInterfaceStateListener.java:82) [356:org.opendaylight.netvirt.natservice-impl:0.6.0.redhat-9]
Jun 05 13:28:22 controller-0 dockerd-current[19119]:         at org.opendaylight.genius.datastoreutils.AsyncDataTreeChangeListenerBase$DataTreeChangeHandler.run(AsyncDataTreeChangeListenerBase.java:167) [259:org.opendaylight.genius.mdsalutil-api:0.4.0.redhat-9]
Jun 05 13:28:22 controller-0 dockerd-current[19119]:         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:?]
Jun 05 13:28:22 controller-0 dockerd-current[19119]:         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:?]
Jun 05 13:28:22 controller-0 dockerd-current[19119]:         at java.lang.Thread.run(Thread.java:748) [?:?]



Jun 05 13:31:13 controller-0 dockerd-current[19119]: java.util.concurrent.TimeoutException: Delivery of system messages timed out and they were dropped.
Jun 05 13:31:13 controller-0 dockerd-current[19119]:         at akka.remote.ReliableDeliverySupervisor$$anonfun$gated$1.applyOrElse(Endpoint.scala:346) [45:com.typesafe.akka.remote:2.5.4]
Jun 05 13:31:13 controller-0 dockerd-current[19119]:         at akka.actor.Actor.aroundReceive(Actor.scala:514) [40:com.typesafe.akka.actor:2.5.4]
Jun 05 13:31:13 controller-0 dockerd-current[19119]:         at akka.actor.Actor.aroundReceive$(Actor.scala:512) [40:com.typesafe.akka.actor:2.5.4]
Jun 05 13:31:13 controller-0 dockerd-current[19119]:         at akka.remote.ReliableDeliverySupervisor.aroundReceive(Endpoint.scala:203) [45:com.typesafe.akka.remote:2.5.4]
Jun 05 13:31:13 controller-0 dockerd-current[19119]:         at akka.actor.ActorCell.receiveMessage(ActorCell.scala:527) [40:com.typesafe.akka.actor:2.5.4]
Jun 05 13:31:13 controller-0 dockerd-current[19119]:         at akka.actor.ActorCell.invoke(ActorCell.scala:496) [40:com.typesafe.akka.actor:2.5.4]
Jun 05 13:31:13 controller-0 dockerd-current[19119]:         at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257) [40:com.typesafe.akka.actor:2.5.4]
Jun 05 13:31:13 controller-0 dockerd-current[19119]:         at akka.dispatch.Mailbox.run(Mailbox.scala:224) [40:com.typesafe.akka.actor:2.5.4]
Jun 05 13:31:13 controller-0 dockerd-current[19119]:         at akka.dispatch.Mailbox.exec(Mailbox.scala:234) [40:com.typesafe.akka.actor:2.5.4]
Jun 05 13:31:13 controller-0 dockerd-current[19119]:         at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) [40:com.typesafe.akka.actor:2.5.4]
Jun 05 13:31:13 controller-0 dockerd-current[19119]:         at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) [40:com.typesafe.akka.actor:2.5.4]
Jun 05 13:31:13 controller-0 dockerd-current[19119]:         at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) [40:com.typesafe.akka.actor:2.5.4]
Jun 05 13:31:13 controller-0 dockerd-current[19119]:         at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) [40:com.typesafe.akka.actor:2.5.4]

Comment 4 Michael Vorburger 2018-06-14 11:01:52 UTC
Upstream NETVIRT-1133 seems to be the same, but was closed with Cannot Reproduce. Let's open that one if we have a reliable way to reproduce it here?

Comment 5 Tim Rozet 2018-06-14 14:27:08 UTC
Yeah I saw it again last night after running OOO update.

Comment 6 Aswin Suryanarayanan 2018-06-14 16:36:04 UTC
What OOO update do? Does it delete neutron resources ?

Comment 7 Tim Rozet 2018-06-15 18:09:14 UTC
No, it restarts each ODL sequentially and updates it to a newer version.  I'm not 100% certain it is the same NPE, but I saw an NPE in Fib and figured it was the same one was I was glancing through the logs post update. I think there were 2 nova instances at the time which had FIP associated with them before update. We can try to reproduce.

Comment 8 Tim Rozet 2018-06-15 18:09:14 UTC
No, it restarts each ODL sequentially and updates it to a newer version.  I'm not 100% certain it is the same NPE, but I saw an NPE in Fib and figured it was the same one was I was glancing through the logs post update. I think there were 2 nova instances at the time which had FIP associated with them before update. We can try to reproduce.

Comment 9 Josh Hershberg 2018-06-21 08:17:40 UTC
Gerrit patches posted upstream. see attachments

Comment 21 Janki 2018-07-17 04:17:17 UTC
I don't see this exception in karaf logs with opendaylight-8.3.0-1.el7ost.noarch.

Comment 23 errata-xmlrpc 2018-07-19 13:53:05 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:2215