Description of problem:

As described in Bug https://jira.opendaylight.org/browse/NETVIRT-362 :

ModifiedNodeDoesNotExistException: Node /(urn:opendaylight:netvirt:fibmanager?revision=2015-03-30)fibEntries/vrfTables does not exist. Cannot apply modification to its children.

(^ Bug was closed as not reproduced).

Version-Release number of selected component (if applicable):
14   -p 2018-12-20.4

How reproducible:
Sometimes

Steps to Reproduce:
1. Install OSP + ODL
2. Run CSIT tests
2. Check karaf.log for "ModifiedNodeDoesNotExistException" on all controllers.


Actual results:


*** Found on controller-2: /var/log/containers/opendaylight/karaf/logs/karaf.log ***

2018-12-23T13:57:13,689 | DEBUG | ForkJoinPool-1-worker-3 | org.opendaylight.netvirt.vpnmanager.VpnInterfaceManager      | VpnInterfaceManager: VrfEntries for 49154815186021:br-ex-patch:trunk removed successfully
2018-12-23T13:57:13,690 | ERROR | ForkJoinPool-1-worker-3 | org.opendaylight.infrautils.jobcoordinator.internal.JobCoordinatorImpl | Job still failed on final retry: JobEntry{key='VPNINTERFACE-49154815186021:br-ex-patch:trunk', mainWorker=org.opendaylight.netvirt.vpnmanager.VpnInterfaceManager$$Lambda$1925/63775466@3c8886f4, rollbackWorker=null, retryCount=3/3, futures=[com.google.common.util.concurrent.ImmediateFuture$ImmediateFailedFuture@572bcc3c[status=FAILURE, cause=[java.lang.RuntimeException: TransactionCommitFailedException{message=Data did not pass validation for path /(urn:opendaylight:netvirt:fibmanager?revision=2015-03-30)fibEntries/vrfTables/vrfTables[{(urn:opendaylight:netvirt:fibmanager?revision=2015-03-30)routeDistinguisher=5e753beb-62cb-4ce8-9c3a-c0b7a4341c0f}], errorList=[RpcError [message=Data did not pass validation for path /(urn:opendaylight:netvirt:fibmanager?revision=2015-03-30)fibEntries/vrfTables/vrfTables[{(urn:opendaylight:netvirt:fibmanager?revision=2015-03-30)routeDistinguisher=5e753beb-62cb-4ce8-9c3a-c0b7a4341c0f}], severity=ERROR, errorType=APPLICATION, tag=operation-failed, applicationTag=null, info=null, cause=org.opendaylight.yangtools.yang.data.api.schema.tree.ModifiedNodeDoesNotExistException: Node /(urn:opendaylight:netvirt:fibmanager?revision=2015-03-30)fibEntries/vrfTables/vrfTables[{(urn:opendaylight:netvirt:fibmanager?revision=2015-03-30)routeDistinguisher=5e753beb-62cb-4ce8-9c3a-c0b7a4341c0f}] does not exist. Cannot apply modification to its children.]]}]], com.google.common.util.concurrent.ImmediateFuture$ImmediateSuccessfulCheckedFuture@4dc56aad[status=SUCCESS, result=[null]], com.google.common.util.concurrent.ImmediateFuture$ImmediateSuccessfulCheckedFuture@3ee3a25c[status=SUCCESS, result=[null]]]}
	... 30 more
	at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) [?:?]
	at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) [?:?]
	at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) [?:?]
	at java.util.concurrent.ForkJoinTask$RunnableExecuteAction.exec(ForkJoinTask.java:1402) [?:?]
	at org.opendaylight.controller.cluster.datastore.ShardDataTree.lambda$processNextPendingTransaction$0(ShardDataTree.java:740) ~[?:?]
	at org.opendaylight.genius.infra.ManagedNewTransactionRunnerImpl.callWithNewWriteOnlyTransactionAndSubmit(ManagedNewTransactionRunnerImpl.java:48) ~[?:?]
	at org.opendaylight.genius.mdsalutil.MDSALUtil.syncDelete(MDSALUtil.java:620) ~[?:?]
	at org.opendaylight.infrautils.jobcoordinator.internal.JobCoordinatorImpl$MainTask.runWithUncheckedExceptionLogging(JobCoordinatorImpl.java:416) [277:org.opendaylight.infrautils.jobcoordinator-impl:1.3.4.redhat-7]
	at org.opendaylight.infrautils.utils.concurrent.LoggingUncaughtThreadDeathContextRunnable.run(LoggingUncaughtThreadDeathContextRunnable.java:60) [283:org.opendaylight.infrautils.util:1.3.4.redhat-7]
	at org.opendaylight.netvirt.fibmanager.FibManagerImpl.removeFibEntry(FibManagerImpl.java:147) ~[?:?]
	at org.opendaylight.netvirt.fibmanager.FibUtil.removeFibEntry(FibUtil.java:395) ~[?:?]
	at org.opendaylight.netvirt.vpnmanager.VpnInterfaceManager.lambda$null$12(VpnInterfaceManager.java:1293) ~[?:?]
	at org.opendaylight.netvirt.vpnmanager.VpnInterfaceManager.lambda$null$13(VpnInterfaceManager.java:1247) ~[?:?]
	at org.opendaylight.netvirt.vpnmanager.VpnInterfaceManager.lambda$null$14(VpnInterfaceManager.java:1246) ~[?:?]
	at org.opendaylight.netvirt.vpnmanager.VpnInterfaceManager.lambda$removeVpnInterfaceFromVpn$15(VpnInterfaceManager.java:1245) ~[?:?]
	at org.opendaylight.netvirt.vpnmanager.VpnInterfaceManager.processVpnInterfaceDown(VpnInterfaceManager.java:1329) ~[?:?]
	at org.opendaylight.netvirt.vpnmanager.VpnInterfaceManager.removeAdjacenciesFromVpn(VpnInterfaceManager.java:1369) ~[?:?]
	at org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractDataTreeTip.validate(AbstractDataTreeTip.java:38) ~[?:?]
	at org.opendaylight.yangtools.yang.data.impl.schema.tree.RootModificationApplyOperation.checkApplicable(RootModificationApplyOperation.java:71) ~[?:?]
	at Proxy593224be_b506_4647_8b40_22dce2816927.removeFibEntry(Unknown Source) ~[?:?]
	at Proxyc108a86a_015e_4c10_8519_cf2e3e8047e4.removeFibEntry(Unknown Source) ~[?:?]
java.lang.RuntimeException: TransactionCommitFailedException{message=Data did not pass validation for path /(urn:opendaylight:netvirt:fibmanager?revision=2015-03-30)fibEntries/vrfTables/vrfTables[{(urn:opendaylight:netvirt:fibmanager?revision=2015-03-30)routeDistinguisher=5e753beb-62cb-4ce8-9c3a-c0b7a4341c0f}], errorList=[RpcError [message=Data did not pass validation for path /(urn:opendaylight:netvirt:fibmanager?revision=2015-03-30)fibEntries/vrfTables/vrfTables[{(urn:opendaylight:netvirt:fibmanager?revision=2015-03-30)routeDistinguisher=5e753beb-62cb-4ce8-9c3a-c0b7a4341c0f}], severity=ERROR, errorType=APPLICATION, tag=operation-failed, applicationTag=null, info=null, cause=org.opendaylight.yangtools.yang.data.api.schema.tree.ModifiedNodeDoesNotExistException: Node /(urn:opendaylight:netvirt:fibmanager?revision=2015-03-30)fibEntries/vrfTables/vrfTables[{(urn:opendaylight:netvirt:fibmanager?revision=2015-03-30)routeDistinguisher=5e753beb-62cb-4ce8-9c3a-c0b7a4341c0f}] does not exist. Cannot apply modification to its children.]]}
....(undercloud) [stack@undercloud-0 ~]$ 


Expected results:


Additional info:

ALL CSIT Tests passed:
https://rhos-qe-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/DFG-opendaylight-odl-netvirt-14_director-rhel-virthost-3cont_2comp-ipv4-vxlan-ha-csit/125/robot/report/report.html#totals?all

However, this error did occur, so it might not be caught during CSIT execution.