Description of problem: When running a Browbeat+Rally use case in my downstream scale CI which does: Create a network Create a sbunet Create a router Attach router to subnet and oublic network Boot VM with floating IP Ping VM with concurrency 8 and times set to 50 we see that some VMs remain unpingable even after 300 seconds and we see a lot of the following exceptions in karaf logs ERROR | opendaylight-cluster-data-akka.actor.default-dispatcher-646 | LocalThreePhaseCommitCohort | 233 - org.opendaylight.controller.sal-distributed-datastore - 1.7.0.redhat-5 | Failed to prepare transaction member-1-datastore-operational-fe-0-txn-94468-0 on backend org.opendaylight.controller.md.sal.common.api.data.OptimisticLockFailedException: Optimistic lock failed. at org.opendaylight.controller.cluster.datastore.ShardDataTree.lambda$processNextPendingTransaction$0(ShardDataTree.java:740) ~[233:org.opendaylight.controller.sal-distributed-datastore:1.7.0.redhat-5] at org.opendaylight.controller.cluster.datastore.ShardDataTree.processNextPending(ShardDataTree.java:778) [233:org.opendaylight.controller.sal-distributed-datastore:1.7.0.redhat-5] at org.opendaylight.controller.cluster.datastore.ShardDataTree.processNextPendingTransaction(ShardDataTree.java:725) [233:org.opendaylight.controller.sal-distributed-datastore:1.7.0.redhat-5] at org.opendaylight.controller.cluster.datastore.ShardDataTree.startCanCommit(ShardDataTree.java:808) [233:org.opendaylight.controller.sal-distributed-datastore:1.7.0.redhat-5] at org.opendaylight.controller.cluster.datastore.SimpleShardDataTreeCohort.canCommit(SimpleShardDataTreeCohort.java:84) [233:org.opendaylight.controller.sal-distributed-datastore:1.7.0.redhat-5] at org.opendaylight.controller.cluster.datastore.CohortEntry.canCommit(CohortEntry.java:97) [233:org.opendaylight.controller.sal-distributed-datastore:1.7.0.redhat-5] at org.opendaylight.controller.cluster.datastore.ShardCommitCoordinator.handleCanCommit(ShardCommitCoordinator.java:236) [233:org.opendaylight.controller.sal-distributed-datastore:1.7.0.redhat-5] at org.opendaylight.controller.cluster.datastore.ShardCommitCoordinator.handleReadyLocalTransaction(ShardCommitCoordinator.java:200) [233:org.opendaylight.controller.sal-distributed-datastore:1.7.0.redhat-5] at org.opendaylight.controller.cluster.datastore.Shard.handleReadyLocalTransaction(Shard.java:731) [233:org.opendaylight.controller.sal-distributed-datastore:1.7.0.redhat-5] at org.opendaylight.controller.cluster.datastore.Shard.handleNonRaftCommand(Shard.java:333) [233:org.opendaylight.controller.sal-distributed-datastore:1.7.0.redhat-5] at org.opendaylight.controller.cluster.raft.RaftActor.handleCommand(RaftActor.java:270) [218:org.opendaylight.controller.sal-akka-raft:1.7.0.redhat-5] at org.opendaylight.controller.cluster.common.actor.AbstractUntypedPersistentActor.onReceiveCommand(AbstractUntypedPersistentActor.java:44) [226:org.opendaylight.controller.sal-clustering-commons:1.7.0.redhat-5] at akka.persistence.UntypedPersistentActor.onReceive(PersistentActor.scala:274) [43:com.typesafe.akka.persistence:2.5.4] at org.opendaylight.controller.cluster.common.actor.MeteringBehavior.apply(MeteringBehavior.java:104) [226:org.opendaylight.controller.sal-clustering-commons:1.7.0.redhat-5] at akka.actor.ActorCell$$anonfun$become$1.applyOrElse(ActorCell.scala:545) [40:com.typesafe.akka.actor:2.5.4] at akka.actor.Actor.aroundReceive(Actor.scala:514) [40:com.typesafe.akka.actor:2.5.4] at akka.actor.Actor.aroundReceive$(Actor.scala:512) [40:com.typesafe.akka.actor:2.5.4] at akka.persistence.UntypedPersistentActor.akka$persistence$Eventsourced$$super$aroundReceive(PersistentActor.scala:272) [43:com.typesafe.akka.persistence:2.5.4] at akka.persistence.Eventsourced$$anon$1.stateReceive(Eventsourced.scala:663) [43:com.typesafe.akka.persistence:2.5.4] at akka.persistence.Eventsourced.aroundReceive(Eventsourced.scala:183) [43:com.typesafe.akka.persistence:2.5.4] at akka.persistence.Eventsourced.aroundReceive$(Eventsourced.scala:182) [43:com.typesafe.akka.persistence:2.5.4] at akka.persistence.UntypedPersistentActor.aroundReceive(PersistentActor.scala:272) [43:com.typesafe.akka.persistence:2.5.4] at akka.actor.ActorCell.receiveMessage(ActorCell.scala:527) [40:com.typesafe.akka.actor:2.5.4] at akka.actor.ActorCell.invoke(ActorCell.scala:496) [40:com.typesafe.akka.actor:2.5.4] at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257) [40:com.typesafe.akka.actor:2.5.4] at akka.dispatch.Mailbox.run(Mailbox.scala:224) [40:com.typesafe.akka.actor:2.5.4] at akka.dispatch.Mailbox.exec(Mailbox.scala:234) [40:com.typesafe.akka.actor:2.5.4] at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) [40:com.typesafe.akka.actor:2.5.4] at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) [40:com.typesafe.akka.actor:2.5.4] at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) [40:com.typesafe.akka.actor:2.5.4] at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) [40:com.typesafe.akka.actor:2.5.4] Caused by: org.opendaylight.yangtools.yang.data.api.schema.tree.ConflictingModificationAppliedException: Node was deleted by other transaction. at org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractNodeContainerModificationStrategy.checkTouchApplicable(AbstractNodeContainerModificationStrategy.java:287) ~[430:org.opendaylight.yangtools.yang-data-impl:2.0.1.redhat-5] at org.opendaylight.yangtools.yang.data.impl.schema.tree.SchemaAwareApplyOperation.checkApplicable(SchemaAwareApplyOperation.java:132) ~[430:org.opendaylight.yangtools.yang-data-impl:2.0.1.redhat-5] at org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractNodeContainerModificationStrategy.checkChildPreconditions(AbstractNodeContainerModificationStrategy.java:307) ~[430:org.opendaylight.yangtools.yang-data-impl:2.0.1.redhat-5] at org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractNodeContainerModificationStrategy.checkTouchApplicable(AbstractNodeContainerModificationStrategy.java:290) ~[430:org.opendaylight.yangtools.yang-data-impl:2.0.1.redhat-5] at org.opendaylight.yangtools.yang.data.impl.schema.tree.SchemaAwareApplyOperation.checkApplicable(SchemaAwareApplyOperation.java:132) ~[430:org.opendaylight.yangtools.yang-data-impl:2.0.1.redhat-5] at org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractNodeContainerModificationStrategy.checkChildPreconditions(AbstractNodeContainerModificationStrategy.java:307) ~[430:org.opendaylight.yangtools.yang-data-impl:2.0.1.redhat-5] at org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractNodeContainerModificationStrategy.checkTouchApplicable(AbstractNodeContainerModificationStrategy.java:290) ~[430:org.opendaylight.yangtools.yang-data-impl:2.0.1.redhat-5] at org.opendaylight.yangtools.yang.data.impl.schema.tree.SchemaAwareApplyOperation.checkApplicable(SchemaAwareApplyOperation.java:132) ~[430:org.opendaylight.yangtools.yang-data-impl:2.0.1.redhat-5] at org.opendaylight.yangtools.yang.data.impl.schema.tree.StructuralContainerModificationStrategy.checkApplicable(StructuralContainerModificationStrategy.java:101) ~[430:org.opendaylight.yangtools.yang-data-impl:2.0.1.redhat-5] at org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractNodeContainerModificationStrategy.checkChildPreconditions(AbstractNodeContainerModificationStrategy.java:307) ~[430:org.opendaylight.yangtools.yang-data-impl:2.0.1.redhat-5] at org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractNodeContainerModificationStrategy.checkTouchApplicable(AbstractNodeContainerModificationStrategy.java:290) ~[430:org.opendaylight.yangtools.yang-data-impl:2.0.1.redhat-5] at org.opendaylight.yangtools.yang.data.impl.schema.tree.SchemaAwareApplyOperation.checkApplicable(SchemaAwareApplyOperation.java:132) ~[430:org.opendaylight.yangtools.yang-data-impl:2.0.1.redhat-5] at org.opendaylight.yangtools.yang.data.impl.schema.tree.RootModificationApplyOperation.checkApplicable(RootModificationApplyOperation.java:72) ~[430:org.opendaylight.yangtools.yang-data-impl:2.0.1.redhat-5] at org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractDataTreeTip.validate(AbstractDataTreeTip.java:35) ~[430:org.opendaylight.yangtools.yang-data-impl:2.0.1.redhat-5] at org.opendaylight.controller.cluster.datastore.ShardDataTree.lambda$processNextPendingTransaction$0(ShardDataTree.java:732) ~[233:org.opendaylight.controller.sal-distributed-datastore:1.7.0.redhat-5] ... 30 more Version-Release number of selected component (if applicable): OSP 13 opendaylight-8.0.0-5.el7ost.noarch How reproducible: Steps to Reproduce: 100% 1. Install OSP13 + ODL 2. Run scale use case as described above 3. Actual results: Seeing exceptions in karag logs and some vms remain unpingable Expected results: No exceptions and all VMs should be pingable Additional info:
*** This bug has been marked as a duplicate of bug 1567509 ***