Bug 1567508 - Failed to prepare transaction - Optimistic Lock failed
Summary: Failed to prepare transaction - Optimistic Lock failed
Keywords:
Status: CLOSED DUPLICATE of bug 1567509
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: opendaylight
Version: 13.0 (Queens)
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
: ---
Assignee: lpeer
QA Contact: Itzik Brown
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-04-14 14:24 UTC by Sai Sindhur Malleni
Modified: 2018-10-24 12:37 UTC (History)
3 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
N/A
Last Closed: 2018-04-16 06:22:12 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)

Description Sai Sindhur Malleni 2018-04-14 14:24:30 UTC
Description of problem:
When running a Browbeat+Rally use case in my downstream scale CI which does:
Create a network
Create a sbunet
Create a router
Attach router to subnet and oublic network
Boot VM with floating IP
Ping VM

with concurrency 8 and times set to 50 we see that some VMs remain unpingable even after 300 seconds and we see a lot of the following exceptions in karaf logs

 ERROR | opendaylight-cluster-data-akka.actor.default-dispatcher-646 | LocalThreePhaseCommitCohort      | 233 - org.opendaylight.controller.sal-distributed-datastore - 1.7.0.redhat-5 | Failed to prepare transaction member-1-datastore-operational-fe-0-txn-94468-0 on backend
org.opendaylight.controller.md.sal.common.api.data.OptimisticLockFailedException: Optimistic lock failed.
        at org.opendaylight.controller.cluster.datastore.ShardDataTree.lambda$processNextPendingTransaction$0(ShardDataTree.java:740) ~[233:org.opendaylight.controller.sal-distributed-datastore:1.7.0.redhat-5]
        at org.opendaylight.controller.cluster.datastore.ShardDataTree.processNextPending(ShardDataTree.java:778) [233:org.opendaylight.controller.sal-distributed-datastore:1.7.0.redhat-5]
        at org.opendaylight.controller.cluster.datastore.ShardDataTree.processNextPendingTransaction(ShardDataTree.java:725) [233:org.opendaylight.controller.sal-distributed-datastore:1.7.0.redhat-5]
        at org.opendaylight.controller.cluster.datastore.ShardDataTree.startCanCommit(ShardDataTree.java:808) [233:org.opendaylight.controller.sal-distributed-datastore:1.7.0.redhat-5]
        at org.opendaylight.controller.cluster.datastore.SimpleShardDataTreeCohort.canCommit(SimpleShardDataTreeCohort.java:84) [233:org.opendaylight.controller.sal-distributed-datastore:1.7.0.redhat-5]
        at org.opendaylight.controller.cluster.datastore.CohortEntry.canCommit(CohortEntry.java:97) [233:org.opendaylight.controller.sal-distributed-datastore:1.7.0.redhat-5]
        at org.opendaylight.controller.cluster.datastore.ShardCommitCoordinator.handleCanCommit(ShardCommitCoordinator.java:236) [233:org.opendaylight.controller.sal-distributed-datastore:1.7.0.redhat-5]
        at org.opendaylight.controller.cluster.datastore.ShardCommitCoordinator.handleReadyLocalTransaction(ShardCommitCoordinator.java:200) [233:org.opendaylight.controller.sal-distributed-datastore:1.7.0.redhat-5]
        at org.opendaylight.controller.cluster.datastore.Shard.handleReadyLocalTransaction(Shard.java:731) [233:org.opendaylight.controller.sal-distributed-datastore:1.7.0.redhat-5]
        at org.opendaylight.controller.cluster.datastore.Shard.handleNonRaftCommand(Shard.java:333) [233:org.opendaylight.controller.sal-distributed-datastore:1.7.0.redhat-5]
        at org.opendaylight.controller.cluster.raft.RaftActor.handleCommand(RaftActor.java:270) [218:org.opendaylight.controller.sal-akka-raft:1.7.0.redhat-5]
        at org.opendaylight.controller.cluster.common.actor.AbstractUntypedPersistentActor.onReceiveCommand(AbstractUntypedPersistentActor.java:44) [226:org.opendaylight.controller.sal-clustering-commons:1.7.0.redhat-5]
        at akka.persistence.UntypedPersistentActor.onReceive(PersistentActor.scala:274) [43:com.typesafe.akka.persistence:2.5.4]
        at org.opendaylight.controller.cluster.common.actor.MeteringBehavior.apply(MeteringBehavior.java:104) [226:org.opendaylight.controller.sal-clustering-commons:1.7.0.redhat-5]
        at akka.actor.ActorCell$$anonfun$become$1.applyOrElse(ActorCell.scala:545) [40:com.typesafe.akka.actor:2.5.4]
        at akka.actor.Actor.aroundReceive(Actor.scala:514) [40:com.typesafe.akka.actor:2.5.4]
        at akka.actor.Actor.aroundReceive$(Actor.scala:512) [40:com.typesafe.akka.actor:2.5.4]
  at akka.persistence.UntypedPersistentActor.akka$persistence$Eventsourced$$super$aroundReceive(PersistentActor.scala:272) [43:com.typesafe.akka.persistence:2.5.4]
        at akka.persistence.Eventsourced$$anon$1.stateReceive(Eventsourced.scala:663) [43:com.typesafe.akka.persistence:2.5.4]
        at akka.persistence.Eventsourced.aroundReceive(Eventsourced.scala:183) [43:com.typesafe.akka.persistence:2.5.4]
        at akka.persistence.Eventsourced.aroundReceive$(Eventsourced.scala:182) [43:com.typesafe.akka.persistence:2.5.4]
        at akka.persistence.UntypedPersistentActor.aroundReceive(PersistentActor.scala:272) [43:com.typesafe.akka.persistence:2.5.4]
        at akka.actor.ActorCell.receiveMessage(ActorCell.scala:527) [40:com.typesafe.akka.actor:2.5.4]
        at akka.actor.ActorCell.invoke(ActorCell.scala:496) [40:com.typesafe.akka.actor:2.5.4]
        at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257) [40:com.typesafe.akka.actor:2.5.4]
        at akka.dispatch.Mailbox.run(Mailbox.scala:224) [40:com.typesafe.akka.actor:2.5.4]
        at akka.dispatch.Mailbox.exec(Mailbox.scala:234) [40:com.typesafe.akka.actor:2.5.4]
        at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) [40:com.typesafe.akka.actor:2.5.4]
        at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) [40:com.typesafe.akka.actor:2.5.4]
        at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) [40:com.typesafe.akka.actor:2.5.4]
        at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) [40:com.typesafe.akka.actor:2.5.4]
Caused by: org.opendaylight.yangtools.yang.data.api.schema.tree.ConflictingModificationAppliedException: Node was deleted by other transaction.
        at org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractNodeContainerModificationStrategy.checkTouchApplicable(AbstractNodeContainerModificationStrategy.java:287) ~[430:org.opendaylight.yangtools.yang-data-impl:2.0.1.redhat-5]
        at org.opendaylight.yangtools.yang.data.impl.schema.tree.SchemaAwareApplyOperation.checkApplicable(SchemaAwareApplyOperation.java:132) ~[430:org.opendaylight.yangtools.yang-data-impl:2.0.1.redhat-5]
        at org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractNodeContainerModificationStrategy.checkChildPreconditions(AbstractNodeContainerModificationStrategy.java:307) ~[430:org.opendaylight.yangtools.yang-data-impl:2.0.1.redhat-5]
        at org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractNodeContainerModificationStrategy.checkTouchApplicable(AbstractNodeContainerModificationStrategy.java:290) ~[430:org.opendaylight.yangtools.yang-data-impl:2.0.1.redhat-5]
 at org.opendaylight.yangtools.yang.data.impl.schema.tree.SchemaAwareApplyOperation.checkApplicable(SchemaAwareApplyOperation.java:132) ~[430:org.opendaylight.yangtools.yang-data-impl:2.0.1.redhat-5]
        at org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractNodeContainerModificationStrategy.checkChildPreconditions(AbstractNodeContainerModificationStrategy.java:307) ~[430:org.opendaylight.yangtools.yang-data-impl:2.0.1.redhat-5]
        at org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractNodeContainerModificationStrategy.checkTouchApplicable(AbstractNodeContainerModificationStrategy.java:290) ~[430:org.opendaylight.yangtools.yang-data-impl:2.0.1.redhat-5]
        at org.opendaylight.yangtools.yang.data.impl.schema.tree.SchemaAwareApplyOperation.checkApplicable(SchemaAwareApplyOperation.java:132) ~[430:org.opendaylight.yangtools.yang-data-impl:2.0.1.redhat-5]
        at org.opendaylight.yangtools.yang.data.impl.schema.tree.StructuralContainerModificationStrategy.checkApplicable(StructuralContainerModificationStrategy.java:101) ~[430:org.opendaylight.yangtools.yang-data-impl:2.0.1.redhat-5]
        at org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractNodeContainerModificationStrategy.checkChildPreconditions(AbstractNodeContainerModificationStrategy.java:307) ~[430:org.opendaylight.yangtools.yang-data-impl:2.0.1.redhat-5]
        at org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractNodeContainerModificationStrategy.checkTouchApplicable(AbstractNodeContainerModificationStrategy.java:290) ~[430:org.opendaylight.yangtools.yang-data-impl:2.0.1.redhat-5]
        at org.opendaylight.yangtools.yang.data.impl.schema.tree.SchemaAwareApplyOperation.checkApplicable(SchemaAwareApplyOperation.java:132) ~[430:org.opendaylight.yangtools.yang-data-impl:2.0.1.redhat-5]
        at org.opendaylight.yangtools.yang.data.impl.schema.tree.RootModificationApplyOperation.checkApplicable(RootModificationApplyOperation.java:72) ~[430:org.opendaylight.yangtools.yang-data-impl:2.0.1.redhat-5]
        at org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractDataTreeTip.validate(AbstractDataTreeTip.java:35) ~[430:org.opendaylight.yangtools.yang-data-impl:2.0.1.redhat-5]
        at org.opendaylight.controller.cluster.datastore.ShardDataTree.lambda$processNextPendingTransaction$0(ShardDataTree.java:732) ~[233:org.opendaylight.controller.sal-distributed-datastore:1.7.0.redhat-5]
        ... 30 more


Version-Release number of selected component (if applicable):

OSP 13
opendaylight-8.0.0-5.el7ost.noarch
How reproducible:


Steps to Reproduce: 100%
1. Install OSP13 + ODL
2. Run scale use case as described above
3.

Actual results:
Seeing exceptions in karag logs and some vms remain unpingable

Expected results:
No exceptions and all VMs should be pingable

Additional info:

Comment 1 Mike Kolesnik 2018-04-16 06:22:12 UTC

*** This bug has been marked as a duplicate of bug 1567509 ***


Note You need to log in before you can comment on or make changes to this bug.