Bug 1610879

Created attachment 1472128 [details]
script to poll hostconfig - run on each controller in the cluster

Description of problem:

AKKA timeouts when running attached get_odl_hostconfig script.  I was running this script to debug l2 dead agents in bz:


https://bugzilla.redhat.com/show_bug.cgi?id=1519925

Version-Release number of selected component (if applicable):


How reproducible:

Intermittent, when running browbeat rally scenario on cloud20 in scale lab.


Steps to Reproduce:
1. Deploy cluster setup
2. Start attached script, get_host_config on each controller
3. run browbeat 
4. monitor output of get_host_config script on each controller, while the browbeat rally scenario is executing. Look for akka Ask exception




Actual results:

This akka exception occurs intermittently. When this occurs, there is no hostconfig returned, which could result in agents being marked dead.


{ 
   "errors": { 
       "error": [ 
           { 
               "error-info": "akka.pattern.AskTimeoutException: Ask timed out on [ActorSelection[Anchor(akka.tcp://opendaylight-cluster-data
.0.34:2550/), Path(/user/shardmanager-operational/member-2-shard-default-operational/shard-default-member-0:datastore-operational@3:622396_55237#-20
76633334)]] after [5000 ms]. Sender[null] sent message of type \"org.opendaylight.controller.cluster.datastore.messages.ReadData\".\n\tat akka.patte
rn.PromiseActorRef$.$anonfun$defaultOnTimeout$1(AskSupport.scala:595)\n\tat akka.pattern.PromiseActorRef$.$anonfun$apply$1(AskSupport.scala:605)\n\t
at akka.actor.Scheduler$$anon$4.run(Scheduler.scala:140)\n\tat scala.concurrent.Future$InternalCallbackExecutor$.unbatchedExecute(Future.scala:870)\
n\tat scala.concurrent.BatchingExecutor.execute(BatchingExecutor.scala:109)\n\tat scala.concurrent.BatchingExecutor.execute$(BatchingExecutor.scala:
103)\n\tat scala.concurrent.Future$InternalCallbackExecutor$.execute(Future.scala:868)\n\tat akka.actor.LightArrayRevolverScheduler$TaskHolder.execu
teTask(LightArrayRevolverScheduler.scala:328)\n\tat akka.actor.LightArrayRevolverScheduler$$anon$4.executeBucket$1(LightArrayRevolverScheduler.scala
:279)\n\tat akka.actor.LightArrayRevolverScheduler$$anon$4.nextTick(LightArrayRevolverScheduler.scala:283)\n\tat akka.actor.LightArrayRevolverSchedu
ler$$anon$4.run(LightArrayRevolverScheduler.scala:235)\n\tat java.lang.Thread.run(Thread.java:748)\n", 
               "error-message": "Error checking ReadData for path /(urn:opendaylight:neutron?revision=2015-07-12)neutron/hostconfigs", 
               "error-tag": "operation-failed", 
               "error-type": "application" 
           } 
       ] 
   } 
}



Expected results:

Expect to get hostconfig on each and every call with no akka timeouts.



Additional info: