In our setup, there shouldn’t be an inventory shard, which I suspect is the reason the result is the error message shown above.

Do you get anything sensible with

    curl http://172.17.1.14:8081/jolokia/read/org.opendaylight.controller:Category=Shards,name=member-1-shard-default-config,type=DistributedConfigDatastore

instead?

And also 8181 port is disabled. ODL listens to only 8081. Please use username and password too with the above curl.

Hi Stephen/Janki,

There is no luck while adding the credential in following REST call for "DistributedConfigDatastore" and "DistributedOperationalDatastore".

	# curl -u "admin:admin" http://172.17.1.10:8081/jolokia/read/org.opendaylight.controller:Category=Shards,name=member-1-shard-inventory-config,type=DistributedConfigDatastore| python -m json.tool
	# curl -u "admin:admin" http://172.17.1.10:8081/jolokia/read/org.opendaylight.controller:Category=Shards,name=member-1-shard-default-operational,type=DistributedOperationalDatastore| python -m json.tool

Referring to below java exception, it looks like InstanceNotFoundException, which I assume the type doesn't exist. Please have a look into it and keep suggesting me if is there any alternative for ODL Oxygen release.

~~~
    "stacktrace": "javax.management.InstanceNotFoundException: org.opendaylight.controller:Category=Shards,name=member-1-shard-default-operational,type=DistributedOperationalDatastore\n\tat com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getMBean(DefaultMBeanServerInterceptor.java:1095)\n\tat com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getMBeanInfo(DefaultMBeanServerInterceptor.java:1375)\n\tat com.sun.jmx.mbeanserver.JmxMBeanServer.getMBeanInfo(JmxMBeanServer.java:920)\n\tat org.jolokia.handler.ReadHandler$1.execute(ReadHandler.java:46)\n\tat org.jolokia.handler.ReadHandler$1.execute(ReadHandler.java:41)\n\tat org.jolokia.backend.executor.AbstractMBeanServerExecutor.call(AbstractMBeanServerExecutor.java:90)\n\tat org.jolokia.handler.ReadHandler.getMBeanInfo(ReadHandler.java:229)\n\tat org.jolokia.handler.ReadHandler.getAllAttributesNames(ReadHandler.java:241)\n\tat org.jolokia.handler.ReadHandler.resolveAttributes(ReadHandler.java:217)\n\tat org.jolokia.handler.ReadHandler.fetchAttributes(ReadHandler.java:183)\n\tat org.jolokia.handler.ReadHandler.doHandleRequest(ReadHandler.java:118)\n\tat org.jolokia.handler.ReadHandler.doHandleRequest(ReadHandler.java:37)\n\tat org.jolokia.handler.JsonRequestHandler.handleRequest(JsonRequestHandler.java:161)\n\tat org.jolokia.backend.MBeanServerHandler.dispatchRequest(MBeanServerHandler.java:156)\n\tat org.jolokia.backend.LocalRequestDispatcher.dispatchRequest(LocalRequestDispatcher.java:99)\n\tat org.jolokia.backend.BackendManager.callRequestDispatcher(BackendManager.java:428)\n\tat org.jolokia.backend.BackendManager.handleRequest(BackendManager.java:158)\n\tat org.jolokia.http.HttpRequestHandler.executeRequest(HttpRequestHandler.java:197)\n\tat org.jolokia.http.HttpRequestHandler.handleGetRequest(HttpRequestHandler.java:86)\n\tat org.jolokia.http.AgentServlet$4.handleRequest(AgentServlet.java:470)\n\tat org.jolokia.http.AgentServlet.handleSecurely(AgentServlet.java:350)\n\tat org.jolokia.http.AgentServlet.handle(AgentServlet.java:321)\n\tat org.jolokia.http.AgentServlet.doGet(AgentServlet.java:277)\n\tat javax.servlet.http.HttpServlet.service(HttpServlet.java:687)\n\tat javax.servlet.http.HttpServlet.service(HttpServlet.java:790)\n\tat org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:845)\n\tat org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:584)\n\tat org.ops4j.pax.web.service.jetty.internal.HttpServiceServletHandler.doHandle(HttpServiceServletHandler.java:71)\n\tat org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)\n\tat org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)\n\tat org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)\n\tat org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)\n\tat org.ops4j.pax.web.service.jetty.internal.HttpServiceContext.doHandle(HttpServiceContext.java:284)\n\tat org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)\n\tat org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)\n\tat org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)\n\tat org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)\n\tat org.ops4j.pax.web.service.jetty.internal.JettyServerHandlerCollection.handle(JettyServerHandlerCollection.java:80)\n\tat org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)\n\tat org.eclipse.jetty.server.Server.handle(Server.java:534)\n\tat org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)\n\tat org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)\n\tat org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)\n\tat org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)\n\tat org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)\n\tat org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)\n\tat org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)\n\tat org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)\n\tat org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)\n\tat org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)\n\tat java.lang.Thread.run(Thread.java:748)\n",
    "status": 404
}

~~~


Note: Using the admin credential, I am able to fetch below details.

	i) Able to recived lists of schema which available MBeans: 
		# curl -u "admin:admin" http://172.17.1.10:8081/jolokia/list | python -m json.tool

	ii) Successfully read the information about the shards local use the following REST calls.
		# curl -u "admin:admin" http://172.17.1.10:8081/jolokia/read/org.opendaylight.controller:type=DistributedConfigDatastore,Category=ShardManager,name=shard-manager-config| python -m json.tool

	iii) Able to read the operational data store.
		# curl -u "admin:admin" http://172.17.1.10:8081/jolokia/read/org.opendaylight.controller:type=DistributedOperationalDatastore,Category=ShardManager,name=shard-manager-operational| python -m json.tool

	{
	    "request": {
		"mbean": "org.opendaylight.controller:Category=ShardManager,name=shard-manager-operational,type=DistributedOperationalDatastore",
		"type": "read"
	    },
	    "status": 200,
	    "timestamp": 1531145876,
	    "value": {
		"LocalShards": [
		    "member-2-shard-default-operational",
		    "member-2-shard-prefix-configuration-shard-operational",
		    "member-2-shard-entity-ownership-operational"
		],
		"MemberName": "member-2",
		"SyncStatus": true
	    }
	}


Reference Upstream Link: 
https://docs.opendaylight.org/en/stable-carbon/getting-started-guide/common-features/clustering.html

Regards,
Pradipta

Hi Pradipta

As Stephen mentioned in comment #2, inventory shards are absent in our setup so you are getting 404.

Please try the url Stephen suggested. Also, jolokia list doesnot contain any of the shard names which are giving 404.

I got 200 for below urls

1. curl -u odladmin:redhat http://$vip:8081/jolokia/read/org.opendaylight.controller:Category=ShardManager,name=shard-manager-config,type=DistributedConfigDatastore

2. curl -u odladmin:redhat http://$vip:8081/jolokia/read/org.opendaylight.controller:Category=Shards,name=member-1-shard-prefix-configuration-shard-config,type=DistributedConfigDatastore

3. curl -u odladmin:redhat http://$vip:8081/jolokia/read/org.opendaylight.controller:type=DistributedConfigDatastore,Category=ShardManager,name=shard-manager-config| python -m json.tool

4. curl -u odladmin:redhat http://$vip:8081/jolokia/read/org.opendaylight.controller:type=DistributedOperationalDatastore,Category=ShardManager,name=shard-manager-operational| python -m json.tool

5. curl -u odladmin:redhat http://$vip:8081/jolokia/read/org.opendaylight.controller:Category=Shards,name=member-1-shard-default-config,type=DistributedConfigDatastore

@Stephen, which among these should be used to get proper cluster status?
I figured 1 and 2 from monitor.py.

Hi Janki,

Based on the suggestion by Stephen on comment #2, please find my attached output #6 & #7 , where the result is getting fluctuated and I received 404 (error) & 200 status during repeated execution.

Would it be possible for you to share some insight of 404 error details and why the status is fluctuated with 404 and 200 status. Please let me know if you need any additional information for any further investigation.

Would be appreciated if you can share some insight documentation to understand the Cluster architecture in OpenDaylight (Oxygen).

-- Pradipta

For basic cluster status, or strictly speaking, replicated datastore status, there are two URLs to use, one for each datastore:

* the configuration datastore is summarized on

    /jolokia/read/org.opendaylight.controller:type=DistributedConfigDatastore,Category=ShardManager,name=shard-manager-config

* the operational datastore is summarized on

    /jolokia/read/org.opendaylight.controller:type=DistributedOperationalDatastore,Category=ShardManager,name=shard-manager-operational

The health-check in both cases is given by the "value/SyncStatus" field which needs to be true. This information is present for single nodes and clusters, e.g.

    {
      "request": {
        "mbean": "org.opendaylight.controller:Category=ShardManager,name=shard-manager-operational,type=DistributedOperationalDatastore",
        "type": "read"
      },
      "value": {
        "LocalShards": [
          "member-1-shard-default-operational",
          "member-1-shard-prefix-configuration-shard-operational",
          "member-1-shard-topology-operational",
          "member-1-shard-entity-ownership-operational",
          "member-1-shard-inventory-operational"
        ],
        "SyncStatus": true,
        "MemberName": "member-1"
      },
      "timestamp": 1531731543,
      "status": 200
    }

See https://docs.opendaylight.org/en/latest/getting-started-guide/clustering.html for details.

So we need to change the doc accordingly. Pradipta can you please change this to a doc bug.

2 changes need to be made in doc
1. Remove all references to port 8181. Only 8081 is open for ODL
2. Edit cluster check URLs.

Hi Stephen/Janki,

Thank you for sharing the insight of OpenDaylight cluster.

In my comment #4, I was queried to get the difference between the status of 400 & 200 while doing curl for DistributedConfigDatastore. 

As per my understanding, the DistributedConfigDatastore is supposed to store the desired state of the cluster where the status should be 200 (comment #6), But maximum time, it returns with status 400 which indicate the DistributedConfigDatastore cluster tree is in an inconsistent state which threw the "InstanceNotFoundException" (comment #7).

The operational data store tree (DistributedOperationalDatastore) looks good since its returns with the current status 200 and "SyncStatus": true.

- Please help me to understand if is there any dependencies of DistributedOperationalDatastore with DistributedConfigDatastore during sync between content-based routing between two trees.
- How we can fix the DistributedConfigDatastore with consistent status 200. Please keep sharing the recommended steps for it.
- In my current testing on DistributedConfigDatastore has returned with status 400, so I lost the details of RaftState (leader/follower)
  

PS: Once I will fix the issue, I will surely route this bug to the documentation team.

Regards,
Pradipta

The cluster status URL needs to be changed to

/jolokia/read/org.opendaylight.controller:type=DistributedConfigDatastore,Category=ShardManager,name=shard-manager-config

for config datastore and

 /jolokia/read/org.opendaylight.controller:type=DistributedOperationalDatastore,Category=ShardManager,name=shard-manager-operational

for operational datastore.

Assigning to Jo for review.

Jo, the thread of comments is quite long here, so please see comment 24 for the final summary of the changes required.

Affected section: https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/13/html-single/red_hat_opendaylight_installation_and_configuration_guide/#cluster-monitoring

Update complete. 
Agreed with Pradipta that the diagram request is a separate bug. 

https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/13/html/red_hat_opendaylight_installation_and_configuration_guide/high-availability-and-clustering-with-opendaylight#monitoring-with-jolokia