Hide Forgot
Description of problem: WARNING - cthulhu.request_collection on_completion: unknown jid 2016-09-05 16:13:48,845 - WARNING - cthulhu.request_collection on_completion: unknown jid 20160905161348617083, return: {'data': {'timechecks': {'round_status': 'finished', 'epoch': 838, 'round': 4120, 'mons': [{'latency': 0.0, 'skew': 0.0, 'health': 'HEALTH_OK', 'name': 'mon-node-8'}, {'latency': 0.00063, 'skew': -0.0, 'health': 'HEALTH_OK', 'name': 'mon-node-9'}, {'latency': 0.000625, 'skew': -8e-06, 'health': 'HEALTH_OK', 'name': 'mon-node-10'}]}, 'summary': [], 'health': {'health_services': [{'mons': [{'last_updated': '2016-09-05 16:12:01.460363', 'name': 'mon-node-8', 'avail_percent': 86, 'kb_total': 52403200, 'kb_avail': 45444112, 'health': 'HEALTH_OK', 'kb_used': 6959088, 'store_stats': {'bytes_total': 509153533, 'bytes_log': 65536, 'last_updated': '0.000000', 'bytes_misc': 476567086, 'bytes_sst': 32520911}}, {'last_updated': '2016-09-05 16:11:30.270949', 'name': 'mon-node-9', 'avail_percent': 90, 'kb_total': 52403200, 'kb_avail': 47449400, 'health': 'HEALTH_OK', 'kb_used': 4953800, 'store_stats': {'bytes_total': 3166965952, 'bytes_log': 196608, 'last_updated': '0.000000', 'bytes_misc': 3150612591, 'bytes_sst': 16156753}}, {'last_updated': '2016-09-05 16:12:17.256264', 'name': 'mon-node-10', 'avail_percent': 83, 'kb_total': 52403200, 'kb_avail': 43594356, 'health': 'HEALTH_OK', 'kb_used': 8808844, 'store_stats': {'bytes_total': 3877745593, 'bytes_log': 0, 'last_updated': '0.000000', 'bytes_misc': 3861010186, 'bytes_sst': 16735407}}]}]}, 'overall_status': 'HEALTH_OK', 'detail': []}, 'version': 'bfca48ce8d6c43c0276dc2eb4c298ba4', 'type': 'health', 'fsid': '444f54b1-f97f-43d8-85b7-d5a02daac39a'} Version-Release number of selected component (if applicable): Red Hat Ceph Storage 1.3.2 calamari-clients-1.3-2.el7cp.x86_64 calamari-server-1.3.3-1.el7cp.x86_64 /opt/calamari/venv/lib/python2.7/site-packages/calamari_cthulhu-0.1-py2.7.egg/cthulhu How reproducible: Always
This warning message is coming from this code: https://github.com/ceph/calamari/blob/master/cthulhu/cthulhu/manager/request_collection.py#L274 ~~~~ def on_completion(self, data): """ Callback for when a salt/job/<jid>/ret event is received, in which we find the UserRequest that created the job, and inform it of completion so that it can progress. """ with self._lock: jid = data['jid'] result = data['return'] log.debug("on_completion: jid=%s data=%s" % (jid, data)) try: request = self.get_by_jid(jid) log.debug("on_completion: jid %s belongs to request %s" % (jid, request.id)) except KeyError: log.warning("on_completion: unknown jid {0}, return: {1}".format(jid, result)) return ~~~ - From the code it looks like this warning is coming when code is getting 'KeyError' exception. - Here the key is 'jid'. - I think cthulhu request collection on completion thread is not getting correct 'jid' and it is throwing this warning message. Please let us know your inputs.
Vikhyat, Is there any noticable effects beyond the presense of this warning in the cthulhu.log ? Would you please get us a copy of the log so we can determine the frequency you're seeing this warning?
(In reply to Gregory Meno from comment #3) > Vikhyat, > > Is there any noticable effects beyond the presense of this warning in the > cthulhu.log ? Nothing which I have noticed. In calamari GUI all reporting looks fine to me. > > Would you please get us a copy of the log so we can determine the frequency > you're seeing this warning? I am attaching cthulhu.log and it is very frequent.
Created attachment 1202083 [details] cthulhu.log