Attachment 847344 Details for Bug 1050213 – backport of fix for nova

[patch] backport of fix for nova

0001-Don-t-allow-qpid-receiving-thread-to-die.patch (text/plain), 2.90 KB, created by Russell Bryant on 2014-01-08 21:03:19 UTC

(hide)

Description:

Filename:

MIME Type:

Creator: Russell Bryant

Created: 2014-01-08 21:03:19 UTC

Size: 2.90 KB

patch

obsolete

>From 99a3b6f40006c07d356a780e3e1d1fe171ebdcb8 Mon Sep 17 00:00:00 2001
>From: Russell Bryant <rbryant@redhat.com>
>Date: Wed, 8 Jan 2014 15:59:44 -0500
>Subject: [PATCH] Don't allow qpid receiving thread to die
>
>This patch is a partial backport of
>22ec8ff616a799085239e3e529daeeefea6366c4 in oslo-incubator.
>
>https://review.openstack.org/#/c/32235/13
>
>This patch ensures that the thread created by consume_in_thread() can
>not be killed off by an unexpected exception.
>
>Change-Id: I4370045b450b2b4b9b3bde1f6f3654cdecc722e2
>---
> nova/openstack/common/excutils.py      | 32 ++++++++++++++++++++++++++++++++
> nova/openstack/common/rpc/impl_qpid.py |  1 +
> 2 files changed, 33 insertions(+)
>
>diff --git a/nova/openstack/common/excutils.py b/nova/openstack/common/excutils.py
>index bfd3fc0..dcf47c1 100644
>--- a/nova/openstack/common/excutils.py
>+++ b/nova/openstack/common/excutils.py
>@@ -22,6 +22,7 @@ Exception related utilities.
> import contextlib
> import logging
> import sys
>+import time
> import traceback
> 
> from nova.openstack.common.gettextutils import _
>@@ -49,3 +50,34 @@ def save_and_reraise_exception():
>                       traceback.format_exception(type_, value, tb))
>         raise
>     raise type_, value, tb
>+
>+
>+def forever_retry_uncaught_exceptions(infunc):
>+    def inner_func(*args, **kwargs):
>+        last_log_time = 0
>+        last_exc_message = None
>+        exc_count = 0
>+        while True:
>+            try:
>+                return infunc(*args, **kwargs)
>+            except Exception as exc:
>+                this_exc_message = str(exc)
>+                if this_exc_message == last_exc_message:
>+                    exc_count += 1
>+                else:
>+                    exc_count = 1
>+                # Do not log any more frequently than once a minute unless
>+                # the exception message changes
>+                cur_time = int(time.time())
>+                if (cur_time - last_log_time > 60 or
>+                        this_exc_message != last_exc_message):
>+                    logging.exception(
>+                        _('Unexpected exception occurred %d time(s)... '
>+                          'retrying.') % exc_count)
>+                    last_log_time = cur_time
>+                    last_exc_message = this_exc_message
>+                    exc_count = 0
>+                # This should be a very rare event. In case it isn't, do
>+                # a sleep.
>+                time.sleep(1)
>+    return inner_func
>diff --git a/nova/openstack/common/rpc/impl_qpid.py b/nova/openstack/common/rpc/impl_qpid.py
>index a7aebc1..55a1c29 100644
>--- a/nova/openstack/common/rpc/impl_qpid.py
>+++ b/nova/openstack/common/rpc/impl_qpid.py
>@@ -575,6 +575,7 @@ class Connection(object):
> 
>     def consume_in_thread(self):
>         """Consumer from all queues/consumers in a greenthread"""
>+        @excutils.forever_retry_uncaught_exceptions
>         def _consumer_thread():
>             try:
>                 self.consume()
>-- 
>1.8.3.1
>

Actions: View | Diff

Attachments on bug 1050213: 847344 | 847690