Bug 975047 - Bad log level for messages
Bad log level for messages
Status: CLOSED DUPLICATE of bug 967631
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-swift (Show other bugs)
3.0
Unspecified Unspecified
medium Severity high
: ---
: ---
Assigned To: Pete Zaitcev
Haim
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2013-06-17 09:48 EDT by Lon Hohberger
Modified: 2016-04-26 09:50 EDT (History)
4 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2013-06-17 09:53:26 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Lon Hohberger 2013-06-17 09:48:38 EDT
Description of problem:

Upon updating from openstack-swift-1.8.0-3.el6ost to openstack-swift-1.8.0-4.el6ost using 'yum update -y', the following error starts occurring:

Message from syslogd@localhost at Jun 17 09:43:15 ...
 �<130>proxy-server UNCAUGHT EXCEPTION#012Traceback (most recent call last):#012  File "/usr/bin/swift-proxy-server", line 22, in <module>#012    run_wsgi(conf_file, 'proxy-server', default_port=8080, **options)#012  File "/usr/lib/python2.6/site-packages/swift/common/wsgi.py", line 187, in run_wsgi#012    run_server()#012  File "/usr/lib/python2.6/site-packages/swift/common/wsgi.py", line 149, in run_server#012    global_conf={'log_name': log_name})#012  File "/usr/lib/python2.6/site-packages/PasteDeploy-1.5.0-py2.6.egg/paste/deploy/loadwsgi.py", line 247, in loadapp#012    return loadobj(APP, uri, name=name, **kw)#012  File "/usr/lib/python2.6/site-packages/PasteDeploy-1.5.0-py2.6.egg/paste/deploy/loadwsgi.py", line 271, in loadobj#012    global_conf=global_conf)#012  File "/usr/lib/python2.6/site-packages/PasteDeploy-1.5.0-py2.6.egg/paste/deploy/loadwsgi.py", line 296, in loadcontext#012    global_conf=global_conf)#012  File "/usr/lib/python2.6/site-packages/PasteDeploy-1.5.0-py2.6.egg/paste/deploy/loadwsgi.py", line 317, in _loadconfig#012    loader = ConfigLoader(path)#012  File "/usr/lib/python2.6/site-packages/PasteDeploy-1.5.0-py2.6.egg/paste/deploy/loadwsgi.py", line 393, in __init__#012    with open(filename) as f:#012IOError: [Errno 13] Permission denied: '/etc/swift/proxy-server.conf'

The problem is two fold:
 - an update process probably should be a bit cleaner and
   not blow up like this, and
 - this should not be at LOG_EMERG level, or if it is, it
   should be tracked and quiesced after the first one or
   two

Due to the nature that this is logged at LOG_EMERG (or perhaps LOG_ALERT) and logged in a VERY tight loop, my system is presently entirely unusable - all PTYs receive this message.
Comment 1 Lon Hohberger 2013-06-17 09:50:21 EDT
ssh $HOST "killall -9 swift-proxy-server"  made the madness end.
Comment 2 Lon Hohberger 2013-06-17 09:53:26 EDT

*** This bug has been marked as a duplicate of bug 967631 ***
Comment 3 Pete Zaitcev 2013-06-17 11:52:13 EDT
Lon, please don't touch anything, if you can. I was unable to extract
the following two things from Jaroslav

 - How the wrong permissions occur in the first place. He alluded that
"Swift changed the permissions", which sounds too crazy. A spec that
specifies that in %dir? I can't find anything like that! So I need
your /var/log/yum.log (I know that you mentioned precise rpm versions
in the report, thanks a lot)

 - How the heck the console spam happens to begin with. We never
dump at LOG_EMERG. Very puzzling! I thought his rsyslog config was
screwed, but perhaps not?
Comment 4 Jaroslav Henner 2013-06-17 16:20:53 EDT
If swift is not changing anything, how do you explain the following?

[root@folsom-rhel6 ~]# ll -d /etc/swift
drwx------. 4 swift swift 4096 16. čen 10.21 /etc/swift
[root@folsom-rhel6 ~]# chmod u-r /etc/swift
[root@folsom-rhel6 ~]# ll -d /etc/swift
d-wx------. 4 swift swift 4096 16. čen 10.21 /etc/swift
[root@folsom-rhel6 ~]# /etc/init.d/openstack-swift-proxy restart
Stopping openstack-swift-proxy:                            [  OK  ]
Starting openstack-swift-proxy:                            [  OK  ]
[root@folsom-rhel6 ~]# ll -d /etc/swift
drwx------. 4 swift swift 4096 16. čen 10.21 /etc/swift

Yes, It is quite crazy, I agree.
Comment 7 Pete Zaitcev 2013-06-19 20:26:33 EDT
Day's result 6/19, again on Lon's VM:

rsyslog receives message with BOM, looking like this by strace:

recvmsg(0, {msg_name(0)=NULL, msg_iov(1)=[{"\357\273\277<146>proxy-server UNCAUGHT EX"..., 2048}] ......

Perhaps there's a unicode-type string in the traceback, causing
this behaviour. Still no clue why or how Keystone middleware triggers it.

URL:
 http://docs.python.org/2/howto/logging-cookbook.html#inserting-a-bom-into-messages-sent-to-a-sysloghandler

Note You need to log in before you can comment on or make changes to this bug.