Bug 975047 - Bad log level for messages
Bad log level for messages
Status: CLOSED DUPLICATE of bug 967631
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-swift (Show other bugs)
Unspecified Unspecified
medium Severity high
: ---
: ---
Assigned To: Pete Zaitcev
Depends On:
  Show dependency treegraph
Reported: 2013-06-17 09:48 EDT by Lon Hohberger
Modified: 2016-04-26 09:50 EDT (History)
4 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Last Closed: 2013-06-17 09:53:26 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---

Attachments (Terms of Use)

  None (edit)
Description Lon Hohberger 2013-06-17 09:48:38 EDT
Description of problem:

Upon updating from openstack-swift-1.8.0-3.el6ost to openstack-swift-1.8.0-4.el6ost using 'yum update -y', the following error starts occurring:

Message from syslogd@localhost at Jun 17 09:43:15 ...
 �<130>proxy-server UNCAUGHT EXCEPTION#012Traceback (most recent call last):#012  File "/usr/bin/swift-proxy-server", line 22, in <module>#012    run_wsgi(conf_file, 'proxy-server', default_port=8080, **options)#012  File "/usr/lib/python2.6/site-packages/swift/common/wsgi.py", line 187, in run_wsgi#012    run_server()#012  File "/usr/lib/python2.6/site-packages/swift/common/wsgi.py", line 149, in run_server#012    global_conf={'log_name': log_name})#012  File "/usr/lib/python2.6/site-packages/PasteDeploy-1.5.0-py2.6.egg/paste/deploy/loadwsgi.py", line 247, in loadapp#012    return loadobj(APP, uri, name=name, **kw)#012  File "/usr/lib/python2.6/site-packages/PasteDeploy-1.5.0-py2.6.egg/paste/deploy/loadwsgi.py", line 271, in loadobj#012    global_conf=global_conf)#012  File "/usr/lib/python2.6/site-packages/PasteDeploy-1.5.0-py2.6.egg/paste/deploy/loadwsgi.py", line 296, in loadcontext#012    global_conf=global_conf)#012  File "/usr/lib/python2.6/site-packages/PasteDeploy-1.5.0-py2.6.egg/paste/deploy/loadwsgi.py", line 317, in _loadconfig#012    loader = ConfigLoader(path)#012  File "/usr/lib/python2.6/site-packages/PasteDeploy-1.5.0-py2.6.egg/paste/deploy/loadwsgi.py", line 393, in __init__#012    with open(filename) as f:#012IOError: [Errno 13] Permission denied: '/etc/swift/proxy-server.conf'

The problem is two fold:
 - an update process probably should be a bit cleaner and
   not blow up like this, and
 - this should not be at LOG_EMERG level, or if it is, it
   should be tracked and quiesced after the first one or

Due to the nature that this is logged at LOG_EMERG (or perhaps LOG_ALERT) and logged in a VERY tight loop, my system is presently entirely unusable - all PTYs receive this message.
Comment 1 Lon Hohberger 2013-06-17 09:50:21 EDT
ssh $HOST "killall -9 swift-proxy-server"  made the madness end.
Comment 2 Lon Hohberger 2013-06-17 09:53:26 EDT

*** This bug has been marked as a duplicate of bug 967631 ***
Comment 3 Pete Zaitcev 2013-06-17 11:52:13 EDT
Lon, please don't touch anything, if you can. I was unable to extract
the following two things from Jaroslav

 - How the wrong permissions occur in the first place. He alluded that
"Swift changed the permissions", which sounds too crazy. A spec that
specifies that in %dir? I can't find anything like that! So I need
your /var/log/yum.log (I know that you mentioned precise rpm versions
in the report, thanks a lot)

 - How the heck the console spam happens to begin with. We never
dump at LOG_EMERG. Very puzzling! I thought his rsyslog config was
screwed, but perhaps not?
Comment 4 Jaroslav Henner 2013-06-17 16:20:53 EDT
If swift is not changing anything, how do you explain the following?

[root@folsom-rhel6 ~]# ll -d /etc/swift
drwx------. 4 swift swift 4096 16. čen 10.21 /etc/swift
[root@folsom-rhel6 ~]# chmod u-r /etc/swift
[root@folsom-rhel6 ~]# ll -d /etc/swift
d-wx------. 4 swift swift 4096 16. čen 10.21 /etc/swift
[root@folsom-rhel6 ~]# /etc/init.d/openstack-swift-proxy restart
Stopping openstack-swift-proxy:                            [  OK  ]
Starting openstack-swift-proxy:                            [  OK  ]
[root@folsom-rhel6 ~]# ll -d /etc/swift
drwx------. 4 swift swift 4096 16. čen 10.21 /etc/swift

Yes, It is quite crazy, I agree.
Comment 7 Pete Zaitcev 2013-06-19 20:26:33 EDT
Day's result 6/19, again on Lon's VM:

rsyslog receives message with BOM, looking like this by strace:

recvmsg(0, {msg_name(0)=NULL, msg_iov(1)=[{"\357\273\277<146>proxy-server UNCAUGHT EX"..., 2048}] ......

Perhaps there's a unicode-type string in the traceback, causing
this behaviour. Still no clue why or how Keystone middleware triggers it.


Note You need to log in before you can comment on or make changes to this bug.