Description of problem: I am seeing multiple invalid user token in message log. not sure if this is indicating something is broken Sep 17 13:37:52 dell-undercloud ironic-api: error: [Errno 32] Broken pipe Sep 17 13:37:53 dell-undercloud proxy-server: Invalid user token - deferring reject downstream Sep 17 13:37:53 dell-undercloud proxy-server: 192.0.2.1 192.0.2.1 17/Sep/2015/20/37/53 GET /info HTTP/1.0 200 - - - - 738 - txe13d02b5f046434088dcb-0055fb24a1 - 0.0005 - - 1442522273.167885065 1442522273.168349028 - Sep 17 13:37:53 dell-undercloud account-server: Beginning replication run Sep 17 13:37:53 dell-undercloud account-server: Replication run OVER Sep 17 13:37:53 dell-undercloud account-server: Attempted to replicate 1 dbs in 0.00306 seconds (326.47054/s) Sep 17 13:37:53 dell-undercloud account-server: Removed 0 dbs Sep 17 13:37:53 dell-undercloud account-server: 0 successes, 0 failures Sep 17 13:37:53 dell-undercloud account-server: no_change:0 ts_repl:0 diff:0 rsync:0 diff_capped:0 hashmatch:0 empty:0 Sep 17 13:37:55 dell-undercloud ironic-api: 192.0.2.1 - - [17/Sep/2015 13:37:55] "GET / HTTP/1.0" 200 328 Sep 17 13:37:55 dell-undercloud ironic-api: Traceback (most recent call last): Sep 17 13:37:55 dell-undercloud ironic-api: File "/usr/lib64/python2.7/wsgiref/handlers.py", line 86, in run Sep 17 13:37:55 dell-undercloud ironic-api: self.finish_response() Sep 17 13:37:55 dell-undercloud ironic-api: File "/usr/lib64/python2.7/wsgiref/handlers.py", line 128, in finish_response Sep 17 13:37:55 dell-undercloud ironic-api: self.write(data) Sep 17 13:37:55 dell-undercloud ironic-api: File "/usr/lib64/python2.7/wsgiref/handlers.py", line 212, in write Sep 17 13:37:55 dell-undercloud ironic-api: self.send_headers() Sep 17 13:37:55 dell-undercloud ironic-api: File "/usr/lib64/python2.7/wsgiref/handlers.py", line 270, in send_headers Sep 17 13:37:55 dell-undercloud ironic-api: self.send_preamble() Sep 17 13:37:55 dell-undercloud ironic-api: File "/usr/lib64/python2.7/wsgiref/handlers.py", line 197, in send_preamble Sep 17 13:37:55 dell-undercloud ironic-api: self._write('Server: %s\r\n' % self.server_software) Sep 17 13:37:55 dell-undercloud ironic-api: File "/usr/lib64/python2.7/socket.py", line 324, in write Sep 17 13:37:55 dell-undercloud ironic-api: self.flush() Sep 17 13:37:55 dell-undercloud ironic-api: File "/usr/lib64/python2.7/socket.py", line 303, in flush Sep 17 13:37:55 dell-undercloud ironic-api: self._sock.sendall(view[write_offset:write_offset+buffer_size]) Sep 17 13:37:55 dell-undercloud ironic-api: File "/usr/lib/python2.7/site-packages/eventlet/greenio/base.py", line 376, in sendall Sep 17 13:37:55 dell-undercloud ironic-api: tail = self.send(data, flags) Sep 17 13:37:55 dell-undercloud ironic-api: File "/usr/lib/python2.7/site-packages/eventlet/greenio/base.py", line 359, in send Sep 17 13:37:55 dell-undercloud ironic-api: total_sent += fd.send(data[total_sent:], flags) Sep 17 13:37:55 dell-undercloud ironic-api: error: [Errno 104] Connection reset by peer Sep 17 13:37:55 dell-undercloud ironic-api: 192.0.2.1 - - [17/Sep/2015 13:37:55] "GET / HTTP/1.0" 500 59 Sep 17 13:37:55 dell-undercloud ironic-api: Traceback (most recent call last): Sep 17 13:37:55 dell-undercloud ironic-api: File "/usr/lib64/python2.7/SocketServer.py", line 593, in process_request_thread Sep 17 13:37:55 dell-undercloud ironic-api: self.finish_request(request, client_address) Sep 17 13:37:55 dell-undercloud ironic-api: File "/usr/lib64/python2.7/SocketServer.py", line 334, in finish_request Sep 17 13:37:55 dell-undercloud ironic-api: self.RequestHandlerClass(request, client_address, self) Sep 17 13:37:55 dell-undercloud ironic-api: File "/usr/lib64/python2.7/SocketServer.py", line 651, in __init__ Sep 17 13:37:55 dell-undercloud ironic-api: self.finish() Sep 17 13:37:55 dell-undercloud ironic-api: File "/usr/lib64/python2.7/SocketServer.py", line 710, in finish Sep 17 13:37:55 dell-undercloud ironic-api: self.wfile.close() Sep 17 13:37:55 dell-undercloud ironic-api: File "/usr/lib64/python2.7/socket.py", line 279, in close Sep 17 13:37:55 dell-undercloud ironic-api: self.flush() Sep 17 13:37:55 dell-undercloud ironic-api: File "/usr/lib64/python2.7/socket.py", line 303, in flush Sep 17 13:37:55 dell-undercloud ironic-api: self._sock.sendall(view[write_offset:write_offset+buffer_size]) Sep 17 13:37:55 dell-undercloud ironic-api: File "/usr/lib/python2.7/site-packages/eventlet/greenio/base.py", line 376, in sendall Sep 17 13:37:55 dell-undercloud ironic-api: tail = self.send(data, flags) Sep 17 13:37:55 dell-undercloud ironic-api: File "/usr/lib/python2.7/site-packages/eventlet/greenio/base.py", line 359, in send Sep 17 13:37:55 dell-undercloud ironic-api: total_sent += fd.send(data[total_sent:], flags) Sep 17 13:37:55 dell-undercloud ironic-api: error: [Errno 32] Broken pipe Sep 17 13:37:55 dell-undercloud proxy-server: Invalid user token - deferring reject downstream Sep 17 13:37:55 dell-undercloud proxy-server: 192.0.2.1 192.0.2.1 17/Sep/2015/20/37/55 GET /info HTTP/1.0 200 - - - - 738 - txb4f5c9c76369408dad618-0055fb24a3 - 0.0003 - - 1442522275.170485020 1442522275.170789003 - Sep 17 13:37:57 dell-undercloud ironic-api: 192.0.2.1 - - [17/Sep/2015 13:37:57] "GET / HTTP/1.0" 200 328 Sep 17 13:37:57 dell-undercloud ironic-api: Traceback (most recent call last): Sep 17 13:37:57 dell-undercloud ironic-api: File "/usr/lib64/python2.7/wsgiref/handlers.py", line 86, in run Sep 17 13:37:57 dell-undercloud ironic-api: self.finish_response() Sep 17 13:37:57 dell-undercloud ironic-api: File "/usr/lib64/python2.7/wsgiref/handlers.py", line 128, in finish_response Sep 17 13:37:57 dell-undercloud ironic-api: self.write(data) Sep 17 13:37:57 dell-undercloud ironic-api: File "/usr/lib64/python2.7/wsgiref/handlers.py", line 212, in write Sep 17 13:37:57 dell-undercloud ironic-api: self.send_headers() Sep 17 13:37:57 dell-undercloud ironic-api: File "/usr/lib64/python2.7/wsgiref/handlers.py", line 270, in send_headers Sep 17 13:37:57 dell-undercloud ironic-api: self.send_preamble() Sep 17 13:37:57 dell-undercloud ironic-api: File "/usr/lib64/python2.7/wsgiref/handlers.py", line 197, in send_preamble Sep 17 13:37:57 dell-undercloud ironic-api: self._write('Server: %s\r\n' % self.server_software) Sep 17 13:37:57 dell-undercloud ironic-api: File "/usr/lib64/python2.7/socket.py", line 324, in write Sep 17 13:37:57 dell-undercloud ironic-api: self.flush() Sep 17 13:37:57 dell-undercloud ironic-api: File "/usr/lib64/python2.7/socket.py", line 303, in flush Sep 17 13:37:57 dell-undercloud ironic-api: self._sock.sendall(view[write_offset:write_offset+buffer_size]) Sep 17 13:37:57 dell-undercloud ironic-api: File "/usr/lib/python2.7/site-packages/eventlet/greenio/base.py", line 376, in sendall Sep 17 13:37:57 dell-undercloud ironic-api: tail = self.send(data, flags) Sep 17 13:37:57 dell-undercloud ironic-api: File "/usr/lib/python2.7/site-packages/eventlet/greenio/base.py", line 359, in send Sep 17 13:37:57 dell-undercloud ironic-api: total_sent += fd.send(data[total_sent:], flags) Sep 17 13:37:57 dell-undercloud ironic-api: error: [Errno 104] Connection reset by peer Sep 17 13:37:57 dell-undercloud ironic-api: 192.0.2.1 - - [17/Sep/2015 13:37:57] "GET / HTTP/1.0" 500 59 Sep 17 13:37:57 dell-undercloud ironic-api: Traceback (most recent call last): Sep 17 13:37:57 dell-undercloud ironic-api: File "/usr/lib64/python2.7/SocketServer.py", line 593, in process_request_thread Sep 17 13:37:57 dell-undercloud ironic-api: self.finish_request(request, client_address) Sep 17 13:37:57 dell-undercloud ironic-api: File "/usr/lib64/python2.7/SocketServer.py", line 334, in finish_request Sep 17 13:37:57 dell-undercloud ironic-api: self.RequestHandlerClass(request, client_address, self) Sep 17 13:37:57 dell-undercloud ironic-api: File "/usr/lib64/python2.7/SocketServer.py", line 651, in __init__ Sep 17 13:37:57 dell-undercloud ironic-api: self.finish() Sep 17 13:37:57 dell-undercloud ironic-api: File "/usr/lib64/python2.7/SocketServer.py", line 710, in finish Sep 17 13:37:57 dell-undercloud ironic-api: self.wfile.close() Sep 17 13:37:57 dell-undercloud ironic-api: File "/usr/lib64/python2.7/socket.py", line 279, in close Sep 17 13:37:57 dell-undercloud ironic-api: self.flush() Sep 17 13:37:57 dell-undercloud ironic-api: File "/usr/lib64/python2.7/socket.py", line 303, in flush Sep 17 13:37:57 dell-undercloud ironic-api: self._sock.sendall(view[write_offset:write_offset+buffer_size]) Sep 17 13:37:57 dell-undercloud ironic-api: File "/usr/lib/python2.7/site-packages/eventlet/greenio/base.py", line 376, in sendall Sep 17 13:37:57 dell-undercloud ironic-api: tail = self.send(data, flags) Sep 17 13:37:57 dell-undercloud ironic-api: File "/usr/lib/python2.7/site-packages/eventlet/greenio/base.py", line 359, in send Sep 17 13:37:57 dell-undercloud ironic-api: total_sent += fd.send(data[total_sent:], flags) Sep 17 13:37:57 dell-undercloud ironic-api: error: [Errno 32] Broken pipe Sep 17 13:37:57 dell-undercloud proxy-server: Invalid user token - deferring reject downstream Seeing this on two undercloud installation Version-Release number of selected component (if applicable): How reproducible: Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Additional info:
the message file is uploaded to https://bigswitch.box.com/s/e4lq3acakey8xq2za09irplgtg9noxs8
*** This bug has been marked as a duplicate of bug 1246525 ***