Bug 1328471

Summary:	Repeating "ironic-api" errors in /var/log/messages on the undercloud node after upgrade to 8
Product:	Red Hat OpenStack	Reporter:	Ola Pavlenko <opavlenk>
Component:	openstack-ironic	Assignee:	Lucas Alvares Gomes <lmartins>
Status:	CLOSED WONTFIX	QA Contact:	Raviv Bar-Tal <rbartal>
Severity:	high	Docs Contact:
Priority:	unspecified
Version:	8.0 (Liberty)	CC:	mburns, rhel-osp-director-maint, srevivo
Target Milestone:	async	Keywords:	ZStream
Target Release:	---
Hardware:	x86_64
OS:	Linux
Whiteboard:
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2016-11-14 15:38:06 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:

Description Ola Pavlenko 2016-04-19 13:14:00 UTC

Description of problem:
Found these 2 bugs that was previously opened for the same issue:
https://bugzilla.redhat.com/show_bug.cgi?id=1286407
https://bugzilla.redhat.com/show_bug.cgi?id=1246525
Having same errors in messages log on undercloud node after upgrade to ospd 8

Version-Release number of selected component (if applicable):
[stack@instack ~]$ rpm -qa | grep rhos
rhos-release-1.0.39-1.noarch
[stack@instack ~]$ rpm -qa | grep ironic
python-ironic-inspector-client-1.2.0-6.el7ost.noarch
openstack-ironic-conductor-4.2.2-4.el7ost.noarch
openstack-ironic-inspector-2.2.5-2.el7ost.noarch
openstack-ironic-api-4.2.2-4.el7ost.noarch
python-ironicclient-0.8.1-1.el7ost.noarch
openstack-ironic-common-4.2.2-4.el7ost.noarch
[stack@instack ~]$ rpm -qa | grep puppet
puppet-3.6.2-2.el7.noarch
openstack-puppet-modules-7.0.17-1.el7ost.noarch
openstack-tripleo-puppet-elements-0.0.5-1.el7ost.noarch


How reproducible:


Steps to Reproduce:
1.install ospd 7, deploy overcloud and populate.
2.upgrade the undercloud node to the latest ospd puddle 
3.check for errors

Actual results:
/var/log/messages is full of errors:

Apr 19 09:04:22 instack neutron-server: _http_log_response /usr/lib/python2.7/site-packages/keystoneclient/session.py:215
Apr 19 09:04:22 instack neutron-server: 2016-04-19 09:04:22.640 22406 INFO neutron.wsgi [req-b230906c-3229-49d3-a324-bd23b825262a 21ff40b651304608ab
9b2f29aaa94b3a 4403c305790743d0ae2e60c19d3b6145 - - -] 192.0.2.1 - - [19/Apr/2016 09:04:22] "GET //v2.0/lb/members.json HTTP/1.1" 404 266 0.195483
Apr 19 09:04:22 instack neutron-server: 2016-04-19 09:04:22.640 22406 INFO neutron.wsgi [req-b230906c-3229-49d3-a324-bd23b825262a 21ff40b651304608ab
9b2f29aaa94b3a 4403c305790743d0ae2e60c19d3b6145 - - -] 192.0.2.1 - - [19/Apr/2016 09:04:22] "GET //v2.0/lb/members.json HTTP/1.1" 404 266 0.195483
Apr 19 09:04:22 instack nova-api: 2016-04-19 09:04:22.728 24490 DEBUG nova.osapi_compute.wsgi.server [-] (24490) accepted ('192.0.2.1', 43655) serve
r /usr/lib/python2.7/site-packages/eventlet/wsgi.py:826
Apr 19 09:04:22 instack nova-api: 2016-04-19 09:04:22.731 24490 DEBUG keystoneclient.session [-] REQ: curl -g -i -X GET http://192.0.2.1:35357/v3/au
th/tokens -H "X-Subject-Token: {SHA1}09d39b04808d41eb58f2a68ecfe41fbaaee0c7bf" -H "User-Agent: python-keystoneclient" -H "Accept: application/json" 
-H "X-Auth-Token: {SHA1}acc7c305c5a3c7f8da29ad0b37d59d52c5ac1d2c" _http_log_request /usr/lib/python2.7/site-packages/keystoneclient/session.py:198
Apr 19 09:04:22 instack ironic-api: 192.0.2.1 - - [19/Apr/2016 09:04:22] "GET / HTTP/1.0" 200 336
Apr 19 09:04:22 instack ironic-api: Traceback (most recent call last):
Apr 19 09:04:22 instack ironic-api: File "/usr/lib64/python2.7/wsgiref/handlers.py", line 86, in run
Apr 19 09:04:22 instack ironic-api: self.finish_response()
Apr 19 09:04:22 instack ironic-api: File "/usr/lib64/python2.7/wsgiref/handlers.py", line 128, in finish_response
Apr 19 09:04:22 instack ironic-api: self.write(data)
Apr 19 09:04:22 instack ironic-api: File "/usr/lib64/python2.7/wsgiref/handlers.py", line 212, in write
Apr 19 09:04:22 instack ironic-api: self.send_headers()
Apr 19 09:04:22 instack ironic-api: File "/usr/lib64/python2.7/wsgiref/handlers.py", line 271, in send_headers
Apr 19 09:04:22 instack ironic-api: self._write(str(self.headers))
Apr 19 09:04:22 instack ironic-api: File "/usr/lib64/python2.7/socket.py", line 324, in write
Apr 19 09:04:22 instack ironic-api: self.flush()
Apr 19 09:04:22 instack ironic-api: File "/usr/lib64/python2.7/socket.py", line 303, in flush
Apr 19 09:04:22 instack ironic-api: self._sock.sendall(view[write_offset:write_offset+buffer_size])
Apr 19 09:04:22 instack ironic-api: File "/usr/lib/python2.7/site-packages/eventlet/greenio/base.py", line 377, in sendall
Apr 19 09:04:22 instack ironic-api: tail = self.send(data, flags)
Apr 19 09:04:22 instack ironic-api: File "/usr/lib/python2.7/site-packages/eventlet/greenio/base.py", line 359, in send
Apr 19 09:04:22 instack ironic-api: total_sent += fd.send(data[total_sent:], flags)
Apr 19 09:04:22 instack ironic-api: error: [Errno 104] Connection reset by peer
Apr 19 09:04:22 instack ironic-api: 192.0.2.1 - - [19/Apr/2016 09:04:22] "GET / HTTP/1.0" 500 59
Apr 19 09:04:22 instack ironic-api: Traceback (most recent call last):
Apr 19 09:04:22 instack ironic-api: File "/usr/lib64/python2.7/SocketServer.py", line 593, in process_request_thread
Apr 19 09:04:22 instack ironic-api: self.finish_request(request, client_address)
Apr 19 09:04:22 instack ironic-api: File "/usr/lib64/python2.7/SocketServer.py", line 334, in finish_request
Apr 19 09:04:22 instack ironic-api: self.RequestHandlerClass(request, client_address, self)
Apr 19 09:04:22 instack ironic-api: File "/usr/lib64/python2.7/SocketServer.py", line 651, in __init__
Apr 19 09:04:22 instack ironic-api: self.finish()
Apr 19 09:04:22 instack ironic-api: File "/usr/lib64/python2.7/SocketServer.py", line 710, in finish
Apr 19 09:04:22 instack ironic-api: self.wfile.close()
Apr 19 09:04:22 instack ironic-api: File "/usr/lib64/python2.7/socket.py", line 279, in close
Apr 19 09:04:22 instack ironic-api: self.flush()
Apr 19 09:04:22 instack ironic-api: File "/usr/lib64/python2.7/socket.py", line 303, in flush
Apr 19 09:04:22 instack ironic-api: self._sock.sendall(view[write_offset:write_offset+buffer_size])
Apr 19 09:04:22 instack ironic-api: File "/usr/lib/python2.7/site-packages/eventlet/greenio/base.py", line 377, in sendall
Apr 19 09:04:22 instack ironic-api: tail = self.send(data, flags)
Apr 19 09:04:22 instack ironic-api: File "/usr/lib/python2.7/site-packages/eventlet/greenio/base.py", line 359, in send
Apr 19 09:04:22 instack ironic-api: total_sent += fd.send(data[total_sent:], flags)
Apr 19 09:04:22 instack ironic-api: error: [Errno 32] Broken pipe
Apr 19 09:04:22 instack nova-api: 2016-04-19 09:04:22.859 24499 DEBUG nova.metadata.wsgi.server [-] (24499) accepted ('192.0.2.11', 58209) server /usr/lib/python2.7/site-packages/eventlet/wsgi.py:826
Apr 19 09:04:22 instack nova-api: 2016-04-19 09:04:22.861 24490 DEBUG keystoneclient.session [-] RESP: [200] content-length: 1632 content-encoding: gzip x-subject-token: {SHA1}09d39b04808d41eb58f2a68ecfe41fbaaee0c7bf vary: X-Auth-Token,Accept-Encoding server: Apache/2.4.6 (Red Hat Enterprise Linux) connection: close date: Tue, 19 Apr 2016 13:04:22 GMT content-type: application/json x-openstack-request-id: req-d4090469-74b5-4780-be4b-68372ff1cd15



Expected results:
No errors

Additional info:
/var/log/messages is here --> http://ikook.tlv.redhat.com/uploads/general/olap/messages.tar.gz

Comment 2 Lucas Alvares Gomes 2016-05-16 10:19:08 UTC

Hi,

Just to confirm, the errors do not cause the API service to stop working, right?

Comment 3 Lucas Alvares Gomes 2016-11-14 15:38:06 UTC

Hi Ola,

This problem is trigged due a bug in python [0], the fix for that in Ironic was to move it to use oslo.service. But unfortunately we can not backport this fix to OSP 8 since it involves adding a new dependency.


[0] http://bugs.python.org/issue14574