1323772 – ironic-api Errno 32 Broken pipe

Bug 1323772 - ironic-api Errno 32 Broken pipe

Summary: ironic-api Errno 32 Broken pipe

Keywords:
Status:	CLOSED CANTFIX
Alias:	None
Product:	Red Hat OpenStack
Classification:	Red Hat
Component:	openstack-ironic
Sub Component:
Version:	7.0 (Kilo)
Hardware:	Unspecified
OS:	Unspecified
Priority:	unspecified
Severity:	unspecified
Target Milestone:	---
Target Release:	8.0 (Liberty)
Assignee:	Lucas Alvares Gomes
QA Contact:	Raviv Bar-Tal
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+	depends on / blocked

Reported:	2016-04-04 16:11 UTC by Dan Yocum
Modified:	2016-08-18 13:50 UTC (History)
CC List:	3 users (show)
Fixed In Version:
Doc Type:	Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed:	2016-08-18 13:50:44 UTC
Target Upstream Version:
Embargoed:

Attachments	(Terms of Use)

Description Dan Yocum 2016-04-04 16:11:10 UTC

Description of problem:

Apr 04 09:07:33 director1.os1.phx2.redhat.com ironic-api[9007]: 10.3.3.1 - - [04/Apr/2016 09:07:33] "GET / HTTP/1.0" 200 354
Apr 04 09:07:33 director1.os1.phx2.redhat.com ironic-api[9007]: Traceback (most recent call last):
Apr 04 09:07:33 director1.os1.phx2.redhat.com ironic-api[9007]: File "/usr/lib64/python2.7/wsgiref/handlers.py", line 86, in run
Apr 04 09:07:33 director1.os1.phx2.redhat.com ironic-api[9007]: self.finish_response()
Apr 04 09:07:33 director1.os1.phx2.redhat.com ironic-api[9007]: File "/usr/lib64/python2.7/wsgiref/handlers.py", line 128, in finish_response
Apr 04 09:07:33 director1.os1.phx2.redhat.com ironic-api[9007]: self.write(data)
Apr 04 09:07:33 director1.os1.phx2.redhat.com ironic-api[9007]: File "/usr/lib64/python2.7/wsgiref/handlers.py", line 212, in write
Apr 04 09:07:33 director1.os1.phx2.redhat.com ironic-api[9007]: self.send_headers()
Apr 04 09:07:33 director1.os1.phx2.redhat.com ironic-api[9007]: File "/usr/lib64/python2.7/wsgiref/handlers.py", line 270, in send_headers
Apr 04 09:07:33 director1.os1.phx2.redhat.com ironic-api[9007]: self.send_preamble()
Apr 04 09:07:33 director1.os1.phx2.redhat.com ironic-api[9007]: File "/usr/lib64/python2.7/wsgiref/handlers.py", line 197, in send_preamble
Apr 04 09:07:33 director1.os1.phx2.redhat.com ironic-api[9007]: self._write('Server: %s\r\n' % self.server_software)
Apr 04 09:07:33 director1.os1.phx2.redhat.com ironic-api[9007]: File "/usr/lib64/python2.7/socket.py", line 324, in write
Apr 04 09:07:33 director1.os1.phx2.redhat.com ironic-api[9007]: self.flush()
Apr 04 09:07:33 director1.os1.phx2.redhat.com ironic-api[9007]: File "/usr/lib64/python2.7/socket.py", line 303, in flush
Apr 04 09:07:33 director1.os1.phx2.redhat.com ironic-api[9007]: self._sock.sendall(view[write_offset:write_offset+buffer_size])
Apr 04 09:07:33 director1.os1.phx2.redhat.com ironic-api[9007]: File "/usr/lib/python2.7/site-packages/eventlet/greenio/base.py", line 377, in sendall
Apr 04 09:07:33 director1.os1.phx2.redhat.com ironic-api[9007]: tail = self.send(data, flags)
Apr 04 09:07:33 director1.os1.phx2.redhat.com ironic-api[9007]: File "/usr/lib/python2.7/site-packages/eventlet/greenio/base.py", line 359, in send
Apr 04 09:07:33 director1.os1.phx2.redhat.com ironic-api[9007]: total_sent += fd.send(data[total_sent:], flags)
Apr 04 09:07:33 director1.os1.phx2.redhat.com ironic-api[9007]: error: [Errno 104] Connection reset by peer
Apr 04 09:07:33 director1.os1.phx2.redhat.com ironic-api[9007]: 10.3.3.1 - - [04/Apr/2016 09:07:33] "GET / HTTP/1.0" 500 59
Apr 04 09:07:33 director1.os1.phx2.redhat.com ironic-api[9007]: Traceback (most recent call last):
Apr 04 09:07:33 director1.os1.phx2.redhat.com ironic-api[9007]: File "/usr/lib64/python2.7/SocketServer.py", line 593, in process_request_thread
Apr 04 09:07:33 director1.os1.phx2.redhat.com ironic-api[9007]: self.finish_request(request, client_address)
Apr 04 09:07:33 director1.os1.phx2.redhat.com ironic-api[9007]: File "/usr/lib64/python2.7/SocketServer.py", line 334, in finish_request
Apr 04 09:07:33 director1.os1.phx2.redhat.com ironic-api[9007]: self.RequestHandlerClass(request, client_address, self)
Apr 04 09:07:33 director1.os1.phx2.redhat.com ironic-api[9007]: File "/usr/lib64/python2.7/SocketServer.py", line 651, in __init__
Apr 04 09:07:33 director1.os1.phx2.redhat.com ironic-api[9007]: self.finish()
Apr 04 09:07:33 director1.os1.phx2.redhat.com ironic-api[9007]: File "/usr/lib64/python2.7/SocketServer.py", line 710, in finish
Apr 04 09:07:33 director1.os1.phx2.redhat.com ironic-api[9007]: self.wfile.close()
Apr 04 09:07:33 director1.os1.phx2.redhat.com ironic-api[9007]: File "/usr/lib64/python2.7/socket.py", line 279, in close
Apr 04 09:07:33 director1.os1.phx2.redhat.com ironic-api[9007]: self.flush()
Apr 04 09:07:33 director1.os1.phx2.redhat.com ironic-api[9007]: File "/usr/lib64/python2.7/socket.py", line 303, in flush
Apr 04 09:07:33 director1.os1.phx2.redhat.com ironic-api[9007]: self._sock.sendall(view[write_offset:write_offset+buffer_size])
Apr 04 09:07:33 director1.os1.phx2.redhat.com ironic-api[9007]: File "/usr/lib/python2.7/site-packages/eventlet/greenio/base.py", line 377, in sendall
Apr 04 09:07:33 director1.os1.phx2.redhat.com ironic-api[9007]: tail = self.send(data, flags)
Apr 04 09:07:33 director1.os1.phx2.redhat.com ironic-api[9007]: File "/usr/lib/python2.7/site-packages/eventlet/greenio/base.py", line 359, in send
Apr 04 09:07:33 director1.os1.phx2.redhat.com ironic-api[9007]: total_sent += fd.send(data[total_sent:], flags)
Apr 04 09:07:33 director1.os1.phx2.redhat.com ironic-api[9007]: error: [Errno 32] Broken pipe
Apr 04 09:07:34 director1.os1.phx2.redhat.com ironic-api[9007]: 2016-04-04 09:07:34.162 9007 DEBUG keystonemiddleware.auth_token [-] Removing headers from request environment: X-Service-Catalog,X-Identity-Status,X-Service-Identity-Status,X-Roles,X-Service-Roles,X-Domain-Name,X-Service-Domain-Name,X-Project-Id,X-Service-Project-Id,X-Project-Domain-Name,X-Service-Project-Domain-Name,X-User-Id,X-Service-User-Id,X-User-Name,X-Service-User-Name,X-Project-Name,X-Service-Project-Name,X-User-Domain-Id,X-Service-User-Domain-Id,X-Domain-Id,X-Service-Domain-Id,X-User-Domain-Name,X-Service-User-Domain-Name,X-Project-Domain-Id,X-Service-Project-Domain-Id,X-Role,X-User,X-Tenant-Name,X-Tenant-Id,X-Tenant _remove_auth_headers /usr/lib/python2.7/site-packages/keystonemiddleware/auth_token/__init__.py:672


Version-Release number of selected component (if applicable):

# rpm -qa | grep ironic
python-ironicclient-0.5.1-12.el7ost.noarch
openstack-ironic-discoverd-1.1.0-8.el7ost.noarch
openstack-ironic-common-2015.1.2-2.el7ost.noarch
openstack-ironic-api-2015.1.2-2.el7ost.noarch
openstack-ironic-conductor-2015.1.2-2.el7ost.noarch
python-ironic-discoverd-1.1.0-8.el7ost.noarch


How reproducible:

Every 

Steps to Reproduce:
1. deploy overcloud
2.
3.

Comment 2 Lucas Alvares Gomes 2016-08-18 13:50:44 UTC

This problem is related to this bug upstream [0] and the cause is a bug in the python SocketServer module [1].

Upstream we've started using oslo.service that overcomes this problem but since it introduces a new dependency it can't be backported to osp 7 or 8.

[0] https://bugs.launchpad.net/ironic/+bug/1296833
[1] http://bugs.python.org/issue14574

Note You need to log in before you can comment on or make changes to this bug.