Bug 1497635

Summary: [UPDATES] Traceback in ironic-inspector: error: [Errno 32] Broken pipe
Product: Red Hat OpenStack Reporter: Yurii Prokulevych <yprokule>
Component: openstack-ironic-inspectorAssignee: RHOS Maint <rhos-maint>
Status: CLOSED DUPLICATE QA Contact: mlammon
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 12.0 (Pike)CC: augol, bfournie, mburns, mcornea, slinaber, yprokule
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-10-02 15:02:25 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Yurii Prokulevych 2017-10-02 09:59:35 UTC
Description of problem:
-----------------------
While performing minor update of RHOS-12 got next traceback in journal:
***
Oct 02 05:55:01 undercloud-0.redhat.local ironic-inspector[9017]: 2017-10-02 05:55:01.796 9017 INFO werkzeug [-] 192.168.24.1 - - [02/Oct/2017 05:55:01] "OPTIONS / HTTP/1.0" 200 -
Oct 02 05:55:01 undercloud-0.redhat.local ironic-inspector[9017]: Traceback (most recent call last):
Oct 02 05:55:01 undercloud-0.redhat.local ironic-inspector[9017]: File "/usr/lib64/python2.7/SocketServer.py", line 295, in _handle_request_noblock
Oct 02 05:55:01 undercloud-0.redhat.local ironic-inspector[9017]: self.process_request(request, client_address)
Oct 02 05:55:01 undercloud-0.redhat.local ironic-inspector[9017]: File "/usr/lib64/python2.7/SocketServer.py", line 321, in process_request
Oct 02 05:55:01 undercloud-0.redhat.local ironic-inspector[9017]: self.finish_request(request, client_address)
Oct 02 05:55:01 undercloud-0.redhat.local ironic-inspector[9017]: File "/usr/lib64/python2.7/SocketServer.py", line 334, in finish_request
Oct 02 05:55:01 undercloud-0.redhat.local ironic-inspector[9017]: self.RequestHandlerClass(request, client_address, self)
Oct 02 05:55:01 undercloud-0.redhat.local ironic-inspector[9017]: File "/usr/lib64/python2.7/SocketServer.py", line 651, in __init__
Oct 02 05:55:01 undercloud-0.redhat.local ironic-inspector[9017]: self.finish()
Oct 02 05:55:01 undercloud-0.redhat.local ironic-inspector[9017]: File "/usr/lib64/python2.7/SocketServer.py", line 710, in finish
Oct 02 05:55:01 undercloud-0.redhat.local ironic-inspector[9017]: self.wfile.close()
Oct 02 05:55:01 undercloud-0.redhat.local ironic-inspector[9017]: File "/usr/lib64/python2.7/socket.py", line 279, in close
Oct 02 05:55:01 undercloud-0.redhat.local ironic-inspector[9017]: self.flush()
Oct 02 05:55:01 undercloud-0.redhat.local ironic-inspector[9017]: File "/usr/lib64/python2.7/socket.py", line 303, in flush
Oct 02 05:55:01 undercloud-0.redhat.local ironic-inspector[9017]: self._sock.sendall(view[write_offset:write_offset+buffer_size])
Oct 02 05:55:01 undercloud-0.redhat.local ironic-inspector[9017]: File "/usr/lib/python2.7/site-packages/eventlet/greenio/base.py", line 390, in sendall
Oct 02 05:55:01 undercloud-0.redhat.local ironic-inspector[9017]: tail = self.send(data, flags)
Oct 02 05:55:01 undercloud-0.redhat.local ironic-inspector[9017]: File "/usr/lib/python2.7/site-packages/eventlet/greenio/base.py", line 384, in send
Oct 02 05:55:01 undercloud-0.redhat.local ironic-inspector[9017]: return self._send_loop(self.fd.send, data, flags)
Oct 02 05:55:01 undercloud-0.redhat.local ironic-inspector[9017]: File "/usr/lib/python2.7/site-packages/eventlet/greenio/base.py", line 371, in _send_loop
Oct 02 05:55:01 undercloud-0.redhat.local ironic-inspector[9017]: return send_method(data, *args)
Oct 02 05:55:01 undercloud-0.redhat.local ironic-inspector[9017]: error: [Errno 32] Broken pipe
Oct 02 05:55:03 undercloud-0.redhat.local ironic-inspector[9017]: 2017-10-02 05:55:03.801 9017 INFO werkzeug [-] 192.168.24.1 - - [02/Oct/2017 05:55:03] "OPTIONS / HTTP/1.0" 200 -


Version-Release number of selected component (if applicable):
-------------------------------------------------------------
openstack-ironic-common-9.1.1-0.20170921133213.96f2438.el7ost.noarch
puppet-ironic-11.3.1-0.20170907060708.13e23f7.el7ost.noarch
python-ironicclient-1.17.0-0.20170906171257.cdff7a0.el7ost.noarch
openstack-ironic-conductor-9.1.1-0.20170921133213.96f2438.el7ost.noarch
python-ironic-inspector-client-2.1.0-0.20170915002324.bdcab9f.el7ost.noarch
openstack-ironic-inspector-6.0.1-0.20170920142417.77e2b1a.el7ost.noarch
openstack-ironic-api-9.1.1-0.20170921133213.96f2438.el7ost.noarch
python-ironic-lib-2.10.0-0.20170906171416.1fa0a5f.el7ost.noarch

Steps to Reproduce:
-------------------
1. Install RHOS-12 (2017-09-28.1)
2. Setup repos on UC (2017-09-29.2)
3. Update python-tripleoclient
    sudo yum update python-tripleoclient
4. Update UC:
    openstack undercloud upgrade 2>&1 | tee undercloud_update.log
5. Check journal for openstack-ironic-inspector unit:
    journalctl -u openstack-ironic-inspector -f

Actual results:
---------------
Traceback in log

Expected results:
-----------------
No traceback in log


Additional info:
----------------
Virtual setup: 3controllers + 2computes + 3ceph
UC+SSL, OC without SSL

Comment 2 Bob Fournier 2017-10-02 13:48:02 UTC
Yurii - could you please supply the ironic-inspector.log as that is not in the sosreport. Are there any other affects noticed other than this traceback in the log during the upgrade?  

This looks like a temporary network issue during the upgrade which caused the socket failure, we can tell by the ironic-inspector.log file if it got resolved.  Thanks.

Comment 4 Bob Fournier 2017-10-02 15:02:25 UTC
This issue with the same traceback is currently being tracked in another BZ so I'm going to make this one a duplicate so we can keep all analysis in one place.

*** This bug has been marked as a duplicate of bug 1477663 ***