Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1815662

Summary: Repeatedly errors: client_io->complete_request() returned Input/output error
Product: Red Hat OpenStack Reporter: David Hill <dhill>
Component: puppet-tripleoAssignee: Francesco Pantano <fpantano>
Status: CLOSED ERRATA QA Contact: Yogev Rabl <yrabl>
Severity: medium Docs Contact:
Priority: medium    
Version: 13.0 (Queens)CC: afariasa, fpantano, gfidente, jdurgin, jhoylaer, jjoyce, jschluet, lhh, mbenjamin, mgarciac, nojha, slinaber, svaezi, tvignaud
Target Milestone: z14Keywords: Triaged, ZStream
Target Release: 13.0 (Queens)   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: puppet-tripleo-8.5.1-18.el7ost Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-12-16 13:55:05 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1850036    
Bug Blocks:    

Description David Hill 2020-03-20 20:19:04 UTC 
Description of problem:
Repeatedly errors: client_io->complete_request() returned Input/output error

Feb 20 16:34:59 overcloud-controller-0 journal: 2020-02-20 16:34:59.347929 7fafd55a3700  1 ====== starting new request req=0x7fafd559cf70 =====
Feb 20 16:34:59 overcloud-controller-0 journal: 2020-02-20 16:34:59.349152 7fafd55a3700  0 ERROR: client_io->complete_request() returned Input/output error
Feb 20 16:34:59 overcloud-controller-0 docker: 2020-02-20 16:34:59.347929 7fafd55a3700  1 ====== starting new request req=0x7fafd559cf70 =====
Feb 20 16:34:59 overcloud-controller-0 docker: 2020-02-20 16:34:59.349152 7fafd55a3700  0 ERROR: client_io->complete_request() returned Input/output error
Feb 20 16:34:59 overcloud-controller-0 docker: 2020-02-20 16:34:59.349173 7fafd55a3700  1 ====== req done req=0x7fafd559cf70 op status=0 http_status=200 ======
Feb 20 16:34:59 overcloud-controller-0 docker: 2020-02-20 16:34:59.349209 7fafd55a3700  1 civetweb: 0x55d75524e000: 172.18.0.11 - - [20/Feb/2020:16:34:59 +0100] "HEAD / HTTP/1.0" 200 182 - -
Feb 20 16:34:59 overcloud-controller-0 journal: 2020-02-20 16:34:59.349173 7fafd55a3700  1 ====== req done req=0x7fafd559cf70 op status=0 http_status=200 ======
Feb 20 16:34:59 overcloud-controller-0 journal: 2020-02-20 16:34:59.349209 7fafd55a3700  1 civetweb: 0x55d75524e000: 172.18.0.11 - - [20/Feb/2020:16:34:59 +0100] "HEAD / HTTP/1.0" 200 182 - -
Feb 20 16:34:59 overcloud-controller-0 ovs-vswitchd: ovs|347528|netdev_tc_offloads(revalidator603)|ERR|Dropped 17 log messages in last 1 seconds (most recently, 1 seconds ago) due to excessive rate
Feb 20 16:34:59 overcloud-controller-0 ovs-vswitchd: ovs|347529|netdev_tc_offloads(revalidator603)|ERR|dump_create: failed to get ifindex for tapda1682c1-76: Operation not supported
Feb 20 16:35:00 overcloud-controller-0 su: (to rabbitmq) root on none
Version-Release number of selected component (if applicable):


How reproducible:
This environment

Steps to Reproduce:
1. Deploy overcloud with ceph_rgw
2.
3.

Actual results:
Error logs like above

Expected results:
No error logs

Additional info:
~~~
  if $ceph_rgw {
    ::tripleo::haproxy::endpoint { 'ceph_rgw':
      public_virtual_ip => $public_virtual_ip,
      internal_ip       => hiera('ceph_rgw_vip', $controller_virtual_ip),
      service_port      => $ports[ceph_rgw_port],
      ip_addresses      => hiera('ceph_rgw_node_ips', $controller_hosts_real),
      server_names      => hiera('ceph_rgw_node_names', $controller_hosts_names_real),
      public_ssl_port   => $ports[ceph_rgw_ssl_port],
      service_network   => $ceph_rgw_network,
      listen_options    => merge($default_listen_options, { 'option' => [ 'httpchk HEAD /' ] }), <================================
    }
  }
~~~

in puppet-tripleo/manifests/haproxy.pp

That would be caused by this commit:

commit dc52f322eaa1bf3048408e5719f77e456a357ef6
Author: Alex Schultz <aschultz>
Date:   Tue Feb 28 16:46:47 2017 +0000

    Add httpchk for http services
    
    The httpchk health check option should help reduce the situtations
    where haproxy thinks the service is up but the service is only
    listening and not actively serving http requests.
    
    Change-Id: I13cc5dcf2eea53731e756d078586ab9a97340912
    Closes-Bug: #1629052

which was around since 2017.   So the problem is not on the openstack configuration side of things and would probably be a haproxy issue OR a ceph-rgw issue while handling HEAD / requests.
Comment 11 Sasan Vaezi 2020-05-07 10:33:06 UTC 
Hi,

whats the status in of this BZ? 

Thanks
Sasan
Comment 37 errata-xmlrpc 2020-12-16 13:55:05 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Red Hat OpenStack Platform 13.0 director bug fix advisory), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:5575