Bug 1647409
| Summary: | octavia_api container is logging continuous tracebacks | |||
|---|---|---|---|---|
| Product: | Red Hat OpenStack | Reporter: | Lars Kellogg-Stedman <lars> | |
| Component: | openstack-tripleo-heat-templates | Assignee: | Brent Eagles <beagles> | |
| Status: | CLOSED ERRATA | QA Contact: | Bruna Bonguardo <bbonguar> | |
| Severity: | urgent | Docs Contact: | ||
| Priority: | urgent | |||
| Version: | 13.0 (Queens) | CC: | amuller, astafeye, beagles, cgoncalves, cmuresan, cyril.grosjean, dbecker, dh3, fiezzi, gkadam, ihrachys, jjoyce, jschluet, kiyyappa, knoha, lars, lpeer, majopela, martinsson.patrik, mburns, mfuruta, morazi, njohnston, rcernin, slinaber, sputhenp, tvignaud | |
| Target Milestone: | z7 | Keywords: | Triaged, ZStream | |
| Target Release: | 13.0 (Queens) | Flags: | beagles:
needinfo-
|
|
| Hardware: | Unspecified | |||
| OS: | Unspecified | |||
| Whiteboard: | ||||
| Fixed In Version: | openstack-tripleo-heat-templates-8.3.1-23.el7ost puppet-tripleo-8.4.1-7.el7ost | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | ||
| Clone Of: | ||||
| : | 1652523 1661311 (view as bug list) | Environment: | ||
| Last Closed: | 2019-07-10 13:03:18 UTC | Type: | Bug | |
| Regression: | --- | Mount Type: | --- | |
| Documentation: | --- | CRM: | ||
| Verified Versions: | Category: | --- | ||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | ||
| Cloudforms Team: | --- | Target Upstream Version: | ||
| Embargoed: | ||||
| Bug Depends On: | 1652523 | |||
| Bug Blocks: | 1618689, 1661311 | |||
Carlos, It's not clear to me that #1517500 is related: that bug seems to be about Docker monitoring support, while the problem I'm seeing here is Octavia responding poorly to the *haproxy* health checks, which are making an HTTP OPTIONS request aganist the octavia api. To answer your specific question, we are running versions of tripleo-common and tripleo-heat-template equal to or more recent than the "fixed in" versions on #1517500, and we continue to have the problem with the octavia-api container. For a sense of scope, it's generating on the order of 20MB of log output every 30 seconds (from running 'timeout 30 docker logs -f octavia_api > output.txt'). Thanks for the additional info, Lars. I can see this issue also on OSP 14. Martin Andre kindly pointed me to https://github.com/openstack/puppet-tripleo/blob/18581893d9353d2c63fa2679b9b8444d002d6874/manifests/haproxy.pp#L1505-L1515 as part of code that potentially needs to be fixed. I will be taking a look at this. Three out of five upstream changes have merged. *** Bug 1639054 has been marked as a duplicate of this bug. *** Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2019:1738 |
The octavia_api container on our OSP13 overcloud is logging the following almost continuously: 172.16.32.20 - - [06/Nov/2018 23:29:04] "OPTIONS / HTTP/1.0" 500 59 Traceback (most recent call last): File "/usr/lib64/python2.7/SocketServer.py", line 295, in _handle_request_noblock self.process_request(request, client_address) File "/usr/lib64/python2.7/SocketServer.py", line 321, in process_request self.finish_request(request, client_address) File "/usr/lib64/python2.7/SocketServer.py", line 334, in finish_request self.RequestHandlerClass(request, client_address, self) File "/usr/lib64/python2.7/SocketServer.py", line 651, in __init__ self.finish() File "/usr/lib64/python2.7/SocketServer.py", line 710, in finish self.wfile.close() File "/usr/lib64/python2.7/socket.py", line 279, in close self.flush() File "/usr/lib64/python2.7/socket.py", line 303, in flush self._sock.sendall(view[write_offset:write_offset+buffer_size]) error: [Errno 32] Broken pipe 172.16.32.16 - - [06/Nov/2018 23:29:04] "OPTIONS / HTTP/1.0" 200 0 Traceback (most recent call last): File "/usr/lib64/python2.7/wsgiref/handlers.py", line 86, in run self.finish_response() File "/usr/lib64/python2.7/wsgiref/handlers.py", line 128, in finish_response self.write(data) File "/usr/lib64/python2.7/wsgiref/handlers.py", line 212, in write self.send_headers() File "/usr/lib64/python2.7/wsgiref/handlers.py", line 270, in send_headers self.send_preamble() File "/usr/lib64/python2.7/wsgiref/handlers.py", line 194, in send_preamble 'Date: %s\r\n' % format_date_time(time.time()) File "/usr/lib64/python2.7/socket.py", line 324, in write self.flush() File "/usr/lib64/python2.7/socket.py", line 303, in flush self._sock.sendall(view[write_offset:write_offset+buffer_size]) error: [Errno 104] Connection reset by peer The request looks like the haproxy health check. I can make an OPTIONS request against the octavia-api using curl without a problem. I've opened this bug against Director because it doesn't seem to be an Octavia bug, per se, but instead either some interaction between octavia and haproxy or some issue with networking.