Bug 1615439

Summary: Capsule overview page errors unable to fetch logs "\xE1" from ASCII-8BIT to UTF-8
Product: Red Hat Satellite Reporter: sthirugn <sthirugn>
Component: Foreman ProxyAssignee: satellite6-bugs <satellite6-bugs>
Status: CLOSED ERRATA QA Contact: Roman Plevka <rplevka>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 6.3.2CC: aruzicka, bbuckingham, bkearney, cduryee, egolov, inecas, rplevka, sthirugn, zhunting
Target Milestone: 6.4.0Keywords: Triaged
Target Release: Unused   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: foreman-proxy-1.18.0.1 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 1624035 (view as bug list) Environment:
Last Closed: 2018-10-16 18:53:25 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
Capsule Overview error
none
Capsules Overview error
none
Capsules UI error none

Description sthirugn@redhat.com 2018-08-13 15:13:43 UTC 
Created attachment 1475603 [details]
Capsule Overview error

Description of problem:
Capsule overview page errors unable to fetch logs "\xE1" from ASCII-8BIT to UTF-8

Version-Release number of selected component (if applicable):
Satellite 6.3.2

How reproducible:
Always

Steps to Reproduce:
1. Make sure that HTTP endpoint is enabled. To do that, install templates plugin (satellite-installer --scenario capsule --foreman-proxy-templates true)
2. Send some UTF-8 decoder unfriendly data TO port 8000
# nc localhost 8000 < /usr/bin/bash
3. Then access logs locally or remotely from capsule:
from satellite:
# curl -s --cert /etc/foreman/client_cert.pem --key /etc/foreman/client_key.pem --cacert /etc/foreman/proxy_ca.pem https://capsule-6-3-qa-rhel7.rhsm-qe-3.example.com:9090/logs
"\xE1" from ASCII-8BIT to UTF-8

from capsule:
# curl --cert /etc/foreman-proxy/foreman_ssl_cert.pem --key /etc/foreman-proxy/foreman_ssl_key.pem --cacert /etc/foreman-proxy/foreman_ssl_ca.pem https://capsule-6-3-qa-rhel7.rhsm-qe-3.example.com:9090/logs/ 
"\xE1" from ASCII-8BIT to UTF-8

Actual results:
There are multiple symptoms:
1. Capsules overview in Satellite UI shows the following error as shown in the attachment.
Failure: ERF50-5345 [Foreman::WrappedException]: Unable to connect ([ProxyAPI::ProxyException]: ERF12-7885 [ProxyAPI::ProxyException]: Unable to fetch logs ([RestClient::BadRequest]: ...)

2. Curl error as shown above in the reproduction steps

3. the foreman-proxy error as shown below. Make sure to turn on the debug log level in capsule (/etc/foreman-proxy/settings.d -> :log_level: DEBUG

==> var/log/foreman-proxy/proxy.log <==
D, [2018-08-13T15:01:10.162972 ] DEBUG -- : accept: 192.168.121.156:49480
D, [2018-08-13T15:01:10.164087 ] DEBUG -- : Rack::Handler::WEBrick is invoked.
D, [2018-08-13T15:01:10.164935 ] DEBUG -- : verifying remote client 192.168.121.156 against trusted_hosts ["sat-6-3-qa-rhel7.rhsm-qe-3.example.com", "capsule-6-3-qa-rhel7.rhsm-qe-3.example.com"]
E, [2018-08-13T15:01:10.166702 ] ERROR -- : "\xE1" from ASCII-8BIT to UTF-8
D, [2018-08-13T15:01:10.166763 ] DEBUG -- : "\xE1" from ASCII-8BIT to UTF-8 (Encoding::UndefinedConversionError)
/usr/share/foreman-proxy/modules/logs/logs_api.rb:15:in `encode'
/usr/share/foreman-proxy/modules/logs/logs_api.rb:15:in `to_json'
/usr/share/foreman-proxy/modules/logs/logs_api.rb:15:in `block in <class:LogsApi>'
/usr/share/gems/gems/sinatra-1.4.7/lib/sinatra/base.rb:1611:in `call'
/usr/share/gems/gems/sinatra-1.4.7/lib/sinatra/base.rb:1611:in `block in compile!'
/usr/share/gems/gems/sinatra-1.4.7/lib/sinatra/base.rb:975:in `[]'
/usr/share/gems/gems/sinatra-1.4.7/lib/sinatra/base.rb:975:in `block (3 levels) in route!'
/usr/share/gems/gems/sinatra-1.4.7/lib/sinatra/base.rb:994:in `route_eval'
/usr/share/gems/gems/sinatra-1.4.7/lib/sinatra/base.rb:975:in `block (2 levels) in route!'
/usr/share/gems/gems/sinatra-1.4.7/lib/sinatra/base.rb:1015:in `block in process_route'
/usr/share/gems/gems/sinatra-1.4.7/lib/sinatra/base.rb:1013:in `catch'
/usr/share/gems/gems/sinatra-1.4.7/lib/sinatra/base.rb:1013:in `process_route'
/usr/share/gems/gems/sinatra-1.4.7/lib/sinatra/base.rb:973:in `block in route!'
/usr/share/gems/gems/sinatra-1.4.7/lib/sinatra/base.rb:972:in `each'
/usr/share/gems/gems/sinatra-1.4.7/lib/sinatra/base.rb:972:in `route!'
/usr/share/gems/gems/sinatra-1.4.7/lib/sinatra/base.rb:1085:in `block in dispatch!'
/usr/share/gems/gems/sinatra-1.4.7/lib/sinatra/base.rb:1067:in `block in invoke'
/usr/share/gems/gems/sinatra-1.4.7/lib/sinatra/base.rb:1067:in `catch'
/usr/share/gems/gems/sinatra-1.4.7/lib/sinatra/base.rb:1067:in `invoke'
/usr/share/gems/gems/sinatra-1.4.7/lib/sinatra/base.rb:1082:in `dispatch!'
/usr/share/gems/gems/sinatra-1.4.7/lib/sinatra/base.rb:907:in `block in call!'
/usr/share/gems/gems/sinatra-1.4.7/lib/sinatra/base.rb:1067:in `block in invoke'
/usr/share/gems/gems/sinatra-1.4.7/lib/sinatra/base.rb:1067:in `catch'
/usr/share/gems/gems/sinatra-1.4.7/lib/sinatra/base.rb:1067:in `invoke'
/usr/share/gems/gems/sinatra-1.4.7/lib/sinatra/base.rb:907:in `call!'
/usr/share/gems/gems/sinatra-1.4.7/lib/sinatra/base.rb:895:in `call'
/usr/share/gems/gems/rack-1.6.4/lib/rack/commonlogger.rb:33:in `call'
/usr/share/gems/gems/sinatra-1.4.7/lib/sinatra/base.rb:219:in `call'
/usr/share/foreman-proxy/lib/proxy/log.rb:109:in `call'
/usr/share/foreman-proxy/lib/proxy/request_id_middleware.rb:9:in `call'
/usr/share/gems/gems/rack-protection-1.5.3/lib/rack/protection/xss_header.rb:18:in `call'
/usr/share/gems/gems/rack-protection-1.5.3/lib/rack/protection/path_traversal.rb:16:in `call'
/usr/share/gems/gems/rack-protection-1.5.3/lib/rack/protection/json_csrf.rb:18:in `call'
/usr/share/gems/gems/rack-protection-1.5.3/lib/rack/protection/base.rb:49:in `call'
/usr/share/gems/gems/rack-protection-1.5.3/lib/rack/protection/base.rb:49:in `call'
/usr/share/gems/gems/rack-protection-1.5.3/lib/rack/protection/frame_options.rb:31:in `call'
/usr/share/gems/gems/rack-1.6.4/lib/rack/nulllogger.rb:9:in `call'
/usr/share/gems/gems/rack-1.6.4/lib/rack/head.rb:13:in `call'
/usr/share/gems/gems/sinatra-1.4.7/lib/sinatra/show_exceptions.rb:25:in `call'
/usr/share/gems/gems/sinatra-1.4.7/lib/sinatra/base.rb:182:in `call'
/usr/share/gems/gems/sinatra-1.4.7/lib/sinatra/base.rb:2013:in `call'
/usr/share/gems/gems/sinatra-1.4.7/lib/sinatra/base.rb:1487:in `block in call'
/usr/share/gems/gems/sinatra-1.4.7/lib/sinatra/base.rb:1787:in `synchronize'
/usr/share/gems/gems/sinatra-1.4.7/lib/sinatra/base.rb:1487:in `call'
/usr/share/gems/gems/rack-1.6.4/lib/rack/urlmap.rb:66:in `block in call'
/usr/share/gems/gems/rack-1.6.4/lib/rack/urlmap.rb:50:in `each'
/usr/share/gems/gems/rack-1.6.4/lib/rack/urlmap.rb:50:in `call'
/usr/share/gems/gems/rack-1.6.4/lib/rack/builder.rb:153:in `call'
/usr/share/gems/gems/rack-1.6.4/lib/rack/handler/webrick.rb:88:in `service'
/usr/share/ruby/webrick/httpserver.rb:138:in `service'
/usr/share/ruby/webrick/httpserver.rb:94:in `run'
/usr/share/ruby/webrick/server.rb:295:in `block in start_thread'
I, [2018-08-13T15:01:10.167171 ]  INFO -- : 192.168.121.156 - - [13/Aug/2018:15:01:10 +0000] "GET /logs/ HTTP/1.1" 400 31 0.0024
D, [2018-08-13T15:01:10.208478 ] DEBUG -- : close: 192.168.121.156:49480

Expected results:
No error when UTF-8 decoder unfriendly data is sent.

Additional info:
- As a temporary workaround, you can restart foreman-proxy in capsule.
- The upstream patch (https://github.com/theforeman/smart-proxy/pull/596) is a one-liner and is already merged. I applied this in /usr/share/foreman-proxy/lib/proxy/log_buffer/decorator.rb and restarted foreman-proxy and all the issue is resolved, all three symptoms I posted above are gone.
Comment 2 sthirugn@redhat.com 2018-08-13 15:16:23 UTC 
Created attachment 1475604 [details]
Capsules Overview error
Comment 3 sthirugn@redhat.com 2018-08-13 15:17:42 UTC 
Created attachment 1475605 [details]
Capsules UI error
Comment 4 Adam Ruzicka 2018-08-14 09:20:32 UTC 
Connecting upstream issue and moving to POST because this is already fixed in upstream.
Comment 11 Ivan Necas 2018-09-14 12:20:14 UTC 
*** Bug 1588190 has been marked as a duplicate of this bug. ***
Comment 12 Roman Plevka 2018-09-19 13:39:42 UTC 
VERIFIED
on sat6.4.0-22

$ nc 10.8.30.219 8000 < /usr/bin/bash
HTTP/1.1 400 Bad Request 
Content-Type: text/html; charset=ISO-8859-1
Server: 
Date: Wed, 19 Sep 2018 12:39:03 GMT
Content-Length: 1226
Connection: close

Ncat: Connection reset by peer.

proxy.log:

D, [2018-09-19T08:39:03.618925 ] DEBUG -- : accept: ::ffff:10.40.204.96:39928
E, [2018-09-19T08:39:03.632741 ] ERROR -- : bad Request-Line `ELF>��@H3@8
                                                                                @@@@h���(g(g PmPm0Pm0��S ��0�0���  ���0$$P�td����B�BQ�tdR�tdPmPm0Pm0�2�2/lib64/ld-linux-x86-64.so.2GNU GNU��GNUI��&��I84��*���	00
           #!Jzd��AP�DDB �	��@�AJ��!Ih�a"r�'.
D, [2018-09-19T08:39:03.634397 ] DEBUG -- : close: ::ffff:10.40.204.96:39928
Comment 13 Bryan Kearney 2018-10-16 18:53:25 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2018:2927