Bug 1268969 - oVirt Hosted Engine is not reachable after engine setup
oVirt Hosted Engine is not reachable after engine setup
Status: CLOSED NOTABUG
Product: ovirt-hosted-engine-setup
Classification: oVirt
Component: General (Show other bugs)
---
x86_64 Linux
unspecified Severity medium (vote)
: ---
: ---
Assigned To: Simone Tiraboschi
Pavel Stehlik
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2015-10-05 14:37 EDT by joselito.angeles
Modified: 2015-10-07 05:31 EDT (History)
7 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2015-10-07 05:31:52 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---
rule-engine: planning_ack?
rule-engine: devel_ack?
rule-engine: testing_ack?


Attachments (Terms of Use)
vdsm, engine, server and setup logs (441.18 KB, application/zip)
2015-10-05 14:37 EDT, joselito.angeles
no flags Details
ovirt hosted engine setup log (309.06 KB, text/plain)
2015-10-05 17:12 EDT, joselito.angeles
no flags Details

  None (edit)
Description joselito.angeles 2015-10-05 14:37:28 EDT
Created attachment 1080022 [details]
vdsm, engine, server and setup logs

Description of problem:
I have deployed ovirt 3.5 hosted engine from a RHEL 7.1 host to a  RHEL 7.1 VM. After the engine setup was completed successfully on the VM, I went back to the host where I deployed the engine and selected option [1] but got the message below. 

[ ERROR ] Engine is still unreachable
          Engine health status page is not yet reachable.
          To continue make a selection from the options below:

If I run vm-status it returns the engine status as up and healthy. 
# hosted-engine --vm-status
--== Host 1 status ==--
Status up-to-date                  : True
Hostname                           : usatl-s-smperf1
Host ID                            : 1
Engine status                      : {"health": "good", "vm": "up", "detail": "up"}
Score                              : 2400
Local maintenance                  : False
Host timestamp                     : 247344
Extra metadata (valid at timestamp):
        metadata_parse_version=1
        metadata_feature_version=1
        timestamp=247344 (Mon Oct  5 13:58:35 2015)
        host-id=1
        score=2400
        maintenance=False
        state=EngineUp

But if I run check-liveliness it returns the message below.
# hosted-engine --check-liveliness
No handlers could be found for logger "otopi.__main__"
Hosted Engine is not up!

I can access and log on to the engine web admin portal  via a browser. I also get  "DB Up! Welcome to Health Status!" if I go to https://<FQDN>/ovirt-engine/services/health

I tried stopping iptables on both host and engine but got the same error back. I have also tried redeploying the engine but ended up with the same problem.

Please advise.

Thanks.

Version-Release number of selected component (if applicable):
ovirt-hosted-engine-setup-1.2.5.3-1.el7.centos.noarch
ovirt-engine-sdk-python-3.5.4.0-1.el7.centos.noarch
libgovirt-0.3.1-3.el7.x86_64
ovirt-hosted-engine-ha-1.2.6.1-1.el7.centos.noarch
ovirt-host-deploy-1.3.2-1.el7.centos.noarch
ovirt-engine-3.5.4.2-1.el7.centos.noarch

Red Hat Enterprise Linux Server release 7.1 (Maipo)
Linux ... 3.10.0-229.el7.x86_64 #1 SMP Thu Jan 29 18:37:38 EST 2015 x86_64 x86_64 x86_64 GNU/Linux

Actual results:
After selecting option 1, got the message below.
[ ERROR ] Engine is still unreachable
          Engine health status page is not yet reachable.

Expected results:
I expect to get the output "Engine Replied:DB Up! Welcome to Health Status!" then the installer will register itself ad the virtualization host

Additional info:
I am using gluster volumes as shared storage.
Comment 1 Simone Tiraboschi 2015-10-05 16:52:12 EDT
Could you please also attach ovirt-hosted-engine-setup logs from /var/log/ovirt-hosted-engine-setup/ on your host?
Comment 2 joselito.angeles 2015-10-05 17:12 EDT
Created attachment 1080080 [details]
ovirt hosted engine setup log

Hi Simone,

Here's the log from the host.

Thanks.
Comment 3 Simone Tiraboschi 2015-10-05 19:12:37 EDT
Could you please try to execute
 curl http://ovirt-engine1.perflab.iint/ovirt-engine/services/health
on the host you are deploying?

Is it getting
 DB Up!Welcome to Health Status!
also from there?
Comment 4 joselito.angeles 2015-10-06 11:17:08 EDT
I am not getting the DB Up!Welcome to Health Status from the host  when I run curl. The host can resolve the ipaddress using the hostname and can also do a reverse lookup though.

[root@usatl-s-smperf1 ~]# curl http://ovirt-engine1.perflab.iint/ovirt-engine/services/health
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">
<html>

<head>
<style>
a:link                  {font:8pt/11pt verdana; color:red}
a:visited               {font:8pt/11pt verdana; color:#4e4e4e}
</style>
<meta HTTP-EQUIV="Content-Type" Content="text-html; charset=Windows-1252">
<title>Cannot find server</title>
</head>

<body bgcolor="white">

<table width="400" cellpadding="3" cellspacing="5">
  <tr>
    <td id="tableProps2" align="left" valign="middle" width="360"><h1 id="textSection1"
    style="COLOR: black; FONT: 13pt/15pt verdana"><span id="errorText">The page cannot be displayed</span></h1>
    </td>
  </tr>
  <tr>
    <td id="tablePropsWidth" width="400" colspan="2"><font
    style="COLOR: black; FONT: 8pt/11pt verdana">The page you are looking for is currently
    unavailable. The Web site might be experiencing technical difficulties, or you may need to
    adjust your browser settings.</font></td>
  </tr>
  <tr>
    <td id="tablePropsWidth" width="400" colspan="2"><font id="LID1"
    style="COLOR: black; FONT: 8pt/11pt verdana"><hr color="#C0C0C0" noshade>
    <p id="LID2">Please try the following:</p><ul>
      <li id="instructionsText2">If you typed the page address in the Address bar, make sure that
        it is spelled correctly.<br>
      </li>
    <li id="instructionsText4">
       If you are trying to reach a secure site, make sure your Security settings can support it.
    </li>
     <li id="list3">Click the <a href="javascript:history.back(1)"> Back</a> button to try another link. </li>
    </ul>
    <p><br>
    </p>
    <h2 style="font:8pt/11pt verdana; color:black">DNS Fail</h2>
    </font></td>
  </tr>
</table>
</body>
</html>
[root@usatl-s-smperf1 ~]# nslookup 172.16.128.131
Server:         172.16.128.21
Address:        172.16.128.21#53

131.128.16.172.in-addr.arpa     name = ovirt-engine1.perflab.iint.

[root@usatl-s-smperf1 ~]# nslookup ovirt-engine1.perflab.iint
Server:         172.16.128.21
Address:        172.16.128.21#53

Name:   ovirt-engine1.perflab.iint
Address: 172.16.128.131

[root@usatl-s-smperf1 ~]# ping ovirt-engine1.perflab.iint.
PING ovirt-engine1.perflab.iint (172.16.128.131) 56(84) bytes of data.
64 bytes from ovirt-engine1 (172.16.128.131): icmp_seq=1 ttl=64 time=0.302 ms
64 bytes from ovirt-engine1 (172.16.128.131): icmp_seq=2 ttl=64 time=0.349 ms
Comment 5 Simone Tiraboschi 2015-10-06 11:51:26 EDT
If the engine is reachable, the heath servlet should respond 'DB Up!Welcome to Health Status!' on http://ovirt-engine1.perflab.iint/ovirt-engine/services/health.

On the first message you said that you get 'DB Up!Welcome to Health Status!' if you access it from the browser, is it still true?

Is there on your network any kind of reverse proxy that requires special authentication or something similar?

"<title>Cannot find server</title>" on the response you posted looks quite surprising, at least it should be <title>404 - Page not found</title>

Could you please check if your request for the health page got traced under 
/var/log/httpd/access_log on your engine VM? if not your host is accessing something else.
Comment 6 joselito.angeles 2015-10-06 12:03:47 EDT
Yes, I can access the health status from a browser and from other servers on the same network except from the deployment host. the requests are being traced on the engine VM. The deployment host is 172.16.128.81, the browser is on 172.16.128.39 and the other host (solaris) is on 172.16.128.103. I still have the session for deploy running on 172.1`6.128.81 so its continually trying to get the health status.

172.16.128.103 - - [06/Oct/2015:11:56:31 -0400] "GET /ovirt-engine/services/health HTTP/1.1" 200 31 "-" "curl/7.21.2 (sparc-sun-solaris2.11) libcurl/7.21.2 OpenSSL/1.0.1h zlib/1.2.3-T4mods libidn/1.19"
172.16.128.81 - - [06/Oct/2015:11:56:39 -0400] "GET /ovirt-engine/services/health HTTP/1.1" 200 31 "-" "Python-urllib/2.7"
172.16.128.81 - - [06/Oct/2015:11:56:49 -0400] "GET /ovirt-engine/services/health HTTP/1.1" 200 31 "-" "Python-urllib/2.7"
172.16.128.81 - - [06/Oct/2015:11:56:59 -0400] "GET /ovirt-engine/services/health HTTP/1.1" 200 31 "-" "Python-urllib/2.7"
172.16.128.81 - - [06/Oct/2015:11:57:09 -0400] "GET /ovirt-engine/services/health HTTP/1.1" 200 31 "-" "Python-urllib/2.7"
...
172.16.128.39 - - [06/Oct/2015:11:59:32 -0400] "GET /ovirt-engine/services/health HTTP/1.1" 200 51 "-" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.1; WOW64; Trident/6.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; .NET4.0C; .NET4.0E)"
172.16.128.39 - - [06/Oct/2015:11:59:32 -0400] "GET /favicon.ico HTTP/1.1" 404 209 "-" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.1; WOW64; Trident/6.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; .NET4.0C; .NET4.0E)"
172.16.128.81 - - [06/Oct/2015:11:59:39 -0400] "GET /ovirt-engine/services/health HTTP/1.1" 200 31 "-" "Python-urllib/2.7"
172.16.128.39 - - [06/Oct/2015:11:59:41 -0400] "GET /ovirt-engine/services/health HTTP/1.1" 200 51 "-" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.1; WOW64; Trident/6.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; .NET4.0C; .NET4.0E)"
172.16.128.81 - - [06/Oct/2015:11:59:49 -0400] "GET /ovirt-engine/services/health HTTP/1.1" 200 31 "-" "Python-urllib/2.7"
Comment 7 Simone Tiraboschi 2015-10-06 13:41:12 EDT
Is 172.16.128.81 your host?
Here I see 200 OK 
172.16.128.81 - - [06/Oct/2015:11:59:49 -0400] "GET /ovirt-engine/services/health HTTP/1.1" 200 31 "-" "Python-urllib/2.7"
Comment 8 joselito.angeles 2015-10-06 13:58:01 EDT
Hi Simone,

It was my proxy server that's causing the issue. I disabled my proxy and was able to get the DB Up! 

Thanks for your help. I appreciate your quick responses.

--Joselito

[root@usatl-s-smperf1 ovirt-hosted-engine-ha]# curl -I http://ovirt-engine1.perflab.iint/ovirt-engine/services/health
HTTP/1.0 200 OK
Date: Tue, 06 Oct 2015 17:55:57 GMT
Server: Apache/2.4.6 (Red Hat Enterprise Linux) OpenSSL/1.0.1e-fips
Content-Type: text/html; charset=UTF-8
Content-Length: 31
X-Cache: MISS from usatl-s-ibmperf3
X-Cache-Lookup: MISS from usatl-s-ibmperf3:3128
Via: 1.0 usatl-s-ibmperf3 (squid/3.1.10)
Connection: keep-alive
[root@usatl-s-smperf1 ovirt-hosted-engine-ha]# env|grep proxy
http_proxy=http://plproxy.perflab.iint:3128
ftp_proxy=ftp://plproxy.perflab.iint:3128
https_proxy=https://plproxy.perflab.iint:3128
[root@usatl-s-smperf1 ovirt-hosted-engine-ha]# unset http_proxy
[root@usatl-s-smperf1 ovirt-hosted-engine-ha]# unset ftp_proxy
[root@usatl-s-smperf1 ovirt-hosted-engine-ha]# unset https_proxy
[root@usatl-s-smperf1 ovirt-hosted-engine-ha]# env|grep proxy
[root@usatl-s-smperf1 ovirt-hosted-engine-ha]#  curl http://ovirt-engine1.perflab.iint/ovirt-engine/services/health
DB Up!Welcome to Health Status![root@usatl-s-smperf1 ovirt-hosted-engine-ha]#
[root@usatl-s-smperf1 ovirt-hosted-engine-ha]# curl -I http://ovirt-engine1.perflab.iint/ovirt-engine/services/health
HTTP/1.1 200 OK
Date: Tue, 06 Oct 2015 17:55:17 GMT
Server: Apache/2.4.6 (Red Hat Enterprise Linux) OpenSSL/1.0.1e-fips
Content-Type: text/html; charset=UTF-8
Content-Length: 31
Comment 9 joselito.angeles 2015-10-06 14:19:21 EDT
I set the no_proxy variable to include all local hosts (including the hosted-engine VM) and it's working fine now with proxy enabled.

Thanks again for your help.

Note You need to log in before you can comment on or make changes to this bug.