Bug 1166002

Summary: [hosted-engine] Setup fails after rhevm installation copleteted with: "Cannot automatically add the host to cluster None: JBWEB000065: HTTP Status 401"
Product: Red Hat Enterprise Virtualization Manager Reporter: Elad <ebenahar>
Component: ovirt-hosted-engine-setupAssignee: Simone Tiraboschi <stirabos>
Status: CLOSED NOTABUG QA Contact: meital avital <mavital>
Severity: urgent Docs Contact:
Priority: unspecified    
Version: 3.5.0CC: didi, ecohen, gklein, iheim, jmoskovc, lsurette, lveyde, msivak, pstehlik, stirabos
Target Milestone: ---Keywords: Reopened, Triaged
Target Release: 3.5.0   
Hardware: x86_64   
OS: Unspecified   
Whiteboard: integration
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2014-11-23 09:49:51 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1157385    
Attachments:
Description Flags
/var/log/ from the host
none
logs 26.11.14 none

Description Elad 2014-11-20 09:09:11 UTC
Created attachment 959264 [details]
/var/log/ from the host

Description of problem:
Deployed hosted-engine, used iscsi. When I got to the phase when rhevm installation was completed on the engine's VM, picked:
(1) Continue setup - engine installation is complete

Then, installation crashed with:

          (1) Continue setup - engine installation is complete
          (2) Power off and restart the VM
          (3) Abort setup
         
          (1, 2, 3)[1]: 1
[ INFO  ] Engine replied: DB Up!Welcome to Health Status!
[ ERROR ] Cannot automatically add the host to cluster None: JBWEB000065: HTTP Status 401 
[ ERROR ] Failed to execute stage 'Closing up': Cannot add the host to cluster None
[ INFO  ] Stage: Clean up
[ INFO  ] Generating answer file '/var/lib/ovirt-hosted-engine-setup/answers/answers-20141120105832.conf'
[ INFO  ] Stage: Pre-termination
[ INFO  ] Stage: Termination


Version-Release number of selected component (if applicable):

rhev 3,5 vt10.1
ovirt-hosted-engine-setup-1.2.1-3.el6ev.noarch
ovirt-hosted-engine-ha-1.2.4-1.el6ev.noarch

Host: 
[root@green-vdsa ~]# cat /etc/redhat-release 
Red Hat Enterprise Linux Server release 6.6 (Santiago)

How reproducible:
Tried once

Steps to Reproduce:
1. hosted-engine --deploy, pick iscsi
2. when rhevm installation is completed on engine's VM, continue the deployment
3.

Actual results:
Deployment failed with the mentioned error.

Expected results:
Deployment should succeed

Additional info:
/var/log/ from the host

Comment 1 Sandro Bonazzola 2014-11-20 10:24:00 UTC
Simone, you should have around a system ready for iSCSI testing, can you try to reproduce?
The engine was up because Engine replied: DB Up!Welcome to Health Status!
But API failed login with unauthorized access.

Comment 2 Simone Tiraboschi 2014-11-20 10:35:15 UTC
Yes, I'll try

Comment 3 Simone Tiraboschi 2014-11-20 14:08:45 UTC
Ok,
reproduced:

[ INFO  ] Engine replied: DB Up!Welcome to Health Status!
[ ERROR ] Cannot automatically add the host to cluster None: HTTP Status 401 
[ ERROR ] Failed to execute stage 'Closing up': Cannot add the host to cluster None
[ INFO  ] Stage: Clean up
[ INFO  ] Generating answer file '/etc/ovirt-hosted-engine/answers.conf'
[ INFO  ] Answer file '/etc/ovirt-hosted-engine/answers.conf' has been updated
[ INFO  ] Stage: Pre-termination
[ INFO  ] Stage: Termination

Entering a wrong engine password when hosted-engine-setup asks it seams to be the only way to reproduce it: the engine is up but hosted-engine-setup is not able to authenticate against its APIs due to the wrong password.
The error message is not that consistent.

We already have another bug about asking again that password if not valid, see:
https://bugzilla.redhat.com/show_bug.cgi?id=1108341

Please reopen if you are able to replicate using the correct password.

Comment 4 Elad 2014-11-23 09:43:46 UTC
Created attachment 960408 [details]
logs 26.11.14

Tried to reproduce. Deployed the hosted-engine on a 'clean' environment (no VM running on the host and no storage domain exists on the used LUN). Failed again, this time with the following:

          (1, 2, 3)[1]: 1
[ INFO  ] Engine replied: DB Up!Welcome to Health Status!
          Enter the name of the cluster to which you want to add the host (Default) [Default]: 
[ INFO  ] Waiting for the host to become operational in the engine. This may take several minutes...
[ INFO  ] Still waiting for VDSM host to become operational...
[ INFO  ] Still waiting for VDSM host to become operational...
[ INFO  ] Still waiting for VDSM host to become operational...
[ ERROR ] The VDSM host was found in a failed state. Please check engine and bootstrap installation logs.
[ ERROR ] Unable to add hosted_engine_1 to the manager
          Please shutdown the VM allowing the system to launch it as a monitored service.
          The system will wait until the VM is down.
[ ERROR ] Failed to execute stage 'Closing up': [Errno 111] Connection refused
[ INFO  ] Stage: Clean up
[ ERROR ] Failed to execute stage 'Clean up': [Errno 111] Connection refused
[ INFO  ] Generating answer file '/var/lib/ovirt-hosted-engine-setup/answers/answers-20141123113619.conf'
[ INFO  ] Stage: Pre-termination
[ INFO  ] Stage: Termination

Comment 5 Elad 2014-11-23 09:44:55 UTC
Although this is not the same error, I'm re-opening this bug

Comment 6 Yedidyah Bar David 2014-11-23 09:49:51 UTC
(In reply to Elad from comment #5)
> Although this is not the same error, I'm re-opening this bug

Please open a new bug with correct summary line and attach relevant logs to it. Thanks!