Bug 1241811

Description of problem:
There are some host issues that make restoring a hosted engine environment less than ideal from a usability viewpoint. The following was experienced during testing for BZ#1232136 (Comments 76, 78, and 87 may be of particular relevance), which involved placing hosted_engine_2 (of two hosted-engine hosts) in maintenance during engine-backup, and using that host to deploy the restored engine (so that its namesake could be easily dropped out from the restored environment to allow it being added anew). 

1) It takes ~10 minutes for the host to become operational in the engine (end of hosted-engine --deploy):
[ INFO  ] Engine replied: DB Up!Welcome to Health Status!
[ INFO  ] Waiting for the host to become operational in the engine. This may take several minutes...
[ INFO  ] Still waiting for VDSM host to become operational...
[ INFO  ] Still waiting for VDSM host to become operational...
[ INFO  ] Still waiting for VDSM host to become operational...
[ INFO  ] Still waiting for VDSM host to become operational...
[ INFO  ] Still waiting for VDSM host to become operational...
[ INFO  ] Still waiting for VDSM host to become operational...
[ INFO  ] Still waiting for VDSM host to become operational...
[ INFO  ] Still waiting for VDSM host to become operational...
[ INFO  ] Still waiting for VDSM host to become operational...
[ INFO  ] Still waiting for VDSM host to become operational...
[ INFO  ] Still waiting for VDSM host to become operational...
[ INFO  ] Still waiting for VDSM host to become operational...
[ INFO  ] Still waiting for VDSM host to become operational...
[ INFO  ] Still waiting for VDSM host to become operational...
[ INFO  ] Still waiting for VDSM host to become operational...
[ INFO  ] Still waiting for VDSM host to become operational...
[ INFO  ] Still waiting for VDSM host to become operational...
[ INFO  ] Still waiting for VDSM host to become operational...
[ INFO  ] Still waiting for VDSM host to become operational...
[ INFO  ] Still waiting for VDSM host to become operational...

2) VDSM instead times out with error:
[ INFO  ] Still waiting for VDSM host to become operational...
[ ERROR ] Timed out while waiting for host to start. Please check the logs.
[ ERROR ] Unable to add hosted_engine_2 to the manager

((Though deployment is ultimately successful:
          Please shutdown the VM allowing the system to launch it as a monitored service.
          The system will wait until the VM is down.
[ INFO  ] Enabling and starting HA services
          Hosted Engine successfully set up
[ INFO  ] Stage: Clean up
[ INFO  ] Generating answer file '/etc/ovirt-hosted-engine/answers.conf'
[ INFO  ] Stage: Pre-termination
[ INFO  ] Stage: Termination
))

3) At end of deployment, the host is in 'Unassigned' state (issue with SPM contention?) 

4) Host needs to be placed into maintenance mode before it can be activated. Placing host in maintenance mode took ~10 minutes, and the Admin Portal didn't alert me when this was finished (Host was in 'Preparing for Maintenance' state, but saw later that there was event in Events tab saying it was in Maintenance mode)

5) Removing hosted-engine host with SPM (hosted_engine_1) cannot be done from within the Admin Portal, but requires force-remove host POST request.

6) Adding hosted_engine_1 runs into the same ~10 minute delay and error:
[ ERROR ] Timed out while waiting for host to start. Please check the logs.
[ ERROR ] Unable to add hosted_engine_1 to the manager
After which it can be manually activated from the Admin Portal.



Version-Release number of selected component (if applicable):
3.4 and 3.5 (quite possibly 3.3 as well, however I have not tested this)

How reproducible:
100% for me

Steps to Reproduce:
1. The various procedures being used are documented here: https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Virtualization/3.4/html-single/Installation_Guide/index.html#sect-Backing_up_and_Restoring_a_Self-Hosted_Environment 

Expected results:
1, 2 and 6) VDSM doesn't time out with error when adding host to environment
3, 4, and 6) Added host should be 'Up'
4) Host shouldn't take ~10 mins to go into maintenance mode; portal should inform user when it's in maintenance
5) Removing host should be able to be done form Administration Portal