Created attachment 952101 [details]
/var/log and /var/lib/ovirt-hosted-engine-setup/ from both hosts

Description of problem:
Deployed hosted-engine in intention to add an new host to the existing hosted-engine setup. The deployment failed in the fetching answer file stage. 

Version-Release number of selected component (if applicable):
rhev3.5 vt7 
ovirt-hosted-engine-setup-1.2.1-2.el7.noarch

How reproducible:
Every time in my environment 

Steps to Reproduce:
1. Have a hosted-engine setup up and running (using iSCSI storage)
2. Deploy a second host:

Please specify the lun id (0, 1, 2, 3) [0]: 3
          The specified storage location already contains a data domain. Is this an additional host setup (Yes, No)[Yes]? 
[ INFO  ] Installing on additional host
          Please specify the Host ID [Must be integer, default: 2]: 
          Please provide storage domain name. [hosted_storage]: 
          Local storage datacenter name is an internal name and currently will not be shown in engine's admin UI.Please enter local datacenter name [hosted_datacenter]:


Actual results:
Deployment fails with:

[WARNING] A configuration file must be supplied to deploy Hosted Engine on an additional host.
          The answer file may be fetched from the first host using scp.
          If you do not want to download it automatically you can abort the setup answering no to the following question.
          Do you want to scp the answer file from the first host? (Yes, No)[Yes]: 
          Please provide the FQDN or IP of the first host: green-vdsc.qa.lab.tlv.redhat.com
          Enter 'root' user password for host green-vdsc.qa.lab.tlv.redhat.com: 
[ ERROR ] Failed to execute stage 'Environment customization': [Errno 2] No such file
[ INFO  ] Stage: Clean up
[ INFO  ] Generating answer file '/var/lib/ovirt-hosted-engine-setup/answers/answers-20141030134631.conf'
[ INFO  ] Stage: Pre-termination
[ INFO  ] Stage: Termination



2014-10-30 13:38:04 DEBUG otopi.plugins.ovirt_hosted_engine_setup.core.remote_answerfile remote_answerfile._fetch_answer_file:106 _fetch_answer_file
2014-10-30 13:38:04 DEBUG otopi.plugins.otopi.dialog.human human.queryString:153 query HOST_FIRST_HOST_ROOT_PASSWORD
2014-10-30 13:38:04 DEBUG otopi.plugins.otopi.dialog.human dialog.__logString:215 DIALOG:SEND                 Enter 'root' user password for host green-vdsc.qa.lab.tlv.redhat.com: 
2014-10-30 13:38:11 DEBUG otopi.context context._executeMethod:152 method exception
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/otopi/context.py", line 142, in _executeMethod
    method['method']()
  File "/usr/share/ovirt-hosted-engine-setup/scripts/../plugins/ovirt-hosted-engine-setup/core/remote_answerfile.py", line 283, in _customization
    self._fetch_answer_file()
  File "/usr/share/ovirt-hosted-engine-setup/scripts/../plugins/ovirt-hosted-engine-setup/core/remote_answerfile.py", line 143, in _fetch_answer_file
    self._tmp_ans
  File "/usr/lib/python2.7/site-packages/paramiko/sftp_client.py", line 636, in get
    file_size = self.stat(remotepath).st_size
  File "/usr/lib/python2.7/site-packages/paramiko/sftp_client.py", line 319, in stat
    t, msg = self._request(CMD_STAT, path)
  File "/usr/lib/python2.7/site-packages/paramiko/sftp_client.py", line 652, in _request
    return self._read_response(num)
  File "/usr/lib/python2.7/site-packages/paramiko/sftp_client.py", line 699, in _read_response
    self._convert_status(msg)
  File "/usr/lib/python2.7/site-packages/paramiko/sftp_client.py", line 725, in _convert_status
    raise IOError(errno.ENOENT, text)
IOError: [Errno 2] No such file
2014-10-30 13:38:11 ERROR otopi.context context._executeMethod:161 Failed to execute stage 'Environment customization': [Errno 2] No such file


Checked the first host (the one with HE installed). Answer file exist there:

[root@green-vdsc ~]# ll /var/lib/ovirt-hosted-engine-setup/answers/
total 8
-rw-r--r--. 1 root root 2197 Oct 30 09:44 answers-20141030094440.conf
-rw-rw----. 1 root root 2465 Oct 30 10:38 answers-20141030103802.conf



Expected results:
The answer file exists on the first host so setup should not fail

Additional info: /var/log and /var/lib/ovirt-hosted-engine-setup/ from both hosts
host1 is the already installed host and host2 is the new additional host