Bug 1479747 - The hosted-engine --deploy fails on first attempt on 'Permission denied'
The hosted-engine --deploy fails on first attempt on 'Permission denied'
Product: Red Hat Enterprise Virtualization Manager
Classification: Red Hat
Component: ovirt-hosted-engine-setup (Show other bugs)
Unspecified Unspecified
medium Severity high
: ovirt-4.1.7
: ---
Assigned To: Ido Rosenzwig
Nikolai Sednev
: Unconfirmed
Depends On:
  Show dependency treegraph
Reported: 2017-08-09 06:28 EDT by Jaroslav Spanko
Modified: 2017-10-15 03:46 EDT (History)
5 users (show)

See Also:
Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
Last Closed: 2017-10-15 00:52:46 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
Verified Versions:
Category: ---
oVirt Team: Integration
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---

Attachments (Terms of Use)
vdsm.log first attempt (33.63 KB, text/plain)
2017-08-09 06:29 EDT, Jaroslav Spanko
no flags Details
supervdsm first attempt (65.33 KB, text/plain)
2017-08-09 06:30 EDT, Jaroslav Spanko
no flags Details
deploy log (365.78 KB, text/plain)
2017-08-09 06:31 EDT, Jaroslav Spanko
no flags Details
hosted-engine-setup second sucesfull attempt (523.81 KB, text/plain)
2017-08-30 05:58 EDT, Jaroslav Spanko
no flags Details

  None (edit)
Description Jaroslav Spanko 2017-08-09 06:28:24 EDT
Description of problem:
The deploy of the HE fails on first attempt with
[ ERROR ] Failed to execute stage 'Misc Configuration' : Connection to storage server failed.
The local server is used as NFS,all the permissions and ownerships are set correctly.

Once the deployment fails and it's redeployed again for the second time, it will get deployed successfully without any issue. 

Version-Release number of selected component (if applicable):
RHEL 7.3 and 7.4

How reproducible:
100% in user env

Actual results:
StorageServerAccessPermissionError: Permission settings on the specified path do not allow access to the storage. Verify permission settings on the specified storage path.: 'path = /rhev/data-center/mnt/_var_lib_ovirt-hosted-engine-setup_tmpdjRwix'

Additional info:
We added debug line to the code and this is the result, as uid,guid is root:root instead of vdsm:kvm 
MainProcess|jsonrpc/7::DEBUG::2017-08-07 18:58:17,432::supervdsmServer::93::SuperVdsm.ServerCallback::(wrapper) call validateAccess with ('vdsm', ('kvm',), '/rhev/data-center/mnt/_var_lib_ovirt-hosted-engine-setup_tmpadUXrT', 7) {}

MainProcess|jsonrpc/7::DEBUG::2017-08-07 18:58:17,436::fileUtils::138::storage.fileUtils::(validateAccess) Permissions for directory '/rhev/data-center/mnt/_var_lib_ovirt-hosted-engine-setup_tmpadUXrT' are '0755', stat output is 'posix.stat_result(st_mode=16877, st_ino=2, st_dev=1793L, st_nlink=3, st_uid=0, st_gid=0, st_size=4096, st_atime=1502112496, st_mtime=1502112496, st_ctime=1502112496)'

MainProcess|jsonrpc/7::WARNING::2017-08-07 18:58:17,436::fileUtils::141::storage.fileUtils::(validateAccess) Permission denied for directory: /rhev/data-center/mnt/_var_lib_ovirt-hosted-engine-setup_tmpadUXrT with permissions: 7

Permissions during deploy
drwxr-xr-x root root ?                                ..
drwxr-xr-x root root ?                                .
drwxr-xr-x root root ?                                answers
-rw------- vdsm kvm  ?                                tmp9SjQGH

drwxr-xr-x vdsm kvm ?                                ..
drwxrwxrwx vdsm kvm ?                      
drwxr-xr-x vdsm kvm ?                                .
Comment 1 Jaroslav Spanko 2017-08-09 06:29 EDT
Created attachment 1311155 [details]
vdsm.log first attempt
Comment 2 Jaroslav Spanko 2017-08-09 06:30 EDT
Created attachment 1311156 [details]
supervdsm first attempt
Comment 3 Jaroslav Spanko 2017-08-09 06:31 EDT
Created attachment 1311157 [details]
deploy log
Comment 4 Sandro Bonazzola 2017-08-09 07:19:49 EDT
Please check the NFS configuration, ensure it's not exported with root_squash.
Looks like a not supported configuration of the NFS storage.
See also bug #1466234
Comment 5 Jaroslav Spanko 2017-08-09 07:41:59 EDT
Hi Sandro

You're right , it is exported with root_squash as default for RHEL

What is strange is that I'm not able to reproduce this in our env with the same export.
I will ask to change to non_root_squash and let you know.

Comment 6 Ido Rosenzwig 2017-08-15 07:00:55 EDT
Hi Jaroslav,

Can you please update?
Did you try with non_root_squash ?

Comment 7 Jaroslav Spanko 2017-08-16 06:23:39 EDT
Hi Ido
Unfortunately still no , we're waiting for the CU update.
As i wrote is 100% reproducible in user env, we were not able to reproduce it yet.
Comment 8 Jaroslav Spanko 2017-08-23 04:41:10 EDT
We got answer from the customer regarding this, unfortunately no_root_squash did not help
the problem still persists with RHEL 7.4
During past week, we deployed the self-hosted engine setup on 5 servers out of which 3 are deployed remotely (from PuTTY) and 2 are deployed directly from the host itself.
The deployment performed remotely (via PuTTY) on the 3 servers is successful in the very first attempt where as the deployment performed directly on 2 servers is failed in the first attempt as usual.

Comment 12 Jaroslav Spanko 2017-08-30 05:58 EDT
Created attachment 1319974 [details]
hosted-engine-setup second sucesfull attempt
Comment 13 Ido Rosenzwig 2017-08-30 08:56:58 EDT
Hi Jaroslav,

I didn't manage to reproduce it so far.
I've tried to deploy it via GUI on RHEL 7.3 with the following NFS configuration:


which is exactly like the customer's configuration (without root_squash).

I believe this is a problem with the customer's configuration.

please provide the answer from the customer after he tried the configuration suggested in comment 10 .

Thank you.
Comment 14 Ido Rosenzwig 2017-10-15 00:52:46 EDT
No answer was given in more then a month. 
Closing the bug.
Comment 15 Jaroslav Spanko 2017-10-15 03:46:51 EDT
Hi Ido
As i wrote you, the CU still do not replied so I'm ok with close.
Thanks for your help.

Note You need to log in before you can comment on or make changes to this bug.