Bug 1479747 - The hosted-engine --deploy fails on first attempt on 'Permission denied' [NEEDINFO]
The hosted-engine --deploy fails on first attempt on 'Permission denied'
Status: NEW
Product: Red Hat Enterprise Virtualization Manager
Classification: Red Hat
Component: ovirt-hosted-engine-setup (Show other bugs)
4.0.0
Unspecified Unspecified
medium Severity high
: ovirt-4.1.7
: ---
Assigned To: Ido Rosenzwig
Nikolai Sednev
: Unconfirmed
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2017-08-09 06:28 EDT by Jaroslav Spanko
Modified: 2017-09-19 08:20 EDT (History)
5 users (show)

See Also:
Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed:
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: Integration
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---
irosenzw: needinfo? (jspanko)


Attachments (Terms of Use)
vdsm.log first attempt (33.63 KB, text/plain)
2017-08-09 06:29 EDT, Jaroslav Spanko
no flags Details
supervdsm first attempt (65.33 KB, text/plain)
2017-08-09 06:30 EDT, Jaroslav Spanko
no flags Details
deploy log (365.78 KB, text/plain)
2017-08-09 06:31 EDT, Jaroslav Spanko
no flags Details
hosted-engine-setup second sucesfull attempt (523.81 KB, text/plain)
2017-08-30 05:58 EDT, Jaroslav Spanko
no flags Details

  None (edit)
Description Jaroslav Spanko 2017-08-09 06:28:24 EDT
Description of problem:
The deploy of the HE fails on first attempt with
[ ERROR ] Failed to execute stage 'Misc Configuration' : Connection to storage server failed.
The local server is used as NFS,all the permissions and ownerships are set correctly.
RHV_STORAGE/DATA_DOMAIN
                <world>(rw,async,wdelay,root_squash,no_subtree_check,sec=sys,rw,secure,root_squash,no_all_squash)
/RHV_STORAGE/RHVM_DATA
                <world>(rw,async,wdelay,root_squash,no_subtree_check,sec=sys,rw,secure,root_squash,no_all_squash)

Once the deployment fails and it's redeployed again for the second time, it will get deployed successfully without any issue. 

Version-Release number of selected component (if applicable):
RHEL 7.3 and 7.4

How reproducible:
100% in user env

Actual results:
StorageServerAccessPermissionError: Permission settings on the specified path do not allow access to the storage. Verify permission settings on the specified storage path.: 'path = /rhev/data-center/mnt/_var_lib_ovirt-hosted-engine-setup_tmpdjRwix'


Additional info:
We added debug line to the code and this is the result, as uid,guid is root:root instead of vdsm:kvm 
MainProcess|jsonrpc/7::DEBUG::2017-08-07 18:58:17,432::supervdsmServer::93::SuperVdsm.ServerCallback::(wrapper) call validateAccess with ('vdsm', ('kvm',), '/rhev/data-center/mnt/_var_lib_ovirt-hosted-engine-setup_tmpadUXrT', 7) {}

MainProcess|jsonrpc/7::DEBUG::2017-08-07 18:58:17,436::fileUtils::138::storage.fileUtils::(validateAccess) Permissions for directory '/rhev/data-center/mnt/_var_lib_ovirt-hosted-engine-setup_tmpadUXrT' are '0755', stat output is 'posix.stat_result(st_mode=16877, st_ino=2, st_dev=1793L, st_nlink=3, st_uid=0, st_gid=0, st_size=4096, st_atime=1502112496, st_mtime=1502112496, st_ctime=1502112496)'

MainProcess|jsonrpc/7::WARNING::2017-08-07 18:58:17,436::fileUtils::141::storage.fileUtils::(validateAccess) Permission denied for directory: /rhev/data-center/mnt/_var_lib_ovirt-hosted-engine-setup_tmpadUXrT with permissions: 7

Permissions during deploy
/var/lib/ovirt-hosted-engine-setup/:
drwxr-xr-x root root ?                                ..
drwxr-xr-x root root ?                                .
drwxr-xr-x root root ?                                answers
-rw------- vdsm kvm  ?                                tmp9SjQGH

/rhev/data-center/mnt:
drwxr-xr-x vdsm kvm ?                                ..
drwxrwxrwx vdsm kvm ?                                10.53.197.129:_RHV__STORAGE_RHVM__DATA
drwxr-xr-x vdsm kvm ?                                .
Comment 1 Jaroslav Spanko 2017-08-09 06:29 EDT
Created attachment 1311155 [details]
vdsm.log first attempt
Comment 2 Jaroslav Spanko 2017-08-09 06:30 EDT
Created attachment 1311156 [details]
supervdsm first attempt
Comment 3 Jaroslav Spanko 2017-08-09 06:31 EDT
Created attachment 1311157 [details]
deploy log
Comment 4 Sandro Bonazzola 2017-08-09 07:19:49 EDT
Please check the NFS configuration, ensure it's not exported with root_squash.
Looks like a not supported configuration of the NFS storage.
See also bug #1466234
Comment 5 Jaroslav Spanko 2017-08-09 07:41:59 EDT
Hi Sandro

You're right , it is exported with root_squash as default for RHEL
/RHV_STORAGE/RHVM_DATA
<world>(rw,async,wdelay,root_squash,no_subtree_check,sec=sys,rw,secure,root_squash,no_all_squash)

What is strange is that I'm not able to reproduce this in our env with the same export.
I will ask to change to non_root_squash and let you know.

Thanks
Comment 6 Ido Rosenzwig 2017-08-15 07:00:55 EDT
Hi Jaroslav,

Can you please update?
Did you try with non_root_squash ?

Thanks
Comment 7 Jaroslav Spanko 2017-08-16 06:23:39 EDT
Hi Ido
Unfortunately still no , we're waiting for the CU update.
As i wrote is 100% reproducible in user env, we were not able to reproduce it yet.
Thanks
Comment 8 Jaroslav Spanko 2017-08-23 04:41:10 EDT
Hi 
We got answer from the customer regarding this, unfortunately no_root_squash did not help
~~~
the problem still persists with RHEL 7.4
During past week, we deployed the self-hosted engine setup on 5 servers out of which 3 are deployed remotely (from PuTTY) and 2 are deployed directly from the host itself.
The deployment performed remotely (via PuTTY) on the 3 servers is successful in the very first attempt where as the deployment performed directly on 2 servers is failed in the first attempt as usual.
~~~

Thanks
Comment 12 Jaroslav Spanko 2017-08-30 05:58 EDT
Created attachment 1319974 [details]
hosted-engine-setup second sucesfull attempt
Comment 13 Ido Rosenzwig 2017-08-30 08:56:58 EDT
Hi Jaroslav,

I didn't manage to reproduce it so far.
I've tried to deploy it via GUI on RHEL 7.3 with the following NFS configuration:

   *(rw,async,wdelay,no_subtree_check,sec=sys,rw,secure,no_all_squash)


which is exactly like the customer's configuration (without root_squash).

I believe this is a problem with the customer's configuration.

please provide the answer from the customer after he tried the configuration suggested in comment 10 .

Thank you.

Note You need to log in before you can comment on or make changes to this bug.