Bug 1291705

The fix for bug 1286565 restores support for replica counts other than 1 and 3 with a warning in VDSM side.

Having an alert in the engine is an entire new mechanism, and out of scope for oVirt 3.6.z.

Yaniv, is this still relevant in 4.2? IIRC, we relaxed the support statement?

(In reply to Allon Mureinik from comment #2)
> Yaniv, is this still relevant in 4.2? IIRC, we relaxed the support statement?

It's either 1 or 3 still. I don't see this changing anytime soon.

imo it is still relevant and I suppose not supported replica number was issue for 'permissions error' in BZ1259441.

2017-10-05 14:32:27,383+0200 INFO  (jsonrpc/5) [vdsm.api] START connectStorageServer(domType=7, spUUID=u'00000000-0000-0000-0000-000000000000', conList=[{u'id': u'00000000-0000-0000-0000-000000000000', u'connecti
on': u'10-37-138-193.example.com:/data', u'iqn': u'', u'user': u'', u'tpgt': u'1', u'vfs_type': u'glusterfs', u'password': '********', u'port': u''}], options=None) from=::ffff:10.37.138.3,40352, 
flow_id=d9b4dad2-0ef1-40b5-8108-53e1f725c62e, task_id=351a7bea-a2e7-4230-85a7-386950acc453 (api:46)
2017-10-05 14:32:27,422+0200 INFO  (jsonrpc/6) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.01 seconds (__init__:630)
2017-10-05 14:32:27,517+0200 WARN  (jsonrpc/5) [storage.StorageServer.MountConnection] Unsupported replica count (2) for volume u'data', please upgrade volume to replica 3 (storageServer:282)
2017-10-05 14:32:27,518+0200 INFO  (jsonrpc/5) [storage.Mount] mounting 10-37-138-193.example.com:/data at /rhev/data-center/mnt/glusterSD/10-37-138-193.example.com:_data (mount:20
5)
2017-10-05 14:32:27,861+0200 INFO  (jsonrpc/5) [IOProcessClient] Starting client ioprocess-0 (__init__:319)
2017-10-05 14:32:27,890+0200 INFO  (ioprocess/23856) [IOProcess] Starting ioprocess (__init__:445)
2017-10-05 14:32:27,917+0200 INFO  (jsonrpc/5) [storage.Mount] unmounting /rhev/data-center/mnt/glusterSD/10-37-138-193.example.com:_data (mount:213)
2017-10-05 14:32:27,976+0200 ERROR (jsonrpc/5) [storage.HSM] Could not connect to storageServer (hsm:2357)
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 2354, in connectStorageServer
    conObj.connect()
  File "/usr/lib/python2.7/site-packages/vdsm/storage/storageServer.py", line 174, in connect
    six.reraise(t, v, tb)
  File "/usr/lib/python2.7/site-packages/vdsm/storage/storageServer.py", line 167, in connect
    self.getMountObj().getRecord().fs_file)
  File "/usr/lib/python2.7/site-packages/vdsm/storage/fileSD.py", line 86, in validateDirAccess
    raise se.StorageServerAccessPermissionError(dirPath)
StorageServerAccessPermissionError: Permission settings on the specified path do not allow access to the storage. Verify permission settings on the specified storage path.: 'path = /rhev/data-center/mnt/glusterSD/10-37-138-193.example.com:_data'

In Admin Portal one received a little bit misleading event about permissions error:

- admin portal:

The error message for connection 10-37-138-193.example.com:/data returned by VDSM was: Permission settings on the specified path do not allow access to the storage. Verify permission settings on the specified storage path.

when in fact it is unsupported replica number for gluster volume:

- vdsm:

2017-10-05 14:32:27,517+0200 WARN  (jsonrpc/5) [storage.StorageServer.MountConnection] Unsupported replica count (2) for volume u'data', please upgrade volume to replica 3 (storageServer:282)
2017-10-05 14:32:27,518+0200 INFO  (jsonrpc/5) [storage.Mount] mounting 10-37-138-193.rhev.lab.eng.brq.redhat.com:/data at /rhev/data-center/mnt/glusterSD/10-37-138-193.example.com:_data (mount:20
5)

- engine.log:


2017-10-05 14:32:27,316+02 INFO  [org.ovirt.engine.core.bll.storage.connection.AddStorageServerConnectionCommand] (default task-23) [d9b4dad2-0ef1-40b5-8108-53e1f725c62e] Running command: AddStorageServerConnectionCommand internal: false. Entities affected :  ID: aaa00000
-0000-0000-0000-123456789aaa Type: SystemAction group CREATE_STORAGE_DOMAIN with role type ADMIN
2017-10-05 14:32:27,344+02 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (default task-23) [d9b4dad2-0ef1-40b5-8108-53e1f725c62e] START, ConnectStorageServerVDSCommand(HostName = 10-37-138-193, StorageServerConnectionManagementVDSParamet
ers:{hostId='653be0fe-389a-4231-bb54-8bf52da66d5d', storagePoolId='00000000-0000-0000-0000-000000000000', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='null', connection='10-37-138-193.rhev.lab.eng.brq.redhat.com:/data', iqn='null', vfsType='glus
terfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 398f4160
2017-10-05 14:32:28,039+02 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (default task-23) [d9b4dad2-0ef1-40b5-8108-53e1f725c62e] FINISH, ConnectStorageServerVDSCommand, return: {00000000-0000-0000-0000-000000000000=469}, log id: 398f416
0
2017-10-05 14:32:28,059+02 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-23) [d9b4dad2-0ef1-40b5-8108-53e1f725c62e] EVENT_ID: STORAGE_DOMAIN_ERROR(996), The error message for connection 10-37-138-193.example.com:/data returned by VDSM was: Permission settings on the specified path do not allow access to the storage.
Verify permission settings on the specified storage path.
2017-10-05 14:32:28,060+02 ERROR [org.ovirt.engine.core.bll.storage.connection.FileStorageHelper] (default task-23) [d9b4dad2-0ef1-40b5-8108-53e1f725c62e] The connection with details '10-37-138-193.example.com:/data' failed because of error code '469' and error message is: permission settings on the specified path do not allow access to the storage.
verify permission settings on the specified storage path.
2017-10-05 14:32:28,061+02 ERROR [org.ovirt.engine.core.bll.storage.connection.AddStorageServerConnectionCommand] (default task-23) [d9b4dad2-0ef1-40b5-8108-53e1f725c62e] Command 'org.ovirt.engine.core.bll.storage.connection.AddStorageServerConnectionCommand' failed: EngineException: StorageServerAccessPermissionError (Failed with error StorageServerAccessPermissionError and code 469)
2017-10-05 14:32:28,118+02 ERROR [org.ovirt.engine.core.bll.storage.connection.AddStorageServerConnectionCommand] (default task-23) [d9b4dad2-0ef1-40b5-8108-53e1f725c62e] Transaction rolled-back for command 'org.ovirt.engine.core.bll.storage.connection.AddStorageServerConnectionCommand'.
2017-10-05 14:32:28,132+02 INFO  [org.ovirt.engine.core.bll.storage.connection.AddStorageServerConnectionCommand] (default task-23) [d9b4dad2-0ef1-40b5-8108-53e1f725c62e] Lock freed to object 'EngineLock:{exclusiveLocks='[10-37-138-193.example.com:/data=STORAGE_CONNECTION]', sharedLocks=''}'

This bug has not been marked as blocker for oVirt 4.3.0.
Since we are releasing it tomorrow, January 29th, this bug has been re-targeted to 4.3.1.

No need for UI text. Just make sure it is in documentation

Moving to 4.3.3 being 4.3.2 already released

Tal can you please re-target? didn't make it into 4.3.5

Cleaning target milestone, please retarget.

Cleaning target milestone, please retarget.

Freddy, what exactly is the request? I've read through the thread and it's not clear to me.

We need to make sure that in the documentation it is stated that when adding Gluster storage domain either replica 1 or 3 should be used.

I'm looking for where we mention adding Storage Domains, and Gluster storage. I only find it mentioned in the Adminstration Guide:

ADMINISTRATION GUIDE

13.1.8.2. Restoring the Backup on a New Self-Hosted Engine
https://access.redhat.com/documentation/en-us/red_hat_virtualization/4.3/html-single/administration_guide/index#Backing_up_and_Restoring_a_Self-hosted_Engine

step 17 has this admonition:

---------------------------
Important

Only replica 3 Gluster storage is supported.
---------------------------

I can change this to "Only replica 1 and replica 3 Gluster storage is supported."



Other than that, we don't actually discuss adding a Gluster domain, instead we reference the Gluster docs in this section:

8.6. Preparing and Adding Red Hat Gluster Storage, which references the Gluster docs.
https://access.redhat.com/documentation/en-us/red_hat_virtualization/4.3/html-single/administration_guide/index#Adding_Red_Hat_Gluster_Storage

Freddy, please approve the change I suggested in comment 15.

Freddy,

I created a merge request at: 
https://gitlab.cee.redhat.com/rhci-documentation/docs-Red_Hat_Enterprise_Virtualization/merge_requests/1465/diffs

You can see the changes I made here:
https://pnt-devops-rad-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/ccs-mr-preview-prod/9059/artifact/doc-Administration_Guide/preview/index.html#Backing_up_and_Restoring_a_Self-hosted_Engine
See section 13.1.8.2 Restoring the Backup on a New Self-Hosted Engine, step 17. There's an "Important" there, as I noted in comment 15. I updated it to include replica 1.

Sahina,

Can someone from RHHI team approve the text?

Thanks,
Freddy

(In reply to Fred Rolland from comment #18)
> Sahina,
> 
> Can someone from RHHI team approve the text?
> 
> Thanks,
> Freddy

Looks good to me

(In reply to Sahina Bose from comment #19)
> (In reply to Fred Rolland from comment #18)
> > Sahina,
> > 
> > Can someone from RHHI team approve the text?
> > 
> > Thanks,
> > Freddy
> 
> Looks good to me

I just noticed some additional changes that are needed. In the configuration, the options listed are outdated. Please use the options listed from "Options set on Gluster Storage Volumes to Store Virtual Machine Images" at https://ovirt.org/documentation/admin-guide/chap-Working_with_Gluster_Storage.html

(In reply to Sahina Bose from comment #20)

> I just noticed some additional changes that are needed. In the
> configuration, the options listed are outdated. Please use the options
> listed from "Options set on Gluster Storage Volumes to Store Virtual Machine
> Images" at
> https://ovirt.org/documentation/admin-guide/chap-
> Working_with_Gluster_Storage.html

Can you please add your comments directly in the merge request? Or if you don't have the right permissions, can you please be specific about the changes you want, including the location in the document?

(In reply to Steve Goodman from comment #21)
> (In reply to Sahina Bose from comment #20)
> 
> > I just noticed some additional changes that are needed. In the
> > configuration, the options listed are outdated. Please use the options
> > listed from "Options set on Gluster Storage Volumes to Store Virtual Machine
> > Images" at
> > https://ovirt.org/documentation/admin-guide/chap-
> > Working_with_Gluster_Storage.html
> 
> Can you please add your comments directly in the merge request? Or if you
> don't have the right permissions, can you please be specific about the
> changes you want, including the location in the document?

Done

Hi Sahina,

Please take a (hopefully last) look at my implementation of your comments. If all looks good, I'll move forward with this.

(In reply to Steve Goodman from comment #23)
> Hi Sahina,
> 
> Please take a (hopefully last) look at my implementation of your comments.
> If all looks good, I'll move forward with this.

Commented on your PR. Looks good. Thanks for reworking this!

Closed. This is now merged to master, so it will be published with 4.4 Beta.