Bug 1643730

Summary:	[upgrade] Post upgrade, new options are not available in virt profile
Product:	[Red Hat Storage] Red Hat Gluster Storage	Reporter:	SATHEESARAN <sasundar>
Component:	rhhi	Assignee:	Sahina Bose <sabose>
Status:	CLOSED CURRENTRELEASE	QA Contact:	SATHEESARAN <sasundar>
Severity:	high	Docs Contact:
Priority:	high
Version:	rhhiv-1.5	CC:	guillaume.pavese, lbailey, rhs-bugs, sabose, sankarshan, sasundar
Target Milestone:	---	Keywords:	Tracking
Target Release:	RHHI-V 1.5.z Async
Hardware:	x86_64
OS:	Linux
Whiteboard:
Fixed In Version:	imgbased-1.1.2	Doc Type:	Known Issue
Doc Text:	When a virtualization host is upgraded to Red Hat Hyperconverged infrastructure for Virtualization 1.5, the performance profile for the virt group is not updated with parameters added during development. This means that upgraded deployments are not optimally tuned for most use cases. To work around this issue, add the following options to the /var/lib/glusterd/groups/virt file on each server, or manually set these options on your upgraded volumes. cluster.choose-local=off client.event-threads=4 server.event-threads=4 performance.client-io-threads=on	Story Points:	---
Clone Of:
Clones:	1643733 (view as bug list)		Environment:
Last Closed:	2019-05-20 04:54:49 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:	1643733
Bug Blocks:	1548985

Description SATHEESARAN 2018-10-28 01:13:29 UTC

Description of problem:
-----------------------
After upgrading from RHV 4.1.11, the new virt options are not available in virt profile ( /var/lib/glusterd/groups/virt )

Version-Release number of selected component (if applicable):
--------------------------------------------------------------
RHVH 4.1.11
RHVH 4.2.7

How reproducible:
------------------
6/6

Steps to Reproduce:
-------------------
1. Install RHVH 4.1.11
2. Check for the contents of virt profile 
3. Update to the latest RHVH 4.2.7
4. Check for the contents of virt profile

Actual results:
---------------
New virt profile still points to the old content


Expected results:
------------------
New virt profile should have the new content
Additional info:

Comment 1 SATHEESARAN 2018-10-28 01:15:37 UTC

[root@ ~]# imgbase w
You are on rhvh-4.2.7.3-0.20181026.0+1

[root@ ~]# rpm -qa | grep glusterfs-server
glusterfs-server-3.12.2-25.el7rhgs.x86_64

[root@ ~]# rpm -qf /var/lib/glusterd/groups/virt 
glusterfs-server-3.12.2-25.el7rhgs.x86_64

[root@ ~]# cat /var/lib/glusterd/groups/virt
performance.quick-read=off
performance.read-ahead=off
performance.io-cache=off
performance.low-prio-threads=32
network.remote-dio=enable
cluster.eager-lock=enable
cluster.quorum-type=auto
cluster.server-quorum-type=server
cluster.data-self-heal-algorithm=full
cluster.locking-scheme=granular
cluster.shd-max-threads=8
cluster.shd-wait-qlength=10000
features.shard=on
user.cifs=off


Note that the new options are missing in this virt profile

Comment 2 SATHEESARAN 2018-10-28 01:19:00 UTC

I upgraded the gluster rpms from glusterfs-3.8.4-54.15 to 3.12.2-25
The contents of virt profile were updated.

I suspect this to be the problem with RHVH image update. 
Will just try to understand it with 2 tests

Test1
1. Install RHVH 4.1.11
2. Check for contents
3. Do image update from CLI ( # yum update )
4. Check for contents of virt profile
5. Reboot
6. Check for contents of virt profile

Test2
1. Install RHVH 4.1.11
2. Check for contents
3. Remove virt profile
3. Do image update from CLI ( # yum update )
4. Check for contents of virt profile
5. Reboot
6. Check for contents of virt profile

I will update the results for reference

Comment 3 SATHEESARAN 2018-10-28 02:53:30 UTC

This looks like the RHVH related issue. I have tested with single node upgrade as mentioned in comment2

Test1
-----
1. Created a VM and installed it with RHVH 4.2.6 ( which had glusterfs-3.8.4-54.15.el7rhgs )
2. Updated to RHVH 4.2.7 image ( redhat-virtuaization-host-image-update )
3. After update and reboot, I observed the contents of virt profile remained the same.

Test2
-------
1. Created a VM and installed it with RHVH 4.2.6 ( which had glusterfs-3.8.4-54.15.el7rhgs )
2. Removed the virt profile ( rm -rf /var/lib/glusterd/groups/virt )
2. Updated to RHVH 4.2.7 image ( redhat-virtuaization-host-image-update )
3. After updating, I observed that the virt profile is created new and the contents of virt profile had the recently added params.

Comment 4 SATHEESARAN 2018-10-28 02:58:07 UTC

One possible solution to tackle this problem is to call this issue as known_issue. Ask users to enable the options on the volume for performance improvements.

The missing perf volume options are:
cluster.choose-local=off
client.event-threads=4
server.event-threads=4
performance.client-io-threads=on

They can achieve it using gdeploy too:
[hosts]
host1

[volume1]
action=set
volname=engine
key=cluster.choose-local,client.event-threads,server.event-threads,performance.client-io-threads
value=off,4,4,on

[volume2]
action=set
volname=vmstore
key=cluster.choose-local,client.event-threads,server.event-threads,performance.client-io-threads
value=off,4,4,on

[volume3]
action=set
volname=data
key=cluster.choose-local,client.event-threads,server.event-threads,performance.client-io-threads
value=off,4,4,on

Comment 5 Sahina Bose 2018-10-29 13:03:55 UTC

Sas,
This will need to be documented for existing volumes too - to manually set these options?

Laura, do you think this should be added as a note in the guide as well?

Comment 9 SATHEESARAN 2019-01-14 03:07:32 UTC

This issue is fixed with imgbased-1.1.2 as I could see that from the dependent bug,
and moving this bug ON_QA. @Sahina, could you provide the devel_ack for this bug ?

Comment 10 SATHEESARAN 2019-01-21 11:15:38 UTC

Verified after updating to RHV 4.2.8 RHVH, and the new virt profile contains all the required volume options