Bug 1643730 - [upgrade] Post upgrade, new options are not available in virt profile
Summary: [upgrade] Post upgrade, new options are not available in virt profile
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Red Hat Gluster Storage
Classification: Red Hat
Component: rhhi
Version: rhhiv-1.5
Hardware: x86_64
OS: Linux
high
high
Target Milestone: ---
: RHHI-V 1.5.z Async
Assignee: Sahina Bose
QA Contact: SATHEESARAN
URL:
Whiteboard:
Depends On: 1643733
Blocks: 1548985
TreeView+ depends on / blocked
 
Reported: 2018-10-28 01:13 UTC by SATHEESARAN
Modified: 2019-05-20 04:54 UTC (History)
6 users (show)

Fixed In Version: imgbased-1.1.2
Doc Type: Known Issue
Doc Text:
When a virtualization host is upgraded to Red Hat Hyperconverged infrastructure for Virtualization 1.5, the performance profile for the virt group is not updated with parameters added during development. This means that upgraded deployments are not optimally tuned for most use cases. To work around this issue, add the following options to the /var/lib/glusterd/groups/virt file on each server, or manually set these options on your upgraded volumes. cluster.choose-local=off client.event-threads=4 server.event-threads=4 performance.client-io-threads=on
Clone Of:
: 1643733 (view as bug list)
Environment:
Last Closed: 2019-05-20 04:54:49 UTC
Target Upstream Version:


Attachments (Terms of Use)

Description SATHEESARAN 2018-10-28 01:13:29 UTC
Description of problem:
-----------------------
After upgrading from RHV 4.1.11, the new virt options are not available in virt profile ( /var/lib/glusterd/groups/virt )

Version-Release number of selected component (if applicable):
--------------------------------------------------------------
RHVH 4.1.11
RHVH 4.2.7

How reproducible:
------------------
6/6

Steps to Reproduce:
-------------------
1. Install RHVH 4.1.11
2. Check for the contents of virt profile 
3. Update to the latest RHVH 4.2.7
4. Check for the contents of virt profile

Actual results:
---------------
New virt profile still points to the old content


Expected results:
------------------
New virt profile should have the new content
Additional info:

Comment 1 SATHEESARAN 2018-10-28 01:15:37 UTC
[root@ ~]# imgbase w
You are on rhvh-4.2.7.3-0.20181026.0+1

[root@ ~]# rpm -qa | grep glusterfs-server
glusterfs-server-3.12.2-25.el7rhgs.x86_64

[root@ ~]# rpm -qf /var/lib/glusterd/groups/virt 
glusterfs-server-3.12.2-25.el7rhgs.x86_64

[root@ ~]# cat /var/lib/glusterd/groups/virt
performance.quick-read=off
performance.read-ahead=off
performance.io-cache=off
performance.low-prio-threads=32
network.remote-dio=enable
cluster.eager-lock=enable
cluster.quorum-type=auto
cluster.server-quorum-type=server
cluster.data-self-heal-algorithm=full
cluster.locking-scheme=granular
cluster.shd-max-threads=8
cluster.shd-wait-qlength=10000
features.shard=on
user.cifs=off


Note that the new options are missing in this virt profile

Comment 2 SATHEESARAN 2018-10-28 01:19:00 UTC
I upgraded the gluster rpms from glusterfs-3.8.4-54.15 to 3.12.2-25
The contents of virt profile were updated.

I suspect this to be the problem with RHVH image update. 
Will just try to understand it with 2 tests

Test1
1. Install RHVH 4.1.11
2. Check for contents
3. Do image update from CLI ( # yum update )
4. Check for contents of virt profile
5. Reboot
6. Check for contents of virt profile

Test2
1. Install RHVH 4.1.11
2. Check for contents
3. Remove virt profile
3. Do image update from CLI ( # yum update )
4. Check for contents of virt profile
5. Reboot
6. Check for contents of virt profile

I will update the results for reference

Comment 3 SATHEESARAN 2018-10-28 02:53:30 UTC
This looks like the RHVH related issue. I have tested with single node upgrade as mentioned in comment2

Test1
-----
1. Created a VM and installed it with RHVH 4.2.6 ( which had glusterfs-3.8.4-54.15.el7rhgs )
2. Updated to RHVH 4.2.7 image ( redhat-virtuaization-host-image-update )
3. After update and reboot, I observed the contents of virt profile remained the same.

Test2
-------
1. Created a VM and installed it with RHVH 4.2.6 ( which had glusterfs-3.8.4-54.15.el7rhgs )
2. Removed the virt profile ( rm -rf /var/lib/glusterd/groups/virt )
2. Updated to RHVH 4.2.7 image ( redhat-virtuaization-host-image-update )
3. After updating, I observed that the virt profile is created new and the contents of virt profile had the recently added params.

Comment 4 SATHEESARAN 2018-10-28 02:58:07 UTC
One possible solution to tackle this problem is to call this issue as known_issue. Ask users to enable the options on the volume for performance improvements.

The missing perf volume options are:
cluster.choose-local=off
client.event-threads=4
server.event-threads=4
performance.client-io-threads=on

They can achieve it using gdeploy too:
[hosts]
host1

[volume1]
action=set
volname=engine
key=cluster.choose-local,client.event-threads,server.event-threads,performance.client-io-threads
value=off,4,4,on

[volume2]
action=set
volname=vmstore
key=cluster.choose-local,client.event-threads,server.event-threads,performance.client-io-threads
value=off,4,4,on

[volume3]
action=set
volname=data
key=cluster.choose-local,client.event-threads,server.event-threads,performance.client-io-threads
value=off,4,4,on

Comment 5 Sahina Bose 2018-10-29 13:03:55 UTC
Sas,
This will need to be documented for existing volumes too - to manually set these options?

Laura, do you think this should be added as a note in the guide as well?

Comment 9 SATHEESARAN 2019-01-14 03:07:32 UTC
This issue is fixed with imgbased-1.1.2 as I could see that from the dependent bug,
and moving this bug ON_QA. @Sahina, could you provide the devel_ack for this bug ?

Comment 10 SATHEESARAN 2019-01-21 11:15:38 UTC
Verified after updating to RHV 4.2.8 RHVH, and the new virt profile contains all the required volume options


Note You need to log in before you can comment on or make changes to this bug.