Bug 1626558

Summary:	Tuned profiles are not applied to OpenShift clusters that get upgraded from 3.6
Product:	OpenShift Container Platform	Reporter:	Ryan Howe <rhowe>
Component:	Cluster Version Operator	Assignee:	Russell Teague <rteague>
Status:	CLOSED ERRATA	QA Contact:	liujia <jiajliu>
Severity:	medium	Docs Contact:
Priority:	high
Version:	3.9.0	CC:	aos-bugs, fgrosjea, gpei, jialiu, jokerman, mgugino, mmccomas, sdodson, wmeng
Target Milestone:	---
Target Release:	3.9.z
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:	During an upgrade from 3.7 to 3.9 tuned-profiles-atomic-openshift-node rpm is removed as well as all tuned profiles. The tuned role was not being applied during an upgrade but only during a fresh install. The tuned role is applied during upgrades to ensure tuned profiles are applied appropriately.	Story Points:	---
Clone Of:
Clones:	1694128 1694130 1694131 (view as bug list)		Environment:
Last Closed:	2019-07-05 06:58:57 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	1694128, 1694130, 1694131

Description Ryan Howe 2018-09-07 15:35:13 UTC

Description of problem:

When a cluster gets installed with 3.6 then upgrades 3.7 -> 3.8 -> 3.9+ 

The tuned profile rpm is no login available in 3.9+ and gets removed during the upgrade. The upgrade never runs role: tuned. The only plays that call this role are as follows. 

./playbooks/openshift-loadbalancer/private/config.yml:  - role: tuned
./playbooks/openshift-master/private/config.yml:  - role: tuned
./playbooks/openshift-node/private/configure_nodes.yml:  - role: tuned

In 3.7+ fresh installs the tuned role will be run.


Version-Release number of the following components:
Upgrade playbooks 3.7+

How reproducible:
100%

Steps to Reproduce:
1. Install 3.6 cluster and upgrade to at 3.9
2. Check output of `tuned-adm --list


Actual results:
#  tuned-adm active
Current active profile: virtual-guest

Expected results:
# tuned-adm active
Current active profile: openshift-control-plane 

Additional info:

The upgrade playbooks should always call the tuned role to ensure are set and up to date with the latest.

Comment 1 Scott Dodson 2018-09-10 12:07:33 UTC

Need to make sure that tuned role is called during upgrade plays.

Comment 4 Franck Grosjean 2018-09-28 13:53:55 UTC

Hi,

Possible fix and pr on openshift 3.9. Since there is no direct upgrade 3.10 this should fix the upgrade process.
https://github.com/openshift/openshift-ansible/pull/10266

Franck

Comment 5 Scott Dodson 2018-10-29 20:52:52 UTC

In openshift-ansible-3.9.48-1 and later, ON_QA

Comment 6 liujia 2018-11-07 09:26:56 UTC

Try to reproduce it on 
openshift-ansible-3.9.43-1.git.0.d0bc600.el7.noarch
openshift-ansible-3.7.71-1.git.0.30869d0.el7.noarch
=============================================================================
scenario 1: v3.6-v3.7-v3.9 upgrade vs fresh install of v3.7/v3.9

----> v3.6
[root@ip-172-18-10-199 ~]# rpm -qa|grep tuned
tuned-2.10.0-6.el7.noarch
tuned-profiles-atomic-openshift-node-3.6.173.0.136-1.git.0.db4569c.el7.x86_64
[root@ip-172-18-10-199 ~]# tuned-adm active
Current active profile: atomic-openshift-node-guest

----> v3.6-v3.7 upgrade
[root@ip-172-18-10-199 ~]# rpm -qa|grep tuned
tuned-2.10.0-6.el7.noarch
tuned-profiles-atomic-openshift-node-3.7.71-1.git.0.9dfa536.el7.x86_64
[root@ip-172-18-10-199 ~]# tuned-adm active
Current active profile: atomic-openshift-node-guest

----> fresh v3.7
[root@ip-172-18-15-48 ~]# rpm -qa|grep tuned
tuned-2.10.0-6.el7.noarch
tuned-profiles-atomic-openshift-node-3.7.71-1.git.0.9dfa536.el7.x86_64
[root@ip-172-18-15-48 ~]# tuned-adm active 
Current active profile: openshift-control-plane

----> v3.6-v3.7-v3.9 upgrade
[root@ip-172-18-10-199 ~]# rpm -qa|grep tuned
tuned-2.10.0-6.el7.noarch
[root@ip-172-18-10-199 ~]# tuned-adm active
Current active profile: virtual-guest

----> fresh v3.9
[root@ip-172-18-0-115 ~]# rpm -qa|grep tuned
tuned-2.10.0-6.el7.noarch
[root@ip-172-18-0-115 ~]# tuned-adm active
Current active profile: openshift-control-plane

Result 1: 
An upgraded v3.7 does not work the same with fresh installed v3.7. An upgraded v3.9 from v3.6 does not work the same with fresh installed v3.9.

===============================================================================
scenario 2: v3.7-v3.9 upgrade vs fresh install of v3.9

----> fresh install v3.7
[root@ip-172-18-14-212 ~]# rpm -qa|grep tuned
tuned-2.10.0-6.el7.noarch
tuned-profiles-atomic-openshift-node-3.7.71-1.git.0.9dfa536.el7.x86_64
[root@ip-172-18-14-212 ~]# tuned-adm active 
Current active profile: openshift-control-plane

----> v3.7-v3.9 upgrade
[root@ip-172-18-14-212 ~]# rpm -qa|grep tuned
tuned-2.10.0-6.el7.noarch
[root@ip-172-18-14-212 ~]# tuned-adm active
Current active profile: openshift-control-plane

----> fresh install v3.9
[root@ip-172-18-0-115 ~]# rpm -qa|grep tuned
tuned-2.10.0-6.el7.noarch
[root@ip-172-18-0-115 ~]# tuned-adm active
Current active profile: openshift-control-plane

Result 2: An upgraded v3.9 works the same with fresh installed v3.9. Both of them have not rpm tuned-profiles installed and openshift-control-plane profile active.
==============================================================================

Combined result 1 and result 2, this issue included two issues:
1) "openshift-control-plane" profile missed. This is because "An upgraded v3.7 does not work the same with fresh installed v3.7", which miss "openshift-control-plane" during upgrade and result that later upgrade to v3.9 miss "openshift-control-plane". 
This issue should be fixed in v3.7 openshift-ansible. And should be the major issue in the bug, so change target release version to v3.7.z and assign back.

2) the removed rpm tuned profile pkg. I've no idea the reason why it was removed from v3.9 installer. But according to result 2, the upgrade bahaviro is consistent with fresh install. so if this rpm tuned-profile pkg needed in v3.9, please file a new bug for both of install and upgrade of v3.9 to add rpm tuned pkg back.

Comment 7 Michael Gugino 2018-11-29 18:39:08 UTC

We should fix and backport.

Comment 8 Russell Teague 2019-03-29 15:15:02 UTC

PR should address 3.9 upgrades, https://github.com/openshift/openshift-ansible/pull/10266
tuned-profile pkg not needed in v3.9+

Comment 9 liujia 2019-04-17 07:25:24 UTC

For this bug(targeted for 3.9.z), so we focus on scenario 2: v3.7-v3.9 upgrade vs fresh install of v3.9. Other scenarios will be tracked in cloned bug.

Version:
openshift-ansible-3.9.77-1.git.0.b71d05a.el7.noarch.rpm

----> fresh install v3.7
[root@ip-172-18-9-33 ~]# rpm -qa|grep tuned
tuned-2.10.0-6.el7_6.3.noarch
tuned-profiles-atomic-openshift-node-3.7.111-1.git.0.63595b8.el7.x86_64
[root@ip-172-18-9-33 ~]# tuned-adm active
Current active profile: openshift-control-plane

----> v3.7-v3.9 upgrade
[root@ip-172-18-9-33 ~]# rpm -qa|grep tuned
tuned-2.10.0-6.el7_6.3.noarch
[root@ip-172-18-9-33 ~]# tuned-adm active
Current active profile: openshift-control-plane

----> fresh install v3.9
[root@ip-172-18-13-27 ~]# rpm -qa|grep tuned
tuned-2.10.0-6.el7_6.3.noarch
[root@ip-172-18-13-27 ~]# tuned-adm active
Current active profile: openshift-control-plane

During upgrade, the tuned role was run.

Comment 11 errata-xmlrpc 2019-07-05 06:58:57 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:1642