Bug 1931601

Summary:	OSD pods may be unnecessarily restarted during every reconcile since Azure tuning was backported
Product:	[Red Hat Storage] Red Hat OpenShift Container Storage	Reporter:	Travis Nielsen <tnielsen>
Component:	rook	Assignee:	Travis Nielsen <tnielsen>
Status:	CLOSED ERRATA	QA Contact:	Elad <ebenahar>
Severity:	urgent	Docs Contact:
Priority:	unspecified
Version:	4.6	CC:	madam, muagarwa, ocs-bugs, ratamir, rcyriac
Target Milestone:	---	Keywords:	AutomationTriaged, ZStream
Target Release:	OCS 4.6.3
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	No Doc Update
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2021-03-03 22:53:23 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Travis Nielsen 2021-02-22 18:46:02 UTC

Description of problem (please be detailed as possible and provide log
snippests):

Since the Azure tuning PR was backported with https://bugzilla.redhat.com/show_bug.cgi?id=1925004, this introduced the regression that OSDs will frequently be restarted during the Rook operator reconcile for no reason.

A change had been made some time ago in the 4.7 codebase to use a map of strings to apply the tuning arguments. The map causes the arguments to be in arbitrary ordering, which means the pod spec will be changed and cause a pod update during later reconciles. The fix was made in 4.7, but was not backported to 4.6 since at the time the Azure tuning had not been included in 4.6.

The upstream fix, which is in 4.7 also needs to be backported to 4.6.
https://github.com/rook/rook/pull/7071

Version of all relevant components (if applicable):

This only affects the RC for 4.6.3 since the Azure tuning was backported.

Does this issue impact your ability to continue to work with the product
(please explain in detail what is the user impact)?

No, but OSDs will be restarted frequently as if there is an upgrade when there is in fact not.

Is there any workaround available to the best of your knowledge?

No, reconciles cannot be avoided, although they might not occur more than daily if we're lucky.

Rate from 1 - 5 the complexity of the scenario you performed that caused this
bug (1 - very simple, 5 - very complex)?

Can this issue reproducible?

Yes, just restart the operator in a 4.6.3 RC build and notice that the OSDs will likely restart.

Can this issue reproduce from the UI?

If this is a regression, please provide more details to justify this:

This is a regression related to the Azure tuning PR backport.

Steps to Reproduce:
1. Install 4.6.3
2. Wait for a later Rook operator reconcile (can be triggered by restarting the operator)

Actual results:

The OSDs likely will restart unnecessarily

Expected results:

OSDs should not restart unless there is actually an upgrade.

Comment 7 Elad 2021-03-02 10:51:55 UTC

With OCS 4.6.3 RC6 - v4.6.3-271.ci

Existing OSD pods are not restarted post add capacity

http://magna002.ceph.redhat.com/ocsci-jenkins/openshift-clusters/j009ai3c333-t1/j009ai3c333-t1_20210301T144256/logs/testcases_1614613687/ocs_must_gather/quay-io-rhceph-dev-ocs-must-gather-sha256-c5e0faa3e0de046a1c9863b47287ac848b6d316ca492cc176964fdefdf25d9fb/namespaces/openshift-storage/oc_output/pods_-owide


rook-ceph-mon-c-b7fddb659-l49b5                                   1/1     Running     0          9h     10.130.2.8     ip-10-0-198-143.us-east-2.compute.internal   <none>           <none>
rook-ceph-operator-7bb7854c87-8gkgs                               1/1     Running     0          9h     10.131.2.4     ip-10-0-139-228.us-east-2.compute.internal   <none>           <none>
rook-ceph-osd-0-79c44c64d6-bt6gz                                  1/1     Running     0          9h     10.131.2.11    ip-10-0-139-228.us-east-2.compute.internal   <none>           <none>
rook-ceph-osd-1-c9bb9cbfb-pl29x                                   1/1     Running     0          9h     10.128.4.9     ip-10-0-179-133.us-east-2.compute.internal   <none>           <none>
rook-ceph-osd-2-8599688df8-npftw                                  1/1     Running     0          9h     10.130.2.11    ip-10-0-198-143.us-east-2.compute.internal   <none>           <none>
rook-ceph-osd-3-6c85c7447d-qmrzd                                  1/1     Running     0          88m    10.130.4.5     ip-10-0-188-79.us-east-2.compute.internal    <none>           <none>
rook-ceph-osd-4-688bf97b77-q4k25                                  1/1     Running     0          88m    10.129.4.7     ip-10-0-128-35.us-east-2.compute.internal    <none>           <none>
rook-ceph-osd-5-5f648d6d9-6629c                                   1/1     Running     0          87m    10.131.4.5     ip-10-0-218-44.us-east-2.compute.internal    <none>           <none>



Comparing this to 4.6.3 RC5 (v4.6.3-267.ci), in which this BZ was not fixed:
https://storage-jenkins-csb-ceph.cloud.paas.psi.redhat.com/job/ocs-ci/251/artifact/logs/failed_testcase_ocs_logs_1613665519/test_add_capacity_ocs_logs/ocs_must_gather/quay-io-rhceph-dev-ocs-must-gather-sha256-46e5db11fa62c98c9db85dbdc7a12b8894ab2301cc1cd6533e3b738df934b2fc/namespaces/openshift-storage/oc_output/

rook-ceph-operator-88555d7c-qvzk9                                 1/1     Running     0          95m     10.129.2.8     ip-10-0-176-155.us-west-1.compute.internal   <none>           <none>
rook-ceph-osd-0-7586ffdc88-9pwl5                                  1/1     Running     0          5m11s   10.129.2.86    ip-10-0-176-155.us-west-1.compute.internal   <none>           <none>
rook-ceph-osd-1-54658b8586-xx89j                                  1/1     Running     0          3m48s   10.131.0.44    ip-10-0-198-177.us-west-1.compute.internal   <none>           <none>
rook-ceph-osd-2-546f567b56-nfj2q                                  1/1     Running     0          2m21s   10.128.2.42    ip-10-0-150-160.us-west-1.compute.internal   <none>           <none>
rook-ceph-osd-3-78d64b7fc5-646qm                                  1/1     Running     0          46s     10.128.2.43    ip-10-0-150-160.us-west-1.compute.internal   <none>           <none>
rook-ceph-osd-4-76c44db7c5-ss92v                                  1/1     Running     0          47s     10.129.2.89    ip-10-0-176-155.us-west-1.compute.internal   <none>           <none>
rook-ceph-osd-5-84776fff5d-78bgq                                  1/1     Running     0          47s     10.131.0.45    ip-10-0-198-177.us-west-1.compute.internal   <none>           <none>

Comment 11 errata-xmlrpc 2021-03-03 22:53:23 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Red Hat OpenShift Container Storage 4.6.3 container bug fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2021:0718