Fedora Account System
Red Hat Associate
Red Hat Customer
Description of problem (please be detailed as possible and provide log snippests): Since the Azure tuning PR was backported with https://bugzilla.redhat.com/show_bug.cgi?id=1925004, this introduced the regression that OSDs will frequently be restarted during the Rook operator reconcile for no reason. A change had been made some time ago in the 4.7 codebase to use a map of strings to apply the tuning arguments. The map causes the arguments to be in arbitrary ordering, which means the pod spec will be changed and cause a pod update during later reconciles. The fix was made in 4.7, but was not backported to 4.6 since at the time the Azure tuning had not been included in 4.6. The upstream fix, which is in 4.7 also needs to be backported to 4.6. https://github.com/rook/rook/pull/7071 Version of all relevant components (if applicable): This only affects the RC for 4.6.3 since the Azure tuning was backported. Does this issue impact your ability to continue to work with the product (please explain in detail what is the user impact)? No, but OSDs will be restarted frequently as if there is an upgrade when there is in fact not. Is there any workaround available to the best of your knowledge? No, reconciles cannot be avoided, although they might not occur more than daily if we're lucky. Rate from 1 - 5 the complexity of the scenario you performed that caused this bug (1 - very simple, 5 - very complex)? 1 Can this issue reproducible? Yes, just restart the operator in a 4.6.3 RC build and notice that the OSDs will likely restart. Can this issue reproduce from the UI? NA If this is a regression, please provide more details to justify this: This is a regression related to the Azure tuning PR backport. Steps to Reproduce: 1. Install 4.6.3 2. Wait for a later Rook operator reconcile (can be triggered by restarting the operator) Actual results: The OSDs likely will restart unnecessarily Expected results: OSDs should not restart unless there is actually an upgrade.
With OCS 4.6.3 RC6 - v4.6.3-271.ci Existing OSD pods are not restarted post add capacity http://magna002.ceph.redhat.com/ocsci-jenkins/openshift-clusters/j009ai3c333-t1/j009ai3c333-t1_20210301T144256/logs/testcases_1614613687/ocs_must_gather/quay-io-rhceph-dev-ocs-must-gather-sha256-c5e0faa3e0de046a1c9863b47287ac848b6d316ca492cc176964fdefdf25d9fb/namespaces/openshift-storage/oc_output/pods_-owide rook-ceph-mon-c-b7fddb659-l49b5 1/1 Running 0 9h 10.130.2.8 ip-10-0-198-143.us-east-2.compute.internal <none> <none> rook-ceph-operator-7bb7854c87-8gkgs 1/1 Running 0 9h 10.131.2.4 ip-10-0-139-228.us-east-2.compute.internal <none> <none> rook-ceph-osd-0-79c44c64d6-bt6gz 1/1 Running 0 9h 10.131.2.11 ip-10-0-139-228.us-east-2.compute.internal <none> <none> rook-ceph-osd-1-c9bb9cbfb-pl29x 1/1 Running 0 9h 10.128.4.9 ip-10-0-179-133.us-east-2.compute.internal <none> <none> rook-ceph-osd-2-8599688df8-npftw 1/1 Running 0 9h 10.130.2.11 ip-10-0-198-143.us-east-2.compute.internal <none> <none> rook-ceph-osd-3-6c85c7447d-qmrzd 1/1 Running 0 88m 10.130.4.5 ip-10-0-188-79.us-east-2.compute.internal <none> <none> rook-ceph-osd-4-688bf97b77-q4k25 1/1 Running 0 88m 10.129.4.7 ip-10-0-128-35.us-east-2.compute.internal <none> <none> rook-ceph-osd-5-5f648d6d9-6629c 1/1 Running 0 87m 10.131.4.5 ip-10-0-218-44.us-east-2.compute.internal <none> <none> Comparing this to 4.6.3 RC5 (v4.6.3-267.ci), in which this BZ was not fixed: https://storage-jenkins-csb-ceph.cloud.paas.psi.redhat.com/job/ocs-ci/251/artifact/logs/failed_testcase_ocs_logs_1613665519/test_add_capacity_ocs_logs/ocs_must_gather/quay-io-rhceph-dev-ocs-must-gather-sha256-46e5db11fa62c98c9db85dbdc7a12b8894ab2301cc1cd6533e3b738df934b2fc/namespaces/openshift-storage/oc_output/ rook-ceph-operator-88555d7c-qvzk9 1/1 Running 0 95m 10.129.2.8 ip-10-0-176-155.us-west-1.compute.internal <none> <none> rook-ceph-osd-0-7586ffdc88-9pwl5 1/1 Running 0 5m11s 10.129.2.86 ip-10-0-176-155.us-west-1.compute.internal <none> <none> rook-ceph-osd-1-54658b8586-xx89j 1/1 Running 0 3m48s 10.131.0.44 ip-10-0-198-177.us-west-1.compute.internal <none> <none> rook-ceph-osd-2-546f567b56-nfj2q 1/1 Running 0 2m21s 10.128.2.42 ip-10-0-150-160.us-west-1.compute.internal <none> <none> rook-ceph-osd-3-78d64b7fc5-646qm 1/1 Running 0 46s 10.128.2.43 ip-10-0-150-160.us-west-1.compute.internal <none> <none> rook-ceph-osd-4-76c44db7c5-ss92v 1/1 Running 0 47s 10.129.2.89 ip-10-0-176-155.us-west-1.compute.internal <none> <none> rook-ceph-osd-5-84776fff5d-78bgq 1/1 Running 0 47s 10.131.0.45 ip-10-0-198-177.us-west-1.compute.internal <none> <none>
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Red Hat OpenShift Container Storage 4.6.3 container bug fix update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2021:0718