Description of problem: CNS upgrade failing, After deploy --latest the new pods are going into CrashLoopBackOff. When we try to upgrade heketi2.0.6-1.el7 to heketi3.1, At this point only image is upgraded and packages are not yet upgraded. Verified the upgrade from openshift 3.3 to 3.4 everything looks to be working. # oc get pods NAME READY STATUS RESTARTS AGE aplo-router-3-lz1yb 1/1 Running 0 12m glusterfs-dc-dhcp46-121.lab.eng.blr.redhat.com-1-zqmyl 1/1 Running 0 34m glusterfs-dc-dhcp46-130.lab.eng.blr.redhat.com-1-h7rrs 1/1 Running 1 39m glusterfs-dc-dhcp46-92.lab.eng.blr.redhat.com-1-je0d0 1/1 Running 0 37m heketi-1-d1yjh 1/1 Running 7 37m heketi-2-8ws0m 0/1 CrashLoopBackOff 5 6m heketi-2-deploy 1/1 Running 0 8m mongodb-1-mlwl4 1/1 Running 7 34m Describe Heketi pod: http://pastebin.test.redhat.com/443312 Describe Gluster pod: http://pastebin.test.redhat.com/443300 Describe Gluster DC: http://pastebin.test.redhat.com/443306
Hi, I had a look into Apeksha's machine and saw the old pods are not getting deleted. I manually deleted the old pod and everything looks to be working fine now. After checking the dc mentioned above in our templates have Rolling strategy for update. Looks like we don't support Recreate in heketi-2.0. We already have the fix for this issue[1]. Possible solutions: We can provide a work around for the customers, because even the new build will not have the templates which have dc support. Workaround is to edit the dc to recreate strategy in glusterfs and heketi. This has to be done before upgrade. [1] https://bugzilla.redhat.com/show_bug.cgi?id=1357686 We should also support upgrade from deploymentConfig to daemonSet. -- Ashiq
(In reply to Mohamed Ashiq from comment #1) > Possible solutions: > We can provide a work around for the customers, because even the new build > will not have the templates which have dc support. > > Workaround is to edit the dc to recreate strategy in glusterfs and heketi. > This has to be done before upgrade. Ashiq, is that the *ONLY* work-around required considering the fact that we have moved away from deploymentConfig from the last release to daemonSets in this release?
(In reply to Prasanth from comment #2) > (In reply to Mohamed Ashiq from comment #1) > > > Possible solutions: > > We can provide a work around for the customers, because even the new build > > will not have the templates which have dc support. > > > > Workaround is to edit the dc to recreate strategy in glusterfs and heketi. > > This has to be done before upgrade. > > Ashiq, is that the *ONLY* work-around required considering the fact that we > have moved away from deploymentConfig from the last release to daemonSets in > this release? As discussed the 'dc to ds' change dont have any role here. This bug can actually be closed as it was reported due to suer error. I am closing this bug. Please feel free to reopen if needed.