Created attachment 1289651 [details] screen from KVM console Description of problem: Controller node have kernel panic after minor update osp11 to osp11 latest+rhel7.4(see more in attached screen) Compute node working as expected and have fresh kernel and release 7.4 [heat-admin@compute-0 ~]$ cat /etc/redhat-release Red Hat Enterprise Linux Server release 7.4 Beta (Maipo) Version-Release number of selected component (if applicable): OSP11 How reproducible: Steps to Reproduce: 1.deploy osp11 using infrared infrared virsh -v --host-address 10.9.76.22 --host-key ~/.ssh/id_rsa --cleanup yes && infrared virsh -v --host-address 10.9.76.22 --host-key ~/.ssh/id_rsa --topology-nodes undercloud:1,controller:1,compute:1 -e override.controller.cpu=8 -e override.controller.memory=32768 -e override.undercloud.disks.disk1.size=100G && infrared tripleo-undercloud --version 11 --images-task=rpm && infrared tripleo-overcloud -v --introspect yes --tagging yes --post no --deployment-files virt --version 11 --deploy yes 2. run sudo rhos-release 11 -r 7.4 on undercloud && overcloud nodes 3.perform minor update: https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/11/html/upgrading_red_hat_openstack_platform/ #note - update could failed - https://bugzilla.redhat.com/show_bug.cgi?id=1463227, w/a recreate masquerade for br-ctrplane network Actual results: controller node unable to boot Expected results: controller node working correctly after reboot Additional info:
This bugzilla has been removed from the release and needs to be reviewed and Triaged for another Target Release.
reproduced again compute node working after minor update, controller have kernel panic [stack@undercloud-0 ~]$ ssh heat-admin.24.8 "uname -a" Linux compute-0.localdomain 3.10.0-685.el7.x86_64 #1 SMP Tue Jun 20 00:14:41 EDT 2017 x86_64 x86_64 x86_64 GNU/Linux [stack@undercloud-0 ~]$ ssh heat-admin.24.8 "yum -v repolist" Not loading "rhnplugin" plugin, as it is disabled Loading "product-id" plugin Loading "search-disabled-repos" plugin Loading "subscription-manager" plugin Not root, Subscription Management repositories not updated Config time: 0.102 Yum version: 3.4.3 Setting up Package Sacks pkgsack time: 0.005 Repo-id : rhelosp-11.0-ceph-2.0-mon/x86_64 Repo-name : Ceph 2.0 MON Repo-revision: 1497883293 Repo-updated : Mon Jun 19 14:41:33 2017 Repo-pkgs : 133 Repo-size : 729 M Repo-baseurl : http://pulp.dist.prod.ext.phx2.redhat.com/content/dist/rhel/server/7/7Server/x86_64/ceph-mon/2/os/ Repo-expire : 21,600 second(s) (last: Fri Jun 23 07:38:36 2017) Filter : read-only:present Repo-filename: /etc/yum.repos.d/rhos-release-11.repo Repo-id : rhelosp-11.0-ceph-2.0-osd/x86_64 Repo-name : Ceph 2.0 OSD Repo-revision: 1497883292 Repo-updated : Mon Jun 19 14:41:32 2017 Repo-pkgs : 115 Repo-size : 672 M Repo-baseurl : http://pulp.dist.prod.ext.phx2.redhat.com/content/dist/rhel/server/7/7Server/x86_64/ceph-osd/2/os/ Repo-expire : 21,600 second(s) (last: Fri Jun 23 07:38:36 2017) Filter : read-only:present Repo-filename: /etc/yum.repos.d/rhos-release-11.repo Repo-id : rhelosp-11.0-ceph-2.0-tools/x86_64 Repo-name : Ceph 2.0 Tools Repo-revision: 1497883292 Repo-updated : Mon Jun 19 14:41:32 2017 Repo-pkgs : 153 Repo-size : 230 M Repo-baseurl : http://pulp.dist.prod.ext.phx2.redhat.com/content/dist/rhel/server/7/7Server/x86_64/ceph-tools/2/os/ Repo-expire : 21,600 second(s) (last: Fri Jun 23 07:38:37 2017) Filter : read-only:present Repo-filename: /etc/yum.repos.d/rhos-release-11.repo Repo-id : rhelosp-11.0-devtools-puddle/x86_64 Repo-name : RHOS-11.0 Repo-revision: 1497972122 Repo-updated : Tue Jun 20 15:22:03 2017 Repo-pkgs : 4 Repo-size : 1.2 M Repo-baseurl : http://download.lab.bos.redhat.com/rcm-guest/puddles/OpenStack/11.0-RHEL-7/2017-06-20.2/RH7-RHOS-DEVTOOLS-11.0/x86_64/os Repo-expire : 21,600 second(s) (last: Fri Jun 23 07:38:37 2017) Filter : read-only:present Repo-filename: /etc/yum.repos.d/rhos-release-11.repo Repo-id : rhelosp-11.0-puddle/x86_64 Repo-name : RHOS-11.0 Repo-revision: 1497972077 Repo-updated : Tue Jun 20 15:21:56 2017 Repo-pkgs : 737 Repo-size : 2.0 G Repo-baseurl : http://download.lab.bos.redhat.com/rcm-guest/puddles/OpenStack/11.0-RHEL-7/2017-06-20.2/RH7-RHOS-11.0/x86_64/os Repo-expire : 21,600 second(s) (last: Fri Jun 23 07:38:37 2017) Filter : read-only:present Repo-filename: /etc/yum.repos.d/rhos-release-11.repo Repo-id : rhelosp-rhel-7.4-extras/x86_64 Repo-name : Red Hat Enterprise Linux 7Server - x86_64 - Extras Repo-revision: 1497881688 Repo-updated : Mon Jun 19 14:14:48 2017 Repo-pkgs : 95 Repo-size : 206 M Repo-baseurl : http://download.eng.bos.redhat.com/composes/nightly/EXTRAS-RHEL-7.4/latest-EXTRAS-7-RHEL-7/compose/Server/x86_64/os/ Repo-expire : 21,600 second(s) (last: Fri Jun 23 07:38:37 2017) Filter : read-only:present Repo-filename: /etc/yum.repos.d/rhos-release-rhel-7.4.repo Repo-id : rhelosp-rhel-7.4-ha/x86_64 Repo-name : Red Hat Enterprise Linux 7Server - x86_64 - HA Repo-revision: 1498067830 Repo-updated : Wed Jun 21 17:57:10 2017 Repo-pkgs : 35 Repo-size : 13 M Repo-baseurl : http://download-node-02.eng.bos.redhat.com/composes/nightly/latest-RHEL-7/compose/Server/x86_64/os/addons/HighAvailability/ Repo-expire : 21,600 second(s) (last: Fri Jun 23 07:38:37 2017) Filter : read-only:present Repo-filename: /etc/yum.repos.d/rhos-release-rhel-7.4.repo Repo-id : rhelosp-rhel-7.4-server/x86_64 Repo-name : Red Hat Enterprise Linux 7Server - x86_64 - Server Repo-revision: 1498067805 Repo-updated : Wed Jun 21 17:56:45 2017 Repo-pkgs : 5,142 Repo-size : 3.7 G Repo-baseurl : http://download-node-02.eng.bos.redhat.com/composes/nightly/latest-RHEL-7/compose/Server/x86_64/os/ Repo-expire : 21,600 second(s) (last: Fri Jun 23 07:38:37 2017) Filter : read-only:present Repo-filename: /etc/yum.repos.d/rhos-release-rhel-7.4.repo Repo-id : rhos-release Repo-name : RHOS Release Repo-revision: 1498149399 Repo-updated : Thu Jun 22 16:36:43 2017 Repo-pkgs : 165 Repo-size : 3.0 M Repo-baseurl : http://download-node-02.eng.bos.redhat.com/rcm-guest/puddles/OpenStack/rhos-release/ Repo-expire : 21,600 second(s) (last: Fri Jun 23 07:38:37 2017) Filter : read-only:present Repo-filename: /etc/yum.repos.d/rhos-release.repo Repo-id : rhos-release-extras/7Server Repo-name : RHOS Release Extras Repo-revision: 1443035482 Repo-updated : Wed Sep 23 19:11:23 2015 Repo-pkgs : 2 Repo-size : 655 k Repo-baseurl : http://download-node-02.eng.bos.redhat.com/rcm-guest/puddles/OpenStack/rhos-release/extras/7Server Repo-expire : 21,600 second(s) (last: Fri Jun 23 07:38:37 2017) Filter : read-only:present Repo-filename: /etc/yum.repos.d/rhos-release.repo
for initial deploy I used next virtual h/w infrared virsh -v --host-address $host --host-key ~/.ssh/id_rsa --cleanup yes && infrared virsh -v --host-address $host--host-key ~/.ssh/id_rsa --topology-nodes undercloud:1,controller:1,compute:1 -e override.controller.cpu=8 -e override.controller.memory=32768 -e override.undercloud.disks.disk1.size=100G && infrared tripleo-undercloud --version 11 --images-task=rpm && infrared tripleo-overcloud -v --introspect yes --tagging yes --post no --deployment-files virt --version 11 --deploy yes
Attempt to reproduce against local mirror
hi Ormi, tell us if you were able to reproduce this one. In all cases we would need some sos-report from controller to help debug this one. Thanks,
Probably same rootcause as https://bugzilla.redhat.com/show_bug.cgi?id=1464588 - single node pacemaker deployment. During yum_update.sh run we shut down pacemaker controlled services eg galera and rabbit on the node that is being updated. Openstack services than don't have other node (HA missing) to communicate to and thus fail to shut down when yum is updating them. Root cause is openstack services like nova and neutron not able to finish restart if amqp/mysql is unaccessible and us testing with singlenode pacemaker.
yep. agree that this bz related to https://bugzilla.redhat.com/show_bug.cgi?id=1464588
Closing this one as duplicate. If the pacemaker single node deployment has to be supported let's track this in a RFE. *** This bug has been marked as a duplicate of bug 1464588 ***