Bug 1577951
| Summary: | Stack overcloud creation failed - libvirt qemu terminated (CPU feature arat not found) | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| Product: | Red Hat OpenStack | Reporter: | Noam Manos <nmanos> | ||||||
| Component: | openstack-tripleo | Assignee: | James Slagle <jslagle> | ||||||
| Status: | CLOSED DUPLICATE | QA Contact: | Arik Chernetsky <achernet> | ||||||
| Severity: | high | Docs Contact: | |||||||
| Priority: | unspecified | ||||||||
| Version: | 13.0 (Queens) | CC: | agurenko, kchamart, mburns, oblaut, tfreger | ||||||
| Target Milestone: | --- | Keywords: | Triaged | ||||||
| Target Release: | --- | ||||||||
| Hardware: | x86_64 | ||||||||
| OS: | Linux | ||||||||
| Whiteboard: | |||||||||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |||||||
| Doc Text: | Story Points: | --- | |||||||
| Clone Of: | Environment: | ||||||||
| Last Closed: | 2018-05-25 12:11:33 UTC | Type: | Bug | ||||||
| Regression: | --- | Mount Type: | --- | ||||||
| Documentation: | --- | CRM: | |||||||
| Verified Versions: | Category: | --- | |||||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||||
| Embargoed: | |||||||||
| Attachments: |
|
||||||||
Created attachment 1436199 [details]
Overcloud and virt logs
Created attachment 1436701 [details] sos report from undercloud This sosreport is from another installation attempt. The jenkins job is: https://rhos-qe-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/OSPD-Customized-Deployment-virt/4222/consoleFull Another deployment attempt failed on same issue "CPU feature arat not found". This time it was on a new OS provisioning of RHEL 7.4: [root@puma09 ~]# cat /var/log/libvirt/qemu/compute-0.log 2018-05-15 16:26:19.753+0000: starting up libvirt version: 3.2.0, package: 14.el7_4.9 (Red Hat, Inc. <http://bugzilla.redhat.com/bugzilla>, 2018-01-19-16:10:49, x86-040.build.eng.bos.redhat.com), qemu version: 1.5.3 (qemu-kvm-1.5.3-141.el7_4.5), hostname: puma09.scl.lab.tlv.redhat.com LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin QEMU_AUDIO_DRV=none /usr/libexec/qemu-kvm -name compute-0 -S -machine pc-i440fx-rhel7.0.0,accel=kvm,usb=off,dump-guest-core=off -cpu Westmere,+vme,+ds,+acpi,+ss,+ht,+tm,+pbe,+pclmuldq,+dtes64,+monitor,+ds_cpl,+vmx,+smx,+est,+tm2,+xtpr,+pdcm,+pcid,+dca,+arat,+pdpe1gb,+rdtscp -m 8192 -realtime mlock=off -smp 6,sockets=6,cores=1,threads=1 -uuid db127dff-77db-49c9-a12d-88f915e81bad -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-2-compute-0/monitor.sock,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc,driftfix=slew -global kvm-pit.lost_tick_policy=delay -no-hpet -no-shutdown -global PIIX4_PM.disable_s3=1 -global PIIX4_PM.disable_s4=1 -boot strict=on -device ich9-usb-ehci1,id=usb,bus=pci.0,addr=0x6.0x7 -device ich9-usb-uhci1,masterbus=usb.0,firstport=0,bus=pci.0,multifunction=on,addr=0x6 -device ich9-usb-uhci2,masterbus=usb.0,firstport=2,bus=pci.0,addr=0x6.0x1 -device ich9-usb-uhci3,masterbus=usb.0,firstport=4,bus=pci.0,addr=0x6.0x2 -device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x7 -drive file=/var/lib/libvirt/images/compute-0-disk1.qcow2,format=qcow2,if=none,id=drive-virtio-disk0,cache=unsafe -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x8,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -netdev tap,fd=26,id=hostnet0,vhost=on,vhostfd=28 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:61:1b:38,bus=pci.0,addr=0x3 -netdev tap,fd=29,id=hostnet1,vhost=on,vhostfd=30 -device virtio-net-pci,netdev=hostnet1,id=net1,mac=52:54:00:ff:b0:b2,bus=pci.0,addr=0x4 -netdev tap,fd=31,id=hostnet2,vhost=on,vhostfd=32 -device virtio-net-pci,netdev=hostnet2,id=net2,mac=52:54:00:00:ff:93,bus=pci.0,addr=0x5 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -chardev socket,id=charchannel0,path=/var/lib/libvirt/qemu/channel/target/domain-2-compute-0/org.qemu.guest_agent.0,server,nowait -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=org.qemu.guest_agent.0 -device usb-tablet,id=input0,bus=usb.0,port=1 -vnc 127.0.0.1:1 -vga cirrus -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x9 -object rng-random,id=objrng0,filename=/dev/urandom -device virtio-rng-pci,rng=objrng0,id=rng0,bus=pci.0,addr=0xa -msg timestamp=on char device redirected to /dev/pts/2 (label charserial0) CPU feature arat not found CPU feature arat not found CPU feature arat not found CPU feature arat not found CPU feature arat not found CPU feature arat not found qemu: terminating on signal 15 from pid 25814 2018-05-15 17:16:27.470+0000: shutting down, reason=destroyed [root@puma09 ~]# cat /var/log/libvirt/qemu/controller-0.log 2018-05-15 16:26:24.310+0000: starting up libvirt version: 3.2.0, package: 14.el7_4.9 (Red Hat, Inc. <http://bugzilla.redhat.com/bugzilla>, 2018-01-19-16:10:49, x86-040.build.eng.bos.redhat.com), qemu version: 1.5.3 (qemu-kvm-1.5.3-141.el7_4.5), hostname: puma09.scl.lab.tlv.redhat.com LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin QEMU_AUDIO_DRV=none /usr/libexec/qemu-kvm -name controller-0 -S -machine pc-i440fx-rhel7.0.0,accel=kvm,usb=off,dump-guest-core=off -cpu Westmere,+vme,+ds,+acpi,+ss,+ht,+tm,+pbe,+pclmuldq,+dtes64,+monitor,+ds_cpl,+vmx,+smx,+est,+tm2,+xtpr,+pdcm,+pcid,+dca,+arat,+pdpe1gb,+rdtscp -m 32768 -realtime mlock=off -smp 8,sockets=8,cores=1,threads=1 -uuid 47df02b5-a969-4c0b-9a08-6351336032d9 -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-4-controller-0/monitor.sock,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc,driftfix=slew -global kvm-pit.lost_tick_policy=delay -no-hpet -no-shutdown -global PIIX4_PM.disable_s3=1 -global PIIX4_PM.disable_s4=1 -boot strict=on -device ich9-usb-ehci1,id=usb,bus=pci.0,addr=0x6.0x7 -device ich9-usb-uhci1,masterbus=usb.0,firstport=0,bus=pci.0,multifunction=on,addr=0x6 -device ich9-usb-uhci2,masterbus=usb.0,firstport=2,bus=pci.0,addr=0x6.0x1 -device ich9-usb-uhci3,masterbus=usb.0,firstport=4,bus=pci.0,addr=0x6.0x2 -device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x7 -drive file=/var/lib/libvirt/images/controller-0-disk1.qcow2,format=qcow2,if=none,id=drive-virtio-disk0,cache=unsafe -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x8,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -netdev tap,fd=28,id=hostnet0,vhost=on,vhostfd=30 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:20:83:7e,bus=pci.0,addr=0x3 -netdev tap,fd=31,id=hostnet1,vhost=on,vhostfd=32 -device virtio-net-pci,netdev=hostnet1,id=net1,mac=52:54:00:77:10:83,bus=pci.0,addr=0x4 -netdev tap,fd=33,id=hostnet2,vhost=on,vhostfd=34 -device virtio-net-pci,netdev=hostnet2,id=net2,mac=52:54:00:8d:88:39,bus=pci.0,addr=0x5 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -chardev socket,id=charchannel0,path=/var/lib/libvirt/qemu/channel/target/domain-4-controller-0/org.qemu.guest_agent.0,server,nowait -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=org.qemu.guest_agent.0 -device usb-tablet,id=input0,bus=usb.0,port=1 -vnc 127.0.0.1:3 -vga cirrus -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x9 -object rng-random,id=objrng0,filename=/dev/urandom -device virtio-rng-pci,rng=objrng0,id=rng0,bus=pci.0,addr=0xa -msg timestamp=on char device redirected to /dev/pts/4 (label charserial0) CPU feature arat not found CPU feature arat not found CPU feature arat not found CPU feature arat not found CPU feature arat not found CPU feature arat not found CPU feature arat not found CPU feature arat not found qemu: terminating on signal 15 from pid 25814 2018-05-15 17:16:28.331+0000: shutting down, reason=destroyed [root@puma09 ~]# cat /var/log/libvirt/qemu/undercloud-0.log 2018-05-15 16:26:16.660+0000: starting up libvirt version: 3.2.0, package: 14.el7_4.9 (Red Hat, Inc. <http://bugzilla.redhat.com/bugzilla>, 2018-01-19-16:10:49, x86-040.build.eng.bos.redhat.com), qemu version: 1.5.3 (qemu-kvm-1.5.3-141.el7_4.5), hostname: puma09.scl.lab.tlv.redhat.com LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin QEMU_AUDIO_DRV=none /usr/libexec/qemu-kvm -name undercloud-0 -S -machine pc-i440fx-rhel7.0.0,accel=kvm,usb=off,dump-guest-core=off -cpu Westmere,+vme,+ds,+acpi,+ss,+ht,+tm,+pbe,+pclmuldq,+dtes64,+monitor,+ds_cpl,+vmx,+smx,+est,+tm2,+xtpr,+pdcm,+pcid,+dca,+arat,+pdpe1gb,+rdtscp -m 16384 -realtime mlock=off -smp 2,sockets=2,cores=1,threads=1 -uuid fd45b4e2-ce69-4130-89f6-8fff8b26c5da -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-1-undercloud-0/monitor.sock,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc,driftfix=slew -global kvm-pit.lost_tick_policy=delay -no-hpet -no-shutdown -global PIIX4_PM.disable_s3=1 -global PIIX4_PM.disable_s4=1 -boot strict=on -device ich9-usb-ehci1,id=usb,bus=pci.0,addr=0x6.0x7 -device ich9-usb-uhci1,masterbus=usb.0,firstport=0,bus=pci.0,multifunction=on,addr=0x6 -device ich9-usb-uhci2,masterbus=usb.0,firstport=2,bus=pci.0,addr=0x6.0x1 -device ich9-usb-uhci3,masterbus=usb.0,firstport=4,bus=pci.0,addr=0x6.0x2 -device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x7 -drive file=/var/lib/libvirt/images/undercloud-0-disk1.qcow2,format=qcow2,if=none,id=drive-virtio-disk0,cache=unsafe -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x8,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -netdev tap,fd=25,id=hostnet0,vhost=on,vhostfd=27 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:49:ce:c7,bus=pci.0,addr=0x3 -netdev tap,fd=28,id=hostnet1,vhost=on,vhostfd=29 -device virtio-net-pci,netdev=hostnet1,id=net1,mac=52:54:00:25:83:f0,bus=pci.0,addr=0x4 -netdev tap,fd=30,id=hostnet2,vhost=on,vhostfd=31 -device virtio-net-pci,netdev=hostnet2,id=net2,mac=52:54:00:54:00:37,bus=pci.0,addr=0x5 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -chardev socket,id=charchannel0,path=/var/lib/libvirt/qemu/channel/target/domain-1-undercloud-0/org.qemu.guest_agent.0,server,nowait -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=org.qemu.guest_agent.0 -device usb-tablet,id=input0,bus=usb.0,port=1 -vnc 127.0.0.1:0 -vga cirrus -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x9 -object rng-random,id=objrng0,filename=/dev/urandom -device virtio-rng-pci,rng=objrng0,id=rng0,bus=pci.0,addr=0xa -msg timestamp=on char device redirected to /dev/pts/1 (label charserial0) CPU feature arat not found CPU feature arat not found [root@puma09 ~]# Notice the libvirt version difference between RHEL 7.4 and 7.5: On RHEL 7.4 - libvirt version: 3.2.0, package: 14.el7_4.9 On RHEL 7.5 - libvirt version: 3.9.0, package: 14.el7_5.4 This seems like a duplicate of:
https://bugzilla.redhat.com/show_bug.cgi?id=1576464#c4 --
Hash operation not allowed during iteration
Which is fixed in libvirt-4.3.0-1.el7.
And a build for RHEL 7.5.z is waiting to be done here here:
https://bugzilla.redhat.com/show_bug.cgi?id=1581364 --
Hash operation not allowed during iteration [rhel-7.5.z]
*** This bug has been marked as a duplicate of bug 1581364 *** |
Description of problem: Attempting to install OSP13 on Rhel 7.5 - Openstack overcloud deployment failed: CPU feature arat not found qemu: terminating on signal 15 Version-Release number of selected component (if applicable): [root@puma09 ~]# lsb_release -i -r Distributor ID: RedHatEnterpriseServer Release: 7.5 [root@puma09 ~]# uname -r 3.10.0-862.2.3.el7.x86_64 [root@puma09 ~]# cat /etc/rhosp-release (undercloud) [stack@undercloud-0 ~]$ cat /etc/rhosp-release Red Hat OpenStack Platform release 13.0 (Queens) (undercloud) [stack@undercloud-0 ~]$ lsb_release -i -r Distributor ID: RedHatEnterpriseServer Release: 7.5 (undercloud) [stack@undercloud-0 ~]$ uname -r 3.10.0-862.2.3.el7.x86_64 How reproducible: Often (not always). Steps to Reproduce: Install OSPD 13 with InfraRed on RHEL 7.5 Actual results: *** ir-tripleo-overcloud-deploy.log: PLAY [Verify overcloud deployment] ********************************************* META: ran handlers TASK [fail] ******************************************************************** task path: /home/rhos-ci/jenkins/workspace/OSPD-Customized-Deployment-virt/infrared/plugins/tripleo-overcloud/deploy.yml:138 fatal: [undercloud-0]: FAILED! => { "changed": false } MSG: Overcloud deployment failed... :( NO MORE HOSTS LEFT ************************************************************* to retry, use: --limit @/home/rhos-ci/jenkins/workspace/OSPD-Customized-Deployment-virt/infrared/plugins/tripleo-overcloud/main.retry PLAY RECAP ********************************************************************* hypervisor : ok=2 changed=0 unreachable=0 failed=0 localhost : ok=4 changed=1 unreachable=0 failed=0 undercloud-0 : ok=129 changed=56 unreachable=0 failed=1 ERROR Playbook "/home/rhos-ci/jenkins/workspace/OSPD-Customized-Deployment-virt/infrared/plugins/tripleo-overcloud/main.yml" failed! --- *** [stack@undercloud-0 ~]$ cat overcloud_install.log 2018-05-14 09:44:25Z Heat Stack create failed. Heat Stack create failed. [overcloud.Compute.0.NodeExtraConfig]: CREATE_COMPLETE state changed 2018-05-14 09:44:41Z [overcloud.Compute.0.SshHostPubKey]: CREATE_COMPLETE state changed 2018-05-14 09:44:41Z [overcloud.Compute.0]: CREATE_COMPLETE Stack CREATE completed successfully 2018-05-14 09:44:42Z [overcloud.Compute.0]: CREATE_COMPLETE state changed 2018-05-14 10:58:47Z [overcloud.Compute]: CREATE_FAILED CREATE aborted (Task create from ResourceGroup "Compute" Stack "overcloud" [86d0dce5-fc9e-4bc2-9140-8b943deb13fc] Timed out) 2018-05-14 10:58:47Z [overcloud.Compute]: UPDATE_FAILED Stack UPDATE cancelled 2018-05-14 10:58:47Z [overcloud]: CREATE_FAILED Timed out 2018-05-14 10:58:47Z [overcloud.Compute.1]: CREATE_FAILED Stack CREATE cancelled 2018-05-14 10:58:48Z [overcloud.Compute.1]: CREATE_FAILED resources[1]: Stack CREATE cancelled 2018-05-14 10:58:48Z [overcloud.Compute]: UPDATE_FAILED Resource CREATE failed: resources[1]: Stack CREATE cancelled Stack overcloud CREATE_FAILED overcloud.Compute.1: resource_type: OS::TripleO::Compute physical_resource_id: 0790bca4-a029-42ed-8b22-eec494ea15d6 status: CREATE_FAILED status_reason: | resources[1]: Stack CREATE cancelled ---- *** [root@puma09 ~]# cat /var/log/libvirt/qemu/compute-1.log 2018-05-13 20:20:28.985+0000: starting up libvirt version: 3.9.0, package: 14.el7_5.4 (Red Hat, Inc. <http://bugzilla.redhat.com/bugzilla>, 2018-04-13-20:17:38, x86-040.build.eng.bos.redhat.com), qemu version: 1.5.3 (qemu-kvm-1.5.3-156.el7), hostname: puma09.scl.lab.tlv.redhat.com LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin QEMU_AUDIO_DRV=none /usr/libexec/qemu-kvm -name compute-1 -S -machine pc-i440fx-rhel7.0.0,accel=kvm,usb=off,dump-guest-core=off -cpu Westmere,+vme,+ds,+acpi,+ss,+ht,+tm,+pbe,+pclmuldq,+dtes64,+monitor,+ds_cpl,+vmx,+smx,+est,+tm2,+xtpr,+pdcm,+pcid,+dca,+arat,+pdpe1gb,+rdtscp -m 8192 -realtime mlock=off -smp 6,sockets=6,cores=1,threads=1 -uuid a6d16bf0-c7b7-498c-8434-390a2fc1ca46 -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-3-compute-1/monitor.sock,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc,driftfix=slew -global kvm-pit.lost_tick_policy=delay -no-hpet -no-shutdown -global PIIX4_PM.disable_s3=1 -global PIIX4_PM.disable_s4=1 -boot strict=on -device ich9-usb-ehci1,id=usb,bus=pci.0,addr=0x6.0x7 -device ich9-usb-uhci1,masterbus=usb.0,firstport=0,bus=pci.0,multifunction=on,addr=0x6 -device ich9-usb-uhci2,masterbus=usb.0,firstport=2,bus=pci.0,addr=0x6.0x1 -device ich9-usb-uhci3,masterbus=usb.0,firstport=4,bus=pci.0,addr=0x6.0x2 -device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x7 -drive file=/var/lib/libvirt/images/compute-1-disk1.qcow2,format=qcow2,if=none,id=drive-virtio-disk0,cache=unsafe -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x8,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -netdev tap,fd=28,id=hostnet0,vhost=on,vhostfd=30 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:4a:39:44,bus=pci.0,addr=0x3 -netdev tap,fd=31,id=hostnet1,vhost=on,vhostfd=32 -device virtio-net-pci,netdev=hostnet1,id=net1,mac=52:54:00:2f:cc:95,bus=pci.0,addr=0x4 -netdev tap,fd=33,id=hostnet2,vhost=on,vhostfd=34 -device virtio-net-pci,netdev=hostnet2,id=net2,mac=52:54:00:20:7c:09,bus=pci.0,addr=0x5 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -chardev socket,id=charchannel0,path=/var/lib/libvirt/qemu/channel/target/domain-3-compute-1/org.qemu.guest_agent.0,server,nowait -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=org.qemu.guest_agent.0 -device usb-tablet,id=input0,bus=usb.0,port=1 -vnc 127.0.0.1:2 -vga cirrus -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x9 -object rng-random,id=objrng0,filename=/dev/urandom -device virtio-rng-pci,rng=objrng0,id=rng0,bus=pci.0,addr=0xa -msg timestamp=on char device redirected to /dev/pts/3 (label charserial0) CPU feature arat not found CPU feature arat not found CPU feature arat not found CPU feature arat not found CPU feature arat not found CPU feature arat not found qemu: terminating on signal 15 from pid 24832 2018-05-13 21:08:29.294+0000: shutting down, reason=destroyed Expected results: Overcloud to be installed successfully. Additional info: - Attaching installation logs. - Might be related to Bug 1576464 - Hash operation not allowed during iteration.