ovrit-engine-backned: VMs with Spice display protocol start with 'cirrus' driver which used by 'VNC' and caused BSOD on scale environment. Description: ************ From some reason when starting single VM with Spice, as display protocol, the VM starts with 'cirrus' (which used by VNC), I've encountered a problem of BSOD on guest when initiated 100 VM's at once on one host. The BSOD message indicates the following: "The problem seems to caused by the following file: cirrus" (See: attached screen shot). Although the BSOD caused by 'cirrus' and probably related to the load during VM initiation, VM's with Spice protocol shouldn't be initiate by rhevm-engine with 'cirrus' at all, only with 'qxl'. Scenario - to reproduce the BSOD : ******************************** 1) Created XP VM (Memory: 256MB, 2GB HD, display: Spice) 2) Create Template from that VM 3) Create desktop pool of 300 VM's 4) Initiate 100 VM's at once. Results: ******** - VM's stated with 'cirrus' device. - cirrus device caused BSOD on guests (BSOD only happened when stated 100 VM's on one host.) engine.log: ************ 2012-08-20 09:45:15,754 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-76) Recieved a Device without an address when processing VM c 0457e58-0ffb-4c9f-bd14-e080e812e102 devices, skipping device: {shared=false, iface=fdc, index=0, specParams={vmPayload={file={sysprep.inf=O1NldHVwTWdyVGFnDQpbVW5hdHRlbmRl ZF0NCiAgICBJbnN0YWxsRmlsZXNQYXRoPUM6XGkzODYN CiAgICBPZW1Ta2lwRXVsYT1ZZXMNCiAgICBUYXJnZXRQYXRoPVxXSU5ET1dTDQoNCltHdWlVbmF0 dGVuZGVkXQ0KICAgIEFkbWluUGFzc3dvcmQ9MTIzNDU2DQogICAgRW5jcnlwdGVkQWRtaW5QYXNz d29yZD1OTw0KICAgIE9FTVNraXBSZWdpb25hbD0xDQogICAgVGltZVpvbmU9ODUNCiAgICBPZW1T a2lwV2VsY29tZT0xDQoNCltVc2VyRGF0YV0NCiAgICBQcm9kdWN0S2V5PQ0KICAgIEZ1bGxOYW1l PSJ1c2VyIg0KICAgIE9yZ05hbWU9cmVkaGF0DQogICAgQ29tcHV0ZXJOYW1lPXNwaWNlLXNlcnZl cg0KDQpbRGlzcGxheV0NCiAgICBCaXRzUGVyUGVsPTMyDQogICAgWHJlc29sdXRpb249ODAwDQog ICAgWVJlc29sdXRpb249NjAwDQogICAgVnJlZnJlc2g9NzUNCg0KW1JlZ2lvbmFsU2V0dGluZ3Nd DQogICAgTGFuZ3VhZ2VHcm91cD0xDQogICAgTGFuZ3VhZ2U9MDAwMDA0MDkNCg0KW0lkZW50aWZp Y2F0aW9uXQ0KICAgIEpvaW5Eb21haW49DQogICAgRG9tYWluQWRtaW49DQogICAgRG9tYWluQWRt aW5QYXNzd29yZD0NCg0KW05ldHdvcmtpbmddDQogICAgSW5zdGFsbERlZmF1bHRDb21wb25lbnRz PVllcw0KDQpbQnJhbmRpbmddDQogICAgQnJhbmRJRVVzaW5nVW5hdHRlbmRlZD1ZZXMNCg0KW1By b3h5XQ0KICAgIFByb3h5X0VuYWJsZT0wDQogICAgVXNlX1NhbWVfUHJveHk9MA0KDQpbR3VpUnVu T25jZV0NCiAgICANCg0K }}}, device=floppy, path=, type=disk, readonly=true, deviceId=d18b5606-2922-469b-b478-ecf6673ae202}. 2012-08-20 09:45:15,754 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-76) Recieved a Device without an address when processing VM c 0457e58-0ffb-4c9f-bd14-e080e812e102 devices, skipping device: {specParams={vram=65536}, device=cirrus, type=video, deviceId=c0f79abd-bfff-49bc-a938-12ba324ffea4}. 2012-08-20 09:45:15,754 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-76) Recieved a Device without an address when processing VM c 0457e58-0ffb-4c9f-bd14-e080e812e102 devices, skipping device: {shared=false, iface=ide, index=2, specParams={path=}, device=cdrom, path=/rhev/data-center/b080a021-a2b2-4db2-be23-ce7ddc359f49/2612f0b2-8310-4d1f-adba-b708eb95f5cf/images/11111111-1111-1111-1111-111111111111/RHEV-toolsSetup_3.1_2.iso, type=disk, readonly=true, deviceId=85f6edb1-db8c-44ec-8e1d-548afabd10ab}. 2012-08-20 09:45:15,754 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-76) Recieved a Device without an address when processing VM c0457e58-0ffb-4c9f-bd14-e080e812e102 devices, skipping device: {shared=false, index=0, volumeID=61770a44-8843-4752-886d-fcddf5a51912, propagateErrors=off, format=cow, type=disk, reqsize=0, iface=ide, bootOrder=1, imageID=9cbcd0b6-bd0a-4fe5-8fb4-5497847b3c76, domainID=11c17e0f-74e1-49af-91c3-6132f611a5c8, specParams={}, needExtend=false, optional=false, device=disk, poolID=b080a021-a2b2-4db2-be23-ce7ddc359f49, readonly=false, deviceId=9cbcd0b6-bd0a-4fe5-8fb4-5497847b3c76}. 2012-08-20 09:45:15,754 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-76) Recieved a Device without an address when processing VM c0457e58-0ffb-4c9f-bd14-e080e812e102 devices, skipping device: {nicModel=pv, specParams={}, device=bridge, macAddr=00:1a:4a:23:74:b1, type=interface, deviceId=9a5a5a0a-8e09-4bea-9293-06403e1c0636, network=rhevm}. gal-mig-105 qemu-cmdline : ***************************** /usr/libexec/qemu-kvm-S-Mrhel6.3.0-cpuOpteron_G1-enable-kvm-m256-smp1,sockets=1,cores=1,threads=1-namegal-mig-105-uuid6051ceef-fd93-4c0f-89fa-15eca9b6453c-smbiostype=1,manufacturer=Red Hat,product=RHEV Hypervisor,version=6Server-6.3.0.3.el6,serial=01F92258-C006-E111-9237-E83935C02047_E8:39:35:B6:17:54,uuid=6051ceef-fd93-4c0f-89fa-15eca9b6453c-nodefconfig-nodefaults-chardevsocket,id=charmonitor,path=/var/lib/libvirt/qemu/gal-mig-105.monitor,server,nowait-monchardev=charmonitor,id=monitor,mode=control-rtcbase=2012-08-19T15:37:05,driftfix=slew-no-shutdown-devicevirtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x5-devicepiix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2-drivefile=/var/run/vdsm/payload/6051ceef-fd93-4c0f-89fa-15eca9b6453c.fec30405b53ccdf5ea13312737c900db.img,if=none,id=drive-fdc0-0-0,readonly=on,format=raw,serial=-globalisa-fdc.driveA=drive-fdc0-0-0-drivefile=/rhev/data-center/b080a021-a2b2-4db2-be23-ce7ddc359f49/11c17e0f-74e1-49af-91c3-6132f611a5c8/images/c49c9135-4a95-462d-945f-26d9a1fd9670/327eefee-35a7-4214-8a04-5e11f6995853,if=none,id=drive-ide0-0-0,format=qcow2,serial=c49c9135-4a95-462d-945f-26d9a1fd9670,cache=none,werror=stop,rerror=stop,aio=native-deviceide-drive,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0,bootindex=1-driveif=none,media=cdrom,id=drive-ide0-1-0,readonly=on,format=raw,serial=-deviceide-drive,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0-netdevtap,fd=123,id=hostnet0,vhost=on,vhostfd=135-devicevirtio-net-pci,netdev=hostnet0,id=net0,mac=00:1a:4a:23:27:51,bus=pci.0,addr=0x3-chardevsocket,id=charchannel0,path=/var/lib/libvirt/qemu/channels/gal-mig-105.com.redhat.rhevm.vdsm,server,nowait-devicevirtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=com.redhat.rhevm.vdsm-chardevspicevmc,id=charchannel1,name=vdagent-devicevirtserialport,bus=virtio-serial0.0,nr=2,chardev=charchannel1,id=channel1,name=com.redhat.spice.0-chardevpty,id=charconsole0-devicevirtconsole,chardev=charconsole0,id=console0-spiceport=6090,tls-port=6091,addr=0,x509-dir=/etc/pki/vdsm/libvirt-spice,tls-channel=main,tls-channel=inputs-ken-us-vgacirrus-deviceintel-hda,id=sound0,bus=pci.0,addr=0x4-devicehda-duplex,id=sound0-codec0,bus=sound0.0,cad=0-incomingtcp:0.0.0.0:49168
Created attachment 605690 [details] cirrus_bsod_screen_shot
Created attachment 605701 [details] full_engine.log
Libvirtd XML part from vdsm.log (full vdsm.log attached): ********************************************************** hread-7030::DEBUG::2012-08-20 08:45:17,126::libvirtvm::1345::vm.Vm::(_run) vmId=`c0457e58-0ffb-4c9f-bd14-e080e812e102`::<?xml version="1.0" encoding="utf-8"?> <domain type="kvm"> <name>spice-server</name> <uuid>c0457e58-0ffb-4c9f-bd14-e080e812e102</uuid> <memory>262144</memory> <currentMemory>262144</currentMemory> <vcpu>1</vcpu> <devices> <channel type="unix"> <target name="com.redhat.rhevm.vdsm" type="virtio"/> <source mode="bind" path="/var/lib/libvirt/qemu/channels/spice-server.com.redhat.rhevm.vdsm"/> </channel> <input bus="ps2" type="mouse"/> <channel type="spicevmc"> <target name="com.redhat.spice.0" type="virtio"/> </channel> <graphics autoport="yes" keymap="en-us" listen="0" passwd="*****" passwdValidTo="1970-01-01T00:00:01" port="-1" tlsPort="-1" type="spice"> <channel mode="secure" name="main"/> <channel mode="secure" name="inputs"/> </graphics> <console type="pty"> <target port="0" type="virtio"/> </console> <video> <model heads="1" type="cirrus" vram="65536"/> </video>
Created attachment 605706 [details] vdsm.log
Marking as urgent, regression, targeting beta2. I suspect it has to do with stable PCI addresses.
Note: No need to initiate 100 VM's.. Easy Steps to reproduce: ************************ 1) Create 1 'Server VM' (Default: display protocol = "Spice"). 2) Check the VM at the REHVM DB video | cirrus Data Base View: ***************** engine=# SELECT vm_guid from vms where vm_name ='22'; vm_guid -------------------------------------- ff6d2701-3d63-4c3d-818d-483a2515b9f7 (1 row) engine=# SELECT type,device from vm_device where vm_id ='c0457e58-0ffb-4c9f-bd14-e080e812e102'; type | device ------------+--------------- controller | ide controller | virtio-serial channel | unix channel | spicevmc interface | bridge disk | disk video | cirrus disk | cdrom controller | usb (9 rows)
*** Bug 850782 has been marked as a duplicate of this bug. ***
seems audio devices are screwed up too. Windows XP gets ich6 whereas it should be ac97
(In reply to comment #8) > seems audio devices are screwed up too. > Windows XP gets ich6 whereas it should be ac97 Do we have a bug for that?
We filed https://bugzilla.redhat.com/show_bug.cgi?id=850823 for the sound issue, pls close If there is one already ( could not find).
no, there's not. It may very well be the same root cause
roy sent a patch http://gerrit.ovirt.org/#/c/7414/
What are the steps to reproduce, did you have to change the display type for this to trigger?
(In reply to comment #13) > What are the steps to reproduce, did you have to change the display type for > this to trigger? Easiest way to reproduce getting Cirrus on VMs with Spice display is described in - https://bugzilla.redhat.com/show_bug.cgi?id=850782 Create a VM from template which was exported from RHEVM3.0 setup and imported to RHEVM3.1. but as I understand Omri was able to reproduce that in other scenarios as well. I suspect there are some situations when that happens. We do not have hardware resources to run 100 VMs probably.
fixed in commit 6dda285,7f79efb
*** Bug 852590 has been marked as a duplicate of this bug. ***
need to test the reverse use case from bug 852590 as well
Verified on RHEVM (Build SI23) both cases worked properly : - Server/Desktop VMs with Spice protocol --> boot with: type="qxl" - Server/Desktop Vms with VNC protocol --> boot with: type="cirrus" libvirt.log VM with Spice: ************************** </channel> <graphics autoport="yes" keymap="en-us" listen="0" passwd="*****" passwdValidTo="1970-01-01T00:00:01" port="-1" tlsPort="-1" type="spice"> <channel mode="secure" name="main"/> <channel mode="secure" name="inputs"/> <channel mode="secure" name="cursor"/> <channel mode="secure" name="playback"/> <channel mode="secure" name="record"/> <channel mode="secure" name="display"/> <channel mode="secure" name="usbredir"/> <channel mode="secure" name="smartcard"/> </graphics> <console type="pty"> <target port="0" type="virtio"/> </console> <sound model="ac97"/> <video> <model heads="1" type="qxl" vram="65536"/> </video> libvirt.log VM with VNC: ************************ </channel> <input bus="usb" type="tablet"/> <graphics autoport="yes" keymap="en-us" listen="0" passwd="*****" passwdValidTo="1970-01-01T00:00:01" port="-1" type="vnc"/> <console type="pty"> <target port="0" type="virtio"/> </console> <sound model="ich6"/> <video> <model heads="1" type="cirrus" vram="65536"/> </video>