Bug 849635 - ovrit-engine-backned: VMs with Spice display protocol start with 'cirrus' driver instead of QXL
ovrit-engine-backned: VMs with Spice display protocol start with 'cirrus' dri...
Status: CLOSED CURRENTRELEASE
Product: Red Hat Enterprise Virtualization Manager
Classification: Red Hat
Component: ovirt-engine (Show other bugs)
3.1.0
x86_64 Linux
urgent Severity urgent
: ---
: ---
Assigned To: Roy Golan
virt
: Regression
: 850782 852590 (view as bug list)
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2012-08-20 08:05 EDT by Omri Hochman
Modified: 2015-09-22 09 EDT (History)
13 users (show)

See Also:
Fixed In Version: si16
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2012-12-04 15:05:52 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
cirrus_bsod_screen_shot (16.75 KB, image/png)
2012-08-20 08:07 EDT, Omri Hochman
no flags Details
full_engine.log (174.37 KB, application/octet-stream)
2012-08-20 08:58 EDT, Omri Hochman
no flags Details
vdsm.log (834.10 KB, application/octet-stream)
2012-08-20 09:11 EDT, Omri Hochman
no flags Details

  None (edit)
Description Omri Hochman 2012-08-20 08:05:45 EDT
ovrit-engine-backned: VMs with Spice display protocol start with 'cirrus' driver which used by 'VNC' and caused BSOD on scale environment.

Description:
************
From some reason when starting single VM with Spice, as display protocol, the VM starts with 'cirrus' (which used by VNC), I've encountered a problem of BSOD on guest when initiated 100 VM's at once on one host.
The BSOD message indicates the following: "The problem seems to caused by the following file: cirrus" (See: attached screen shot).
 
Although the BSOD caused by 'cirrus' and probably related to the load during VM initiation,  VM's with Spice protocol shouldn't be initiate by rhevm-engine with 'cirrus' at all, only with 'qxl'.  

Scenario - to reproduce the BSOD :
********************************
1) Created XP VM  (Memory: 256MB, 2GB HD, display: Spice) 
2) Create Template from that VM 
3) Create desktop pool of 300 VM's 
4) Initiate 100 VM's at once. 

Results:
********
- VM's stated with 'cirrus' device.
- cirrus device caused BSOD on guests 
(BSOD only happened when stated 100 VM's on one host.)
 

engine.log:
************
2012-08-20 09:45:15,754 INFO  [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-76) Recieved a Device without an address when processing VM c
0457e58-0ffb-4c9f-bd14-e080e812e102 devices, skipping device: {shared=false, iface=fdc, index=0, specParams={vmPayload={file={sysprep.inf=O1NldHVwTWdyVGFnDQpbVW5hdHRlbmRl
ZF0NCiAgICBJbnN0YWxsRmlsZXNQYXRoPUM6XGkzODYN
CiAgICBPZW1Ta2lwRXVsYT1ZZXMNCiAgICBUYXJnZXRQYXRoPVxXSU5ET1dTDQoNCltHdWlVbmF0
dGVuZGVkXQ0KICAgIEFkbWluUGFzc3dvcmQ9MTIzNDU2DQogICAgRW5jcnlwdGVkQWRtaW5QYXNz
d29yZD1OTw0KICAgIE9FTVNraXBSZWdpb25hbD0xDQogICAgVGltZVpvbmU9ODUNCiAgICBPZW1T
a2lwV2VsY29tZT0xDQoNCltVc2VyRGF0YV0NCiAgICBQcm9kdWN0S2V5PQ0KICAgIEZ1bGxOYW1l
PSJ1c2VyIg0KICAgIE9yZ05hbWU9cmVkaGF0DQogICAgQ29tcHV0ZXJOYW1lPXNwaWNlLXNlcnZl
cg0KDQpbRGlzcGxheV0NCiAgICBCaXRzUGVyUGVsPTMyDQogICAgWHJlc29sdXRpb249ODAwDQog
ICAgWVJlc29sdXRpb249NjAwDQogICAgVnJlZnJlc2g9NzUNCg0KW1JlZ2lvbmFsU2V0dGluZ3Nd
DQogICAgTGFuZ3VhZ2VHcm91cD0xDQogICAgTGFuZ3VhZ2U9MDAwMDA0MDkNCg0KW0lkZW50aWZp
Y2F0aW9uXQ0KICAgIEpvaW5Eb21haW49DQogICAgRG9tYWluQWRtaW49DQogICAgRG9tYWluQWRt
aW5QYXNzd29yZD0NCg0KW05ldHdvcmtpbmddDQogICAgSW5zdGFsbERlZmF1bHRDb21wb25lbnRz
PVllcw0KDQpbQnJhbmRpbmddDQogICAgQnJhbmRJRVVzaW5nVW5hdHRlbmRlZD1ZZXMNCg0KW1By
b3h5XQ0KICAgIFByb3h5X0VuYWJsZT0wDQogICAgVXNlX1NhbWVfUHJveHk9MA0KDQpbR3VpUnVu
T25jZV0NCiAgICANCg0K
}}}, device=floppy, path=, type=disk, readonly=true, deviceId=d18b5606-2922-469b-b478-ecf6673ae202}.
2012-08-20 09:45:15,754 INFO  [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-76) Recieved a Device without an address when processing VM c
0457e58-0ffb-4c9f-bd14-e080e812e102 devices, skipping device: {specParams={vram=65536}, device=cirrus, type=video, deviceId=c0f79abd-bfff-49bc-a938-12ba324ffea4}.
2012-08-20 09:45:15,754 INFO  [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-76) Recieved a Device without an address when processing VM c
0457e58-0ffb-4c9f-bd14-e080e812e102 devices, skipping device: {shared=false, iface=ide, index=2, specParams={path=}, device=cdrom, path=/rhev/data-center/b080a021-a2b2-4db2-be23-ce7ddc359f49/2612f0b2-8310-4d1f-adba-b708eb95f5cf/images/11111111-1111-1111-1111-111111111111/RHEV-toolsSetup_3.1_2.iso, type=disk, readonly=true, deviceId=85f6edb1-db8c-44ec-8e1d-548afabd10ab}.
2012-08-20 09:45:15,754 INFO  [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-76) Recieved a Device without an address when processing VM c0457e58-0ffb-4c9f-bd14-e080e812e102 devices, skipping device: {shared=false, index=0, volumeID=61770a44-8843-4752-886d-fcddf5a51912, propagateErrors=off, format=cow, type=disk, reqsize=0, iface=ide, bootOrder=1, imageID=9cbcd0b6-bd0a-4fe5-8fb4-5497847b3c76, domainID=11c17e0f-74e1-49af-91c3-6132f611a5c8, specParams={}, needExtend=false, optional=false, device=disk, poolID=b080a021-a2b2-4db2-be23-ce7ddc359f49, readonly=false, deviceId=9cbcd0b6-bd0a-4fe5-8fb4-5497847b3c76}.
2012-08-20 09:45:15,754 INFO  [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-76) Recieved a Device without an address when processing VM c0457e58-0ffb-4c9f-bd14-e080e812e102 devices, skipping device: {nicModel=pv, specParams={}, device=bridge, macAddr=00:1a:4a:23:74:b1, type=interface, deviceId=9a5a5a0a-8e09-4bea-9293-06403e1c0636, network=rhevm}.



gal-mig-105 qemu-cmdline :
*****************************
/usr/libexec/qemu-kvm-S-Mrhel6.3.0-cpuOpteron_G1-enable-kvm-m256-smp1,sockets=1,cores=1,threads=1-namegal-mig-105-uuid6051ceef-fd93-4c0f-89fa-15eca9b6453c-smbiostype=1,manufacturer=Red Hat,product=RHEV Hypervisor,version=6Server-6.3.0.3.el6,serial=01F92258-C006-E111-9237-E83935C02047_E8:39:35:B6:17:54,uuid=6051ceef-fd93-4c0f-89fa-15eca9b6453c-nodefconfig-nodefaults-chardevsocket,id=charmonitor,path=/var/lib/libvirt/qemu/gal-mig-105.monitor,server,nowait-monchardev=charmonitor,id=monitor,mode=control-rtcbase=2012-08-19T15:37:05,driftfix=slew-no-shutdown-devicevirtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x5-devicepiix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2-drivefile=/var/run/vdsm/payload/6051ceef-fd93-4c0f-89fa-15eca9b6453c.fec30405b53ccdf5ea13312737c900db.img,if=none,id=drive-fdc0-0-0,readonly=on,format=raw,serial=-globalisa-fdc.driveA=drive-fdc0-0-0-drivefile=/rhev/data-center/b080a021-a2b2-4db2-be23-ce7ddc359f49/11c17e0f-74e1-49af-91c3-6132f611a5c8/images/c49c9135-4a95-462d-945f-26d9a1fd9670/327eefee-35a7-4214-8a04-5e11f6995853,if=none,id=drive-ide0-0-0,format=qcow2,serial=c49c9135-4a95-462d-945f-26d9a1fd9670,cache=none,werror=stop,rerror=stop,aio=native-deviceide-drive,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0,bootindex=1-driveif=none,media=cdrom,id=drive-ide0-1-0,readonly=on,format=raw,serial=-deviceide-drive,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0-netdevtap,fd=123,id=hostnet0,vhost=on,vhostfd=135-devicevirtio-net-pci,netdev=hostnet0,id=net0,mac=00:1a:4a:23:27:51,bus=pci.0,addr=0x3-chardevsocket,id=charchannel0,path=/var/lib/libvirt/qemu/channels/gal-mig-105.com.redhat.rhevm.vdsm,server,nowait-devicevirtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=com.redhat.rhevm.vdsm-chardevspicevmc,id=charchannel1,name=vdagent-devicevirtserialport,bus=virtio-serial0.0,nr=2,chardev=charchannel1,id=channel1,name=com.redhat.spice.0-chardevpty,id=charconsole0-devicevirtconsole,chardev=charconsole0,id=console0-spiceport=6090,tls-port=6091,addr=0,x509-dir=/etc/pki/vdsm/libvirt-spice,tls-channel=main,tls-channel=inputs-ken-us-vgacirrus-deviceintel-hda,id=sound0,bus=pci.0,addr=0x4-devicehda-duplex,id=sound0-codec0,bus=sound0.0,cad=0-incomingtcp:0.0.0.0:49168
Comment 1 Omri Hochman 2012-08-20 08:07:45 EDT
Created attachment 605690 [details]
cirrus_bsod_screen_shot
Comment 2 Omri Hochman 2012-08-20 08:58:57 EDT
Created attachment 605701 [details]
full_engine.log
Comment 3 Omri Hochman 2012-08-20 09:01:11 EDT
Libvirtd XML part from vdsm.log (full vdsm.log attached):
**********************************************************
hread-7030::DEBUG::2012-08-20 08:45:17,126::libvirtvm::1345::vm.Vm::(_run) vmId=`c0457e58-0ffb-4c9f-bd14-e080e812e102`::<?xml version="1.0" encoding="utf-8"?>
<domain type="kvm">
        <name>spice-server</name>
        <uuid>c0457e58-0ffb-4c9f-bd14-e080e812e102</uuid>
        <memory>262144</memory>
        <currentMemory>262144</currentMemory>
        <vcpu>1</vcpu>
        <devices>
                <channel type="unix">
                        <target name="com.redhat.rhevm.vdsm" type="virtio"/>
                        <source mode="bind" path="/var/lib/libvirt/qemu/channels/spice-server.com.redhat.rhevm.vdsm"/>
                </channel>
                <input bus="ps2" type="mouse"/>
                <channel type="spicevmc">
                        <target name="com.redhat.spice.0" type="virtio"/>
                </channel>
                <graphics autoport="yes" keymap="en-us" listen="0" passwd="*****" passwdValidTo="1970-01-01T00:00:01" port="-1" tlsPort="-1" type="spice">
                        <channel mode="secure" name="main"/>
                        <channel mode="secure" name="inputs"/>
                </graphics>
                <console type="pty">
                        <target port="0" type="virtio"/>
                </console>
                <video>
                        <model heads="1" type="cirrus" vram="65536"/>
                </video>
Comment 4 Omri Hochman 2012-08-20 09:11:32 EDT
Created attachment 605706 [details]
vdsm.log
Comment 5 Yaniv Kaul 2012-08-20 09:54:11 EDT
Marking as urgent, regression, targeting beta2.
I suspect it has to do with stable PCI addresses.
Comment 6 Omri Hochman 2012-08-22 04:56:20 EDT
Note: No need to initiate 100 VM's..  

Easy Steps to reproduce:
************************
1) Create 1 'Server VM'  (Default: display protocol = "Spice").
2) Check the VM at the REHVM DB  video      | cirrus

Data Base View:
*****************
engine=# SELECT vm_guid from vms where vm_name ='22';
               vm_guid                
--------------------------------------
 ff6d2701-3d63-4c3d-818d-483a2515b9f7
(1 row)

engine=# SELECT type,device from vm_device where vm_id ='c0457e58-0ffb-4c9f-bd14-e080e812e102';
    type    |    device     
------------+---------------
 controller | ide
 controller | virtio-serial
 channel    | unix
 channel    | spicevmc
 interface  | bridge
 disk       | disk
 video      | cirrus
 disk       | cdrom
 controller | usb
(9 rows)
Comment 7 Yaniv Kaul 2012-08-22 08:42:12 EDT
*** Bug 850782 has been marked as a duplicate of this bug. ***
Comment 8 Michal Skrivanek 2012-08-22 09:10:49 EDT
seems audio devices are screwed up too.
Windows XP gets ich6 whereas it should be ac97
Comment 9 Marian Krcmarik 2012-08-22 09:23:23 EDT
(In reply to comment #8)
> seems audio devices are screwed up too.
> Windows XP gets ich6 whereas it should be ac97

Do we have a bug for that?
Comment 10 Marian Krcmarik 2012-08-22 09:39:10 EDT
We filed https://bugzilla.redhat.com/show_bug.cgi?id=850823 for the sound issue, pls close If there is one already ( could not find).
Comment 11 Michal Skrivanek 2012-08-22 09:42:46 EDT
no, there's not. It may very well be the same root cause
Comment 12 Itamar Heim 2012-08-23 03:03:39 EDT
roy sent a patch http://gerrit.ovirt.org/#/c/7414/
Comment 13 Andrew Cathrow 2012-08-23 08:22:11 EDT
What are the steps to reproduce, did you have to change the display type for this to trigger?
Comment 15 Marian Krcmarik 2012-08-23 08:42:00 EDT
(In reply to comment #13)
> What are the steps to reproduce, did you have to change the display type for
> this to trigger?

Easiest way to reproduce getting Cirrus on VMs with Spice display is described in
- https://bugzilla.redhat.com/show_bug.cgi?id=850782

Create a VM from template which was exported from RHEVM3.0 setup and imported to RHEVM3.1. but as I understand Omri was able to reproduce that in other scenarios as well. I suspect there are some situations when that happens. We do not have hardware resources to run 100 VMs probably.
Comment 16 Eli Mesika 2012-08-26 11:39:35 EDT
fixed in commit 6dda285,7f79efb
Comment 18 Itamar Heim 2012-08-28 23:47:26 EDT
*** Bug 852590 has been marked as a duplicate of this bug. ***
Comment 19 Itamar Heim 2012-08-28 23:48:53 EDT
need to test the reverse use case from bug 852590 as well
Comment 20 Omri Hochman 2012-11-04 09:11:29 EST
Verified on RHEVM (Build SI23) both cases worked properly : 

 - Server/Desktop VMs with Spice protocol --> boot with: type="qxl"
 - Server/Desktop Vms with VNC protocol --> boot with: type="cirrus"
 

libvirt.log VM with Spice: 
**************************
    </channel>
                <graphics autoport="yes" keymap="en-us" listen="0" passwd="*****" passwdValidTo="1970-01-01T00:00:01" port="-1" tlsPort="-1" type="spice">
                        <channel mode="secure" name="main"/>
                        <channel mode="secure" name="inputs"/>
                        <channel mode="secure" name="cursor"/>
                        <channel mode="secure" name="playback"/>
                        <channel mode="secure" name="record"/>
                        <channel mode="secure" name="display"/>
                        <channel mode="secure" name="usbredir"/>
                        <channel mode="secure" name="smartcard"/>
                </graphics>
                <console type="pty">
                        <target port="0" type="virtio"/>
                </console>
                <sound model="ac97"/>
                <video>
                        <model heads="1" type="qxl" vram="65536"/>
                </video>



libvirt.log VM with VNC: 
************************

         </channel>
                <input bus="usb" type="tablet"/>
                <graphics autoport="yes" keymap="en-us" listen="0" passwd="*****" passwdValidTo="1970-01-01T00:00:01" port="-1" type="vnc"/>
                <console type="pty">
                        <target port="0" type="virtio"/>
                </console>
                <sound model="ich6"/>
                <video>
                        <model heads="1" type="cirrus" vram="65536"/>
                </video>

Note You need to log in before you can comment on or make changes to this bug.