Bug 981094

Created attachment 768576 [details]
dumpxml

as requested

1) It's interesting to see that F18 virt-preview is getting libvirt-1.1.0. I had thought that the virt preview for any Fedora "n" was supposed to be the version of the same package in Fedora "n+1". The current libvirt release on F19 is libvirt-1.0.5, not libvirt-1.1.0

2) Actually I've just recently noticed the same thing on F19 with libvirt-1.1.0.

As far as I can tell, it only happens when using spice for the video. Can you verify that your mouse reappears when you switch to VNC as well? Also, can you verify that the mouse comes back if you downgrade to libvirt 1.0.6 (or 1.0.5 if you can't downgrade to 1.0.6)?

(In the meantime since, even with the apparent error in what has been built for f18-virt-preview, the libvirt-1.1.0 package is only in virt-preview and not an official part of Fedora, I think we should change the version from F18 to rawhide (in F19, libvirt-1.1.0 is only in virt-preview too). Any differing opinions on that?

1. The mouse works with VNC

2. The mouse works when I downgrade to 1.0.6

This is fine but you'll want to fix it at some point

Can you provide /var/log/libvirt/qemu/$GUESTNAME.log  showing the cli args used, both when launching with 1.1.0 and 1.0.6, so we can see if there was any relevant change.

it is in the 1st attachment. two of each. there is a qxl difference

Ok so we have:

-vga qxl -global qxl-vga.ram_size=67108864 -global qxl-vga.vram_size=67108864 


vs

-device qxl-vga,id=video0,ram_size=67108864,vram_size=67108864,bus=pci.0,addr=0x2 


which is a result of this change

commit 0ad9025ef4474bbdaac6d0ba6b4b25c60fec3aa3
Author: Guannan Ren <gren>
Date:   Tue Jun 18 14:09:20 2013 +0800

    qemu: set QEMU_CAPS_DEVICE_VIDEO_PRIMARY cap flag in QMP detection
    
    When qemu >= 1.20, it is safe to use -device for primary video
    device as described in 4c993d8ab.


I guess it is not quite as safe as the commit message suggests. No idea why this would affect the mouse pointer though. Something fishy going on indeed

Here's the email thread about the earlier patchset that's referenced in the above commit:

http://www.redhat.com/archives/libvir-list/2012-December/msg00769.html

I can't tell from reading the thread, but I'm guessing that this patch is part of allowing multiple monitors in a guest?

One thing that caught my eye that seems bothersome to me (aside from the fact that the mouse no longer displays) is that the PCI address of the graphics device is completely left up to qemu, so it could potentially end up in a different place if other devices were added (or if qemu's slot allocation algorithm was modified.

Some more facts about the failure:

* The mouse *can* still point at things in the guest - you can see the highlighting of some elements change as the mouse passes over them, and can click on things if you guess the position correctly. It is only the *display* of the mouse pointer that is missing.

* The mouse pointer only fails to display when using the QXL device type along with spice. If you use the standard vga device type and spice, the cursor displays as normal.

* The failure to display the mouse pointer occurs on both a Windows XP guest and Fedora 18 guest.

* In the qemu log, the non-working run of the guest qemu has these additional log messages at startup:

(null):32712): SpiceWorker-Warning **: red_worker.c:11477:dev_destroy_primary_surface: double destroy of primary surface
((null):32712): SpiceWorker-Warning **: red_worker.c:9663:red_create_surface: condition `surface->context.canvas' reached

and also these:

red_channel_client_pre_create_validate: Error client 0x7fbd6ae35390: duplicate channel type 4 id 0
red_channel_client_pre_create_validate: Error client 0x7fbd6ae35390: duplicate channel type 2 id 0

And finally, at least some of the time, the entire guest display is totally black; I haven't figured out the exact condition that causes this though - it doesn't happen 100% of the time.

(In reply to Daniel Berrange from comment #6)
> Ok so we have:
> 
> -vga qxl -global qxl-vga.ram_size=67108864 -global
> qxl-vga.vram_size=67108864 
> 
> 
> vs
> 
> -device
> qxl-vga,id=video0,ram_size=67108864,vram_size=67108864,bus=pci.0,addr=0x2 
> 
> 
> which is a result of this change
> 
> commit 0ad9025ef4474bbdaac6d0ba6b4b25c60fec3aa3
> Author: Guannan Ren <gren>
> Date:   Tue Jun 18 14:09:20 2013 +0800
> 
>     qemu: set QEMU_CAPS_DEVICE_VIDEO_PRIMARY cap flag in QMP detection
>     
>     When qemu >= 1.20, it is safe to use -device for primary video
>     device as described in 4c993d8ab.
> 
> 
> I guess it is not quite as safe as the commit message suggests. No idea why
> this would affect the mouse pointer though. Something fishy going on indeed

This bug aimed to fix bug 823535.
-device qxl-vga is primary video device we can fully control the PCI address of each device(not fixed 0:0:2.0), the non-primary cards use -device qxl, each of these video devices only support one head.
-vga qxl support multiple heads through a single PCI device. the PCI address has to be in 0:0:2.0 slot.

The -device VGA, -device cirrus-vga, -device vmware-svga works fine. And this error informations are thrown out by spice server, So I consider more in spice server direction. There is duplicate bug 980714 tracking this.

(In reply to Gunannan Ren from comment #9)
> (In reply to Daniel Berrange from comment #6)
> > Ok so we have:
> > 
> > -vga qxl -global qxl-vga.ram_size=67108864 -global
> > qxl-vga.vram_size=67108864 
> > 
> > 
> > vs
> > 
> > -device
> > qxl-vga,id=video0,ram_size=67108864,vram_size=67108864,bus=pci.0,addr=0x2 
> > 
> > 
> > which is a result of this change
> > 
> > commit 0ad9025ef4474bbdaac6d0ba6b4b25c60fec3aa3
> > Author: Guannan Ren <gren>
> > Date:   Tue Jun 18 14:09:20 2013 +0800
> > 
> >     qemu: set QEMU_CAPS_DEVICE_VIDEO_PRIMARY cap flag in QMP detection
> >     
> >     When qemu >= 1.20, it is safe to use -device for primary video
> >     device as described in 4c993d8ab.
> > 
> > 
> > I guess it is not quite as safe as the commit message suggests. No idea why
> > this would affect the mouse pointer though. Something fishy going on indeed
> 
> This bug aimed to fix bug 823535.
> -device qxl-vga is primary video device we can fully control the PCI address
> of each device(not fixed 0:0:2.0), the non-primary cards use -device qxl,
> each of these video devices only support one head.
> -vga qxl support multiple heads through a single PCI device. the PCI address
> has to be in 0:0:2.0 slot.

I wonder if it does this for q35-* machinetypes as well. Although a q35 should work fine with the graphics car at that address, it's not a requirement of the spec, and actually probably never occurs in the real world. It's a bit annoying to have such a restriction...

> 
> The -device VGA, -device cirrus-vga, -device vmware-svga works fine. And
> this error informations are thrown out by spice server, So I consider more
> in spice server direction. There is duplicate bug 980714 tracking this.

That BZ (which is filed against RHEL7 spice-server) describes the screen going completely black. I have seen that behavior a couple times, but most of the time my video display is fine, except that the mouse cursor display is missing. 

I'm going to reassign this one to spice (Fedora has no "spice-server" package) so they get a look at it.

Note I'm seeing this in Rawhide as well.

(In reply to Adam Williamson from comment #11)
> Note I'm seeing this in Rawhide as well.

Yeah, from libvirt's POV it really is a rawhide problem. The only reason that we're seeing it in F18 and F19 is because we're running libvirt-1.1.0 out of the virt-preview repo. I don't know if the spice bug is present in the stock spice for F18/F19, but non-virt-preview people wouldn't see it anyway, since older libvirt doesn't exercise the bug.

(and while I'm here - thanks for noticing/fixing the dyslexia in my change to the description, John :-)

Any spice folks care to comment on this, and maybe the general idea of using -vga qxl vs. -device qxl-vga since it seems they exercise different code paths?

Spent a bit of time on this.
What is happening is that QEMU main() in vl.c has:
#ifdef CONFIG_SPICE
    if (using_spice && !qxl_enabled) {
        qemu_spice_display_init(ds);
    }
#endif

qxl_enabled is a macro that expands to
#define qxl_enabled (vga_interface_type == VGA_QXL)

When using -vga qxl, this condition is true, when using the -device syntax, this condition is false as vga_interface_type is VGA_NONE in this case.
In -device vga-qxl case, we end up with an additional call to qemu_spice_display_init().
If vga_interface_type is forced to 5 (VGA_QXL) in gdb so that qemu_spice_display_init() is not called, then the mouse cursor issue goes away.

I don't know what the fix should be though, and I'm about to leave for holidays, so it would be nice if someone can take this over ;)

Should we clone this bug to libvirt to avoid using -device qxl-vga if it detects a broken version of qemu?

Created attachment 776905 [details]
qemu spice display fix

anyone can try the patch please?

(In reply to Eric Blake from comment #15)
> Should we clone this bug to libvirt to avoid using -device qxl-vga if it
> detects a broken version of qemu?

Yes, please.  1.6+ should work /me hopes (need to catch up with post-vacation email backlog and check qemu devel state though).

(In reply to Gerd Hoffmann from comment #16)
> Created attachment 776905 [details]
> qemu spice display fix
> 
> anyone can try the patch please?

After I compile upstream qemu git head, qemu-system-x86_64 failed to boot up guest without patching the attachment, so I can't test it.

((null):7931): SpiceWorker-Warning **: red_worker.c:11477:dev_destroy_primary_surface: double destroy of primary surface
((null):7931): SpiceWorker-Warning **: red_worker.c:9663:red_create_surface: condition `surface->context.canvas' reached
qemu-system-x86_64: /home/gren/qemu/exec.c:1927: memory_access_size: Assertion `l >= access_size_min' failed.

Should I use qemu git head there?

Scratch build for Rawhide with the patch:

http://koji.fedoraproject.org/koji/taskinfo?taskID=5642281

Testing looks good here - I switched my main test VM back to qxl/Spice and I have a cursor. 'ps aux' output shows the qemu command is running with '-device qxl-vga'.

(In reply to Gunannan Ren from comment #18)
> (In reply to Gerd Hoffmann from comment #16)
> > Created attachment 776905 [details]
> > qemu spice display fix
> > 
> > anyone can try the patch please?
> 
> After I compile upstream qemu git head, qemu-system-x86_64 failed to boot up
> guest without patching the attachment, so I can't test it.
> 
> ((null):7931): SpiceWorker-Warning **:
> red_worker.c:11477:dev_destroy_primary_surface: double destroy of primary
> surface
> ((null):7931): SpiceWorker-Warning **: red_worker.c:9663:red_create_surface:
> condition `surface->context.canvas' reached
> qemu-system-x86_64: /home/gren/qemu/exec.c:1927: memory_access_size:
> Assertion `l >= access_size_min' failed.
> 
> Should I use qemu git head there?

git pull again this morning, these errors doesn't be thrown out.
The patch is good for me too.

It would be great to backport this fix to qemu-kvm on RHEL7. We have bugs against virt-manager/virt-viewer tracking this issue too on RHEL7.

Latest patch:

https://lists.gnu.org/archive/html/qemu-devel/2013-07/msg03879.html

qemu 1.6 will be patched; meanwhile see bug 986384 for libvirt's workaround for broken qemu.

Fixed in qemu-1.5.2-1.fc20