Bug 1536883

Frame # 	Function 	Binary 	Source or offset 	Line
1 	raise 	/lib64/libc.so.6 	0x361a7
Build id: 11c58ba248c7c3e0e5fae83349cbac44481d832b 	-
2 	abort 	/lib64/libc.so.6 	0x37898
Build id: 11c58ba248c7c3e0e5fae83349cbac44481d832b 	-
3 	__assert_fail_base 	/lib64/libc.so.6 	0x2efc8
Build id: 11c58ba248c7c3e0e5fae83349cbac44481d832b 	-
4 		/lib64/libc.so.6 	0x2f074
Build id: 11c58ba248c7c3e0e5fae83349cbac44481d832b 	-
5 		/usr/libexec/qemu-kvm 	0x1f3abc
Build id: c57dca16fa2d724d3585001769785ede17a83718 	-
6 	vmstate_save_state 	/usr/libexec/qemu-kvm 	0x23221d
Build id: c57dca16fa2d724d3585001769785ede17a83718 	-
7 	qemu_savevm_state_complete 	/usr/libexec/qemu-kvm 	0x23281b
Build id: c57dca16fa2d724d3585001769785ede17a83718 	-
8 	migration_thread 	/usr/libexec/qemu-kvm 	0x16959c
Build id: c57dca16fa2d724d3585001769785ede17a83718 	-
9 	start_thread 	/lib64/libpthread.so.0 	0x7dd5
Build id: cdf948dc7dfc0b6c3ffb5cf65ce4540a4946b815 	-
10 	__clone 	/lib64/libc.so.6 	0xfe94d
Build id: 11c58ba248c7c3e0e5fae83349cbac44481d832b 	-

Hi Vladimir,
  OK, so hand looking up entry '5' in that trace leeds me to the end of qxl_pre_save:
    assert(d->last_release_offset < d->vga.vram_size);

can you tell me:
  a) How were you running qemu?
  b) What guest was in it?
  c) What did you do just before it failed?

Dave

I think it is fixed by:

86dbcdd9c7 qxl: add migration blocker to avoid pre-save assert

see also

https://bugs.launchpad.net/qemu/+bug/1635339

(In reply to Laurent Vivier from comment #4)
> I think it is fixed by:
> 
> 86dbcdd9c7 qxl: add migration blocker to avoid pre-save assert
> 
> see also
> 
> https://bugs.launchpad.net/qemu/+bug/1635339

Ooh nice spot; yes that's possible, lets see if Vladimir was running Windows with the old driver.

Yeah, Windows 10. Not sure about the driver version, i am not able to use keyboard in the VM now as something is crippled totally.

OK Vladimir, that explains the crash at least.
Hmm now why is the keyboard broken?
How are you running this? Using qemu directly or through virt-manager or boxes or what?

So I found out that it is caused by our fancy Wayland preview. Switching back to X makes the trick. Sorry for confusion. We have the crash only now. 
I executed it in Boxes and in virt-manager both behaved the same.

OK, great; you should find that if you upgrade to the latest guest qxl drivers in the guest the abort goes away ( wddm dod 0.17 version ).

Please try with the updated guest driver and report back!

*** Bug 1537468 has been marked as a duplicate of this bug. ***

libvirtError: internal error: unable to execute QEMU command 'migrate': qxl: guest bug: command not in ram bar

was visible in virt-manager when I tried to save it  otherwise machines seems to stay all the time up not saving itself.

give me some more time to exercise it properly, but so far so good

(In reply to Vladimir Benes from comment #16)
> libvirtError: internal error: unable to execute QEMU command 'migrate': qxl:
> guest bug: command not in ram bar
> 
> was visible in virt-manager when I tried to save it  otherwise machines
> seems to stay all the time up not saving itself.

Excellent - now if you upgrade the guest drivers you should be able to do a full migrate.

Fix included in qemu-kvm-1.5.3-154.el7

(In reply to Dr. David Alan Gilbert from comment #18)
> (In reply to Vladimir Benes from comment #16)
> > libvirtError: internal error: unable to execute QEMU command 'migrate': qxl:
> > guest bug: command not in ram bar
> > 
> > was visible in virt-manager when I tried to save it  otherwise machines
> > seems to stay all the time up not saving itself.
> 
> Excellent - now if you upgrade the guest drivers you should be able to do a
> full migrate.

I am, Windows updated itself in the meantime with old driver, and the behaviour stayed the same, it was not possible to save or pause with the same error. After upgrading driver to version 18 I am able to save again. Thanks. This solution seems to be very robust.

(In reply to Vladimir Benes from comment #23)
> (In reply to Dr. David Alan Gilbert from comment #18)
> > (In reply to Vladimir Benes from comment #16)
> > > libvirtError: internal error: unable to execute QEMU command 'migrate': qxl:
> > > guest bug: command not in ram bar
> > > 
> > > was visible in virt-manager when I tried to save it  otherwise machines
> > > seems to stay all the time up not saving itself.
> > 
> > Excellent - now if you upgrade the guest drivers you should be able to do a
> > full migrate.
> 
> I am, Windows updated itself in the meantime with old driver, and the
> behaviour stayed the same, it was not possible to save or pause with the
> same error. 

Do you mean the bad driver is present in the public windows update system???

> After upgrading driver to version 18 I am able to save again.
> Thanks. This solution seems to be very robust.

(In reply to Dr. David Alan Gilbert from comment #24)
> (In reply to Vladimir Benes from comment #23)
> > (In reply to Dr. David Alan Gilbert from comment #18)
> > > (In reply to Vladimir Benes from comment #16)
> > > > libvirtError: internal error: unable to execute QEMU command 'migrate': qxl:
> > > > guest bug: command not in ram bar
> > > > 
> > > > was visible in virt-manager when I tried to save it  otherwise machines
> > > > seems to stay all the time up not saving itself.
> > > 
> > > Excellent - now if you upgrade the guest drivers you should be able to do a
> > > full migrate.
> > 
> > I am, Windows updated itself in the meantime with old driver, and the
> > behaviour stayed the same, it was not possible to save or pause with the
> > same error. 
> 
> Do you mean the bad driver is present in the public windows update system???
> 
It was pretty huge update but I think I had updated driver before I installed the update and the driver version was back to 15 when everything finished. But not entirely sure will try to reproduce it with some older windows installation. Maybe some restart was missing in between driver install and windows update. Really cannot say for sure now.

I could reproduce this issue by sending savevm to QEMU.

Packages I tested:
qemu-kvm-1.5.3-153.el7.x86_64
spice-qxl-wddm-dod-0.16-Signed.zip


# /usr/libexec/qemu-kvm -name avocado-vt-vm1 -sandbox off -machine pc -nodefaults -vga qxl -monitor stdioe ich9-usb-ehci1,id=usb1,addr=0x1d.7,multifunction=on,bus=pci.0 -device ich9-usb-uhci1,id=usb1.0,multifunction=on,masterbus=usb1.0,addr=0x1d.0,firstport=0,bus=pci.0 -device ich9-usb-uhci2,id=usb1.1,multifunction=on,masterbus=usb1.0,addr=0x1d.2,firstport=2,bus=pci.0 -device ich9-usb-uhci3,id=usb1.2,multifunction=on,masterbus=usb1.0,addr=0x1d.4,firstport=4,bus=pci.0 -drive id=drive_image1,if=none,snapshot=off,aio=native,cache=none,format=qcow2,file=/home/chayang/win10.qcow2 -device virtio-blk-pci,id=image1,drive=drive_image1,bootindex=0,bus=pci.0,addr=0x4 -device virtio-net-pci,mac=9a:5e:5f:60:61:62,id=idrScAZ6,vectors=4,netdev=idFczgUm,bus=pci.0,addr=0x5 -netdev tap,id=idFczgUm,vhost=on -m 4096 -smp 4,cores=2,threads=1,sockets=2 -cpu Penryn,+fsgsbase -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1 -spice disable-ticketing,port=5000 -rtc base=localtime,clock=host,driftfix=slew -boot order=cdn,once=c,menu=on,strict=off -enable-kvm 
 QEMU 1.5.3 monitor - type 'help' for more information
 (qemu) main_channel_link: add main channel client
 main_channel_client_handle_pong: net test: latency 1.807000 ms, bitrate 95616041 bps (91.186563 Mbps)
 inputs_connect: inputs channel client create
 red_qxl_set_cursor_peer: 

 (qemu) 
 (qemu) 
 (qemu) savevm 
 qemu-kvm: /builddir/build/BUILD/qemu-1.5.3/hw/display/qxl.c:2183: qxl_pre_save: Assertion `d->last_release_offset < d->vga.vram_size' failed.
 Aborted (core dumped)

(gdb) bt
#0  0x00007fb213cde1d7 in raise () at /lib64/libc.so.6
#1  0x00007fb213cdf8d0 in abort () at /lib64/libc.so.6
#2  0x00007fb213cd6fcc in __assert_fail_base () at /lib64/libc.so.6
#3  0x00007fb213cd7088 in  () at /lib64/libc.so.6
#4  0x000055b4ca936abc in qxl_pre_save (opaque=<optimized out>) at /usr/src/debug/qemu-1.5.3/hw/display/qxl.c:2183
#5  0x000055b4ca97521d in vmstate_save_state (f=f@entry=0x55b4d5e0a000, vmsd=0x55b4cae136c0 <qxl_vmstate>, opaque=0x55b4cd47c000)
    at /usr/src/debug/qemu-1.5.3/savevm.c:1950
#6  0x000055b4ca97581b in qemu_savevm_state_complete (se=0x55b4cce5c2c0, f=0x55b4d5e0a000) at /usr/src/debug/qemu-1.5.3/savevm.c:1997
#7  0x000055b4ca97581b in qemu_savevm_state_complete (f=f@entry=0x55b4d5e0a000) at /usr/src/debug/qemu-1.5.3/savevm.c:2164
#8  0x000055b4ca976538 in do_savevm (f=0x55b4d5e0a000) at /usr/src/debug/qemu-1.5.3/savevm.c:2226
#9  0x000055b4ca976538 in do_savevm (mon=0x55b4ccde2300, qdict=<optimized out>) at /usr/src/debug/qemu-1.5.3/savevm.c:2664
#10 0x000055b4ca9712a9 in handle_user_command (mon=mon@entry=0x55b4ccde2300, cmdline=<optimized out>)
    at /usr/src/debug/qemu-1.5.3/monitor.c:4010
#11 0x000055b4ca971577 in monitor_command_cb (mon=0x55b4ccde2300, cmdline=<optimized out>, opaque=<optimized out>)
    at /usr/src/debug/qemu-1.5.3/monitor.c:4626
#12 0x000055b4ca8e7034 in readline_handle_byte (rs=0x55b4cd2f4000, ch=<optimized out>) at readline.c:374
#13 0x000055b4ca971504 in monitor_read (opaque=<optimized out>, buf=<optimized out>, size=<optimized out>)
    at /usr/src/debug/qemu-1.5.3/monitor.c:4612
#14 0x000055b4ca8d480b in fd_chr_read (len=<optimized out>, buf=0x7ffe5d9f5f20 "\rE\376\034\262\177", s=0x55b4cce40000)
    at qemu-char.c:167
#15 0x000055b4ca8d480b in fd_chr_read (chan=<optimized out>, cond=<optimized out>, opaque=0x55b4cce40000) at qemu-char.c:850
#16 0x00007fb21c8cc8f9 in g_main_context_dispatch () at /lib64/libglib-2.0.so.0
#17 0x000055b4ca8a6d4a in main_loop_wait () at main-loop.c:187
#18 0x000055b4ca8a6d4a in main_loop_wait (timeout=<optimized out>) at main-loop.c:235
#19 0x000055b4ca8a6d4a in main_loop_wait (nonblocking=<optimized out>) at main-loop.c:475
#20 0x000055b4ca7bc890 in main () at vl.c:1995
#21 0x000055b4ca7bc890 in main (argc=<optimized out>, argv=<optimized out>, envp=<optimized out>) at vl.c:4361


--------------------------------

However,

qemu-kvm-1.5.3-153.el7.x86_64 + qxl-wddm-dod-0.18 + savevm -----> PASS
qemu-kvm-1.5.3-154.el7.x86_64 + qxl-wddm-dod-0.18 + savevm -----> PASS
qemu-kvm-1.5.3-154.el7.x86_64 + qxl-wddm-dod-0.16 + savevm -----> FAIL

David, could you please comment here? With old driver installed, the original issue is still reproducible with latest QEMU.

I had 0.15 before.. I am unable to reproduce on 0.18. Haven't tried 0.16. 

As for update I installed another new system and installed spice-guest-tools from https://www.spice-space.org. Driver 0.18 was installed as a dependency and no issue was seen after update to latest windows.

(In reply to Chao Yang from comment #26)
> I could reproduce this issue by sending savevm to QEMU.

OK, so can you confirm it was fixed in migration but just broken in savevm?

Does :
  https://brewweb.engineering.redhat.com/brew/taskinfo?taskID=15113349

fix it in savevm as well?

(I forgot the old qemu-kvm doesnt' check the dynamic migration blockers in snapshot).

Dave

> Packages I tested:
> qemu-kvm-1.5.3-153.el7.x86_64
> spice-qxl-wddm-dod-0.16-Signed.zip
> 
> 
> # /usr/libexec/qemu-kvm -name avocado-vt-vm1 -sandbox off -machine pc
> -nodefaults -vga qxl -monitor stdioe
> ich9-usb-ehci1,id=usb1,addr=0x1d.7,multifunction=on,bus=pci.0 -device
> ich9-usb-uhci1,id=usb1.0,multifunction=on,masterbus=usb1.0,addr=0x1d.0,
> firstport=0,bus=pci.0 -device
> ich9-usb-uhci2,id=usb1.1,multifunction=on,masterbus=usb1.0,addr=0x1d.2,
> firstport=2,bus=pci.0 -device
> ich9-usb-uhci3,id=usb1.2,multifunction=on,masterbus=usb1.0,addr=0x1d.4,
> firstport=4,bus=pci.0 -drive
> id=drive_image1,if=none,snapshot=off,aio=native,cache=none,format=qcow2,
> file=/home/chayang/win10.qcow2 -device
> virtio-blk-pci,id=image1,drive=drive_image1,bootindex=0,bus=pci.0,addr=0x4
> -device
> virtio-net-pci,mac=9a:5e:5f:60:61:62,id=idrScAZ6,vectors=4,netdev=idFczgUm,
> bus=pci.0,addr=0x5 -netdev tap,id=idFczgUm,vhost=on -m 4096 -smp
> 4,cores=2,threads=1,sockets=2 -cpu Penryn,+fsgsbase -device
> usb-tablet,id=usb-tablet1,bus=usb1.0,port=1 -spice
> disable-ticketing,port=5000 -rtc base=localtime,clock=host,driftfix=slew
> -boot order=cdn,once=c,menu=on,strict=off -enable-kvm 
>  QEMU 1.5.3 monitor - type 'help' for more information
>  (qemu) main_channel_link: add main channel client
>  main_channel_client_handle_pong: net test: latency 1.807000 ms, bitrate
> 95616041 bps (91.186563 Mbps)
>  inputs_connect: inputs channel client create
>  red_qxl_set_cursor_peer: 
> 
>  (qemu) 
>  (qemu) 
>  (qemu) savevm 
>  qemu-kvm: /builddir/build/BUILD/qemu-1.5.3/hw/display/qxl.c:2183:
> qxl_pre_save: Assertion `d->last_release_offset < d->vga.vram_size' failed.
>  Aborted (core dumped)
> 
> (gdb) bt
> #0  0x00007fb213cde1d7 in raise () at /lib64/libc.so.6
> #1  0x00007fb213cdf8d0 in abort () at /lib64/libc.so.6
> #2  0x00007fb213cd6fcc in __assert_fail_base () at /lib64/libc.so.6
> #3  0x00007fb213cd7088 in  () at /lib64/libc.so.6
> #4  0x000055b4ca936abc in qxl_pre_save (opaque=<optimized out>) at
> /usr/src/debug/qemu-1.5.3/hw/display/qxl.c:2183
> #5  0x000055b4ca97521d in vmstate_save_state (f=f@entry=0x55b4d5e0a000,
> vmsd=0x55b4cae136c0 <qxl_vmstate>, opaque=0x55b4cd47c000)
>     at /usr/src/debug/qemu-1.5.3/savevm.c:1950
> #6  0x000055b4ca97581b in qemu_savevm_state_complete (se=0x55b4cce5c2c0,
> f=0x55b4d5e0a000) at /usr/src/debug/qemu-1.5.3/savevm.c:1997
> #7  0x000055b4ca97581b in qemu_savevm_state_complete
> (f=f@entry=0x55b4d5e0a000) at /usr/src/debug/qemu-1.5.3/savevm.c:2164
> #8  0x000055b4ca976538 in do_savevm (f=0x55b4d5e0a000) at
> /usr/src/debug/qemu-1.5.3/savevm.c:2226
> #9  0x000055b4ca976538 in do_savevm (mon=0x55b4ccde2300, qdict=<optimized
> out>) at /usr/src/debug/qemu-1.5.3/savevm.c:2664
> #10 0x000055b4ca9712a9 in handle_user_command (mon=mon@entry=0x55b4ccde2300,
> cmdline=<optimized out>)
>     at /usr/src/debug/qemu-1.5.3/monitor.c:4010
> #11 0x000055b4ca971577 in monitor_command_cb (mon=0x55b4ccde2300,
> cmdline=<optimized out>, opaque=<optimized out>)
>     at /usr/src/debug/qemu-1.5.3/monitor.c:4626
> #12 0x000055b4ca8e7034 in readline_handle_byte (rs=0x55b4cd2f4000,
> ch=<optimized out>) at readline.c:374
> #13 0x000055b4ca971504 in monitor_read (opaque=<optimized out>,
> buf=<optimized out>, size=<optimized out>)
>     at /usr/src/debug/qemu-1.5.3/monitor.c:4612
> #14 0x000055b4ca8d480b in fd_chr_read (len=<optimized out>,
> buf=0x7ffe5d9f5f20 "\rE\376\034\262\177", s=0x55b4cce40000)
>     at qemu-char.c:167
> #15 0x000055b4ca8d480b in fd_chr_read (chan=<optimized out>, cond=<optimized
> out>, opaque=0x55b4cce40000) at qemu-char.c:850
> #16 0x00007fb21c8cc8f9 in g_main_context_dispatch () at
> /lib64/libglib-2.0.so.0
> #17 0x000055b4ca8a6d4a in main_loop_wait () at main-loop.c:187
> #18 0x000055b4ca8a6d4a in main_loop_wait (timeout=<optimized out>) at
> main-loop.c:235
> #19 0x000055b4ca8a6d4a in main_loop_wait (nonblocking=<optimized out>) at
> main-loop.c:475
> #20 0x000055b4ca7bc890 in main () at vl.c:1995
> #21 0x000055b4ca7bc890 in main (argc=<optimized out>, argv=<optimized out>,
> envp=<optimized out>) at vl.c:4361
> 
> 
> --------------------------------
> 
> However,
> 
> qemu-kvm-1.5.3-153.el7.x86_64 + qxl-wddm-dod-0.18 + savevm -----> PASS
> qemu-kvm-1.5.3-154.el7.x86_64 + qxl-wddm-dod-0.18 + savevm -----> PASS
> qemu-kvm-1.5.3-154.el7.x86_64 + qxl-wddm-dod-0.16 + savevm -----> FAIL
> 
> David, could you please comment here? With old driver installed, the
> original issue is still reproducible with latest QEMU.

ROll this back around to post the missing case.

(In reply to Dr. David Alan Gilbert from comment #28)
> (In reply to Chao Yang from comment #26)
> > I could reproduce this issue by sending savevm to QEMU.
> 
> OK, so can you confirm it was fixed in migration but just broken in savevm?
> 

Confirmed. It was fixed in migration.

> Does :
>   https://brewweb.engineering.redhat.com/brew/taskinfo?taskID=15113349
> 
> fix it in savevm as well?
> 

Yes. savevm was blocked by: 

qxl: guest bug: command not in ram bar


> (I forgot the old qemu-kvm doesnt' check the dynamic migration blockers in
> snapshot).
> 
> Dave
>

Fix included in qemu-kvm-1.5.3-155.el7

Verified pass.

with:
qemu-kvm-1.5.3-155.el7.x86_64 + qxl-wddm-dod-0.16 

savevm/migration now is blocked by: qxl: guest bug: command not in ram bar


with
qemu-kvm-1.5.3-155.el7.x86_64 + qxl-wddm-dod-0.18 

savevm/migration works well.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2018:0816