Fedora Account System
Red Hat Associate
Red Hat Customer
1. Start with an F19 host machine (may happen with any host, I don't know) 2. Follow Cole's instructions to create a VM with a UEFI firmware: https://fedoraproject.org/wiki/Testing_secureboot_with_KVM (you only need to go as far as "Install an F18 VM with UEFI", you don't need to do the SB stuff). Use the Fedora 19 Alpha TC2 desktop live image (probably would affect any image, though) 3. Re-configure the VM, using virt-manager, to use qxl and Spice instead of cirrus and VNC 4. Try and boot; when it hits X startup, the machine shuts down This could be in the UEFI firmware, qxl, SPICE...I really don't know.
the UEFI VM boots fine in the default cirrus/VNC configuration. a 'normal' - i.e. using seabios - VM boots fine using qxl/spice. It's just the combination of the UEFI firmware and qxl/spice that explodes.
At least with an F18 VM + UEFI, I don't seem to get any output from spice + qxl combo, but it doesn't seem to crash.
<ajax> adamw: in general anything the guest can do to crash qemu is a qemu bug re-assigning to qemu. The machine actually *powers off*, note, and it boots far enough that I see plymouth.
interestingly, the f18 desktop live image boots okay, no shutdown.
I've seen recent qemu versions crap themselves in spice code and abort when using qxl. Not a very helpful addition, perhaps, but I don't think it's just a "power off". Can you reproduce this when running qemu from the command line, and see output?
but that'd require me to learn how to use qemu!
Logs are at /var/log/libvirt/qemu/${vm-name}.log, anything enlightening in there? Any chance for a stacktrace in case it is a crash? How far comes it at boot? Crashes in UEFI still? Or after loading the guest kernel? Has F19 qxl kms already? If so, does nomodeset change behavior?
All I get in the log is "2013-04-02 16:10:36.846+0000: shutting down" . It crashes long after loading the guest kernel - as I said, it looks like it's about X transition time when it crashes. I'll try and run through qemu and see if I get a stack trace.
oh - no, nomodeset doesn't help.
I can't reproduce this. Host: - RHEL-6.4: - kernel: 2.6.32-358.0.1.el6.x86_64 - libvirt: 0.10.2-18.el6_4.2.x86_64 - qemu-kvm: 0.12.1.2-2.355.el6_4.2.x86_64 - OVMF: manual build at SVN r14242 Guest: - Fedora-19-Nightly-20130401.13-x86_64-Live-desktop.iso <http://koji.fedoraproject.org/koji/taskinfo?taskID=5195507> The OVMF log contains "QemuVideo: QEMU QXL VGA detected". qemu-kvm command line: /usr/libexec/qemu-kvm \ -name fw-ovmf.g-f19n2013040113.e-rhel63 \ -S \ -M rhel6.3.0 \ -enable-kvm \ -bios /root/OVMF.fd \ -m 1536 \ -smp 1,sockets=1,cores=1,threads=1 \ -uuid a43fb61d-3482-e51f-07b4-d96502c44a6a \ -nodefconfig \ -nodefaults \ -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/fw-ovmf.g-f19n2013040113.e-rhel63.monitor,server,nowait \ -mon chardev=charmonitor,id=monitor,mode=control \ -rtc base=utc \ -no-shutdown \ -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 \ -device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x6 \ -drive file=/var/lib/libvirt/images/fw-ovmf.g-f19n2013040113.e-rhel63.img,if=none,id=drive-virtio-disk0,format=qcow2,cache=none \ -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=2 \ -drive file=/filestore/isos/f19/Fedora-19-Nightly-20130401.13-x86_64-Live-desktop.iso,if=none,media=cdrom,id=drive-ide0-1-0,readonly=on,format=raw \ -device ide-drive,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0,bootindex=1 \ -netdev tap,fd=23,id=hostnet0 \ -device rtl8139,netdev=hostnet0,id=net0,mac=52:54:00:3b:9e:35,bus=pci.0,addr=0x3 \ -chardev pty,id=charserial0 \ -device isa-serial,chardev=charserial0,id=serial0 \ -chardev spicevmc,id=charchannel0,name=vdagent \ -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=com.redhat.spice.0 \ -spice port=5900,addr=127.0.0.1,disable-ticketing,seamless-migration=on \ -vga qxl \ -global qxl-vga.ram_size=67108864 \ -global qxl-vga.vram_size=67108864 \ -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5 \ -debugcon file:/tmp/fw-ovmf.g-f19n2013040113.e-rhel63.debug \ -global isa-debugcon.iobase=0x402 \ -global PIIX4_PM.disable_s3=0 \ -global PIIX4_PM.disable_s4=0 The LiveCD boots OK, I can start the installation on the GUI. I'm noticing two things: (1) According to the boot messages/ dmesg, I run into bug 928042. Apparently this switches to user mode setting, so host versions are important. (2) During installation to the hard drive (when selecting the target disk), I somehow managed to procuce the following display problem: the image of the guest's video output is cropped to a smallish rectangle (a few hundred pixels, squared).
On the same host I also changed a preexistent F18 guest to use qxl/spice (from cirrus/vnc). The UEFI GOP produces video output (OVMF setup, shim, grub), but the screen is not updated after grub starts the kernel.
(In reply to comment #11) > On the same host I also changed a preexistent F18 guest to use qxl/spice > (from cirrus/vnc). The UEFI GOP produces video output (OVMF setup, shim, > grub), but the screen is not updated after grub starts the kernel. (I did this only to confirm comment 2 -- the guest seems to work OK otherwise, for example it reacts to the shutdown button correctly.)
Created attachment 731211 [details] screenshot for comment 10 / (2)
laszlo: well, your steps are kinda quite different from mine. I'm not using a nightly but the official TC3 live build, and I think we're using different firmwares (you're using a personal build, I'm using the one from Cole's instructions).
(In reply to comment #14) > laszlo: well, your steps are kinda quite different from mine. I'm not using > a nightly but the official TC3 live build, I would have used an Alpha too if only I could have found one. Apparently my search-fu sucks as I couldn't locate any. > and I think we're using different > firmwares (you're using a personal build, I'm using the one from Cole's > instructions). I think that shouldn't make a difference; BIOS vs. OVMF (whichever build) should matter more. Since we're not using a CSM build ("OVMF-pure-efi.fd", as opposed to "OVMF-with-csm.fd" in Gerd's builds), X could at worst access UEFI runtime services provided by OVMF (it can't call any traditional BIOS function for video initialization etc). I think calling a runtime service (or a crash caused by that) is unlikely. Anyway, could you please setup a Beaker machine reproducing the problem? I dispose over no personal physical box to install F19 to. Thanks.
I have about zero experience using Beaker, but I guess I can try. It won't be #1 on my priority list though :/ TC/RC builds always live at https://dl.fedoraproject.org/pub/alt/stage/ .
[127460.681972] traps: qemu-system-x86[14049] trap divide error ip:7fee1dbb08f4 sp:7fee11952680 error:0 in libspice-server.so.1.6.0[7fee1db49000+10f000] [127460.684540] abrt-hook-ccpp[14105]: segfault at 0 ip (null) sp 00007fffd0a35238 error 14 in abrt-hook-ccpp[400000+5000] [127460.684561] Process 14105(abrt-hook-ccpp) has RLIMIT_CORE set to 1 [127460.684562] Aborting core if that helps at all. seems like the crash somehow chokes up abrt, which is why I'm not getting an abrt notification for it.
(In reply to comment #17) > [127460.681972] traps: qemu-system-x86[14049] trap divide error > ip:7fee1dbb08f4 sp:7fee11952680 error:0 in > libspice-server.so.1.6.0[7fee1db49000+10f000] This is printed by do_trap() [arch/x86/kernel/traps.c] and print_vma_addr() invoked by the former. For the VMA mapping "libspice-server.so.1.6.0" (from the build <http://koji.fedoraproject.org/koji/buildinfo?buildID=400800>): vma_start == 7fee1db49000 ip == 7fee1dbb08f4 == vma_start + 678f4 vma_end == 7fee1dc58000 == vma_start + 10f000 Offset 0x678f4 is in spice_server_port_event(): $ objdump -d usr/lib64/libspice-server.so.1.6.0 00000000000543c0 <spice_server_port_event>: 678f4: f7 3c b9 idivl (%rcx,%rdi,4) The offset into the function is 0x678f4 - 0x543c0 == 13534 Getting the source using "spice-debuginfo-0.12.2-4.fc19.x86_64.rpm": $ gdb usr/lib/debug/usr/lib64/libspice-server.so.1.6.0.debug (gdb) list *(spice_server_port_event+0x13534) 0x678f4 is in lz_encode (lz.c:516). 511 encoder->width = width; 512 encoder->height = height; 513 encoder->stride = stride; 514 515 if (IS_IMAGE_TYPE_PLT[encoder->type]) { 516 if (encoder->stride > (width / PLT_PIXELS_PER_BYTE[encoder->type])) { 517 if (((width % PLT_PIXELS_PER_BYTE[encoder->type]) == 0) || ( 518 (encoder->stride - (width / PLT_PIXELS_PER_BYTE[encoder->type])) > 1)) { 519 encoder->usr->error(encoder->usr, "stride overflows (plt)\n"); 520 } Looks like the (width / PLT_PIXELS_PER_BYTE[encoder->type]) division traps. The array is defined like this, in "spice-common/common/lz_common.h": static const int PLT_PIXELS_PER_BYTE[] = {0, 8, 8, 2, 2, 1}; So either "encoder->type" is 0 (LZ_IMAGE_TYPE_INVALID), or it is out-of-bounds and the random garbage it dereferences happens to be 0. Interestingly, lz_encode() does check for (encoder->type == LZ_IMAGE_TYPE_INVALID), but only after the invalid division above. "encoder->type" seems to come directly from the caller of lz_encode(). The only possible caller seems to be red_lz_compress_image() [server/red_worker.c], which keys "type" off MAP_BITMAP_FMT_TO_LZ_IMAGE_TYPE[src->format] From "server/red_common.h", static const LzImageType MAP_BITMAP_FMT_TO_LZ_IMAGE_TYPE[] = { LZ_IMAGE_TYPE_INVALID, LZ_IMAGE_TYPE_PLT1_LE, LZ_IMAGE_TYPE_PLT1_BE, LZ_IMAGE_TYPE_PLT4_LE, LZ_IMAGE_TYPE_PLT4_BE, LZ_IMAGE_TYPE_PLT8, LZ_IMAGE_TYPE_RGB16, LZ_IMAGE_TYPE_RGB24, LZ_IMAGE_TYPE_RGB32, LZ_IMAGE_TYPE_RGBA, LZ_IMAGE_TYPE_A8 }; (src->format == 0) would be consistent with this, but someone actually knowing spice will have to look into this.
Should the bug be re-assigned to SPICE? Excellent detective work, BTW.
Yes I think this should be reassigned to spice. The spice maintainers will know better. FWIW I tried to "meet in the middle", tracking backwards from red_lz_compress_image() and forward from spice_server_port_event(). I can't find the middle link. The "bottom" parts of the paths to lz_encode() depend on PIPE_ITEM_TYPE_DRAW and PIPE_ITEM_TYPE_IMAGE: display_channel_send_item() -- called via channel funcptr marshall_qxl_drawable() -- depends on PIPE_ITEM_TYPE_DRAW red_marshall_qxl_drawable() fill_mask() | red_display_marshall_upgrade() | red_marshall_qxl_draw_*() fill_bits() red_compress_image() red_lz_compress_image() lz_encode() display_channel_send_item() -- called via channel funcptr red_marshall_image() -- depends on PIPE_ITEM_TYPE_IMAGE red_lz_compress_image() lz_encode() But the "upper half" tries to send a completely different pipe item (not draw operation, also not an image, but a port event): spice_server_port_event() spicevmc_port_send_event() red_channel_pipe_item_init(PIPE_ITEM_TYPE_PORT_EVENT) red_channel_client_pipe_add_push() red_channel_client_pipe_add() red_channel_client_push() red_channel_client_send_item() call via "send_item" funcptr OTOH red_channel_client_pipe_add() seems to imply the presence of a queue. We tack a PORT_EVENT to the far end of the queue, and try to flush the near end. Which is where the image or drawable event might be. I think this needs live event tracing.
Re-assigning to spice, thanks laszlo.
*groan* I looked again at the objdump output. In short I had failed to realize that I had been looking at a shared library. I'd missed the entire point of the PLT. What it means is this: the spice_server_port_event() "anchor" in comment 20 is bogus. We do *not* enter from that function. The fact that the address 0x678f4 happens to fall under "spice_server_port_event" in the objdump output doesn't mean anything; it doesn't imply inlining in any way. IOW the "upper half", from spice_server_port_event() to red_channel_client_send_item(), is completely wrong. Sorry about the confusion. The lower half (display_channel_send_item() to lz_encode() -- including the div-by-0 --, on various paths), remains valid.
I may have found something: (In reply to comment #18) > 0x678f4 is in lz_encode (lz.c:516). > 511 encoder->width = width; > 512 encoder->height = height; > 513 encoder->stride = stride; > 514 > 515 if (IS_IMAGE_TYPE_PLT[encoder->type]) { > 516 if (encoder->stride > (width / > PLT_PIXELS_PER_BYTE[encoder->type])) { The division is only reached if IS_IMAGE_TYPE_PLT[encoder->type] evaluates to non-zero. IS_IMAGE_TYPE_PLT[encoder->type] -- checked before division PLT_PIXELS_PER_BYTE[encoder->type] -- used as divisor From "lz_common.h", typedef enum { LZ_IMAGE_TYPE_INVALID, LZ_IMAGE_TYPE_PLT1_LE, LZ_IMAGE_TYPE_PLT1_BE, // PLT stands for palette LZ_IMAGE_TYPE_PLT4_LE, LZ_IMAGE_TYPE_PLT4_BE, LZ_IMAGE_TYPE_PLT8, LZ_IMAGE_TYPE_RGB16, LZ_IMAGE_TYPE_RGB24, LZ_IMAGE_TYPE_RGB32, LZ_IMAGE_TYPE_RGBA, LZ_IMAGE_TYPE_XXXA, LZ_IMAGE_TYPE_A8 } LzImageType; ... /* access to the arrays is based on the image types */ static const int IS_IMAGE_TYPE_PLT[] = {0, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0}; static const int IS_IMAGE_TYPE_RGB[] = {0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1}; static const int PLT_PIXELS_PER_BYTE[] = {0, 8, 8, 2, 2, 1}; static const int RGB_BYTES_PER_PIXEL[] = {0, 1, 1, 1, 1, 1, 2, 3, 4, 4, 4, 1}; There are more enum constants (12) than elements in the IS_IMAGE_TYPE_PLT array (11). They should be in sync; for example when LZ_IMAGE_TYPE_XXXA was added, both IS_IMAGE_TYPE_* arrays were extended: http://cgit.freedesktop.org/spice/spice-common/commit/?id=9c906e0c However the next similar commit, adding LZ_IMAGE_TYPE_A8, http://cgit.freedesktop.org/spice/spice-common/commit/?id=c0b048eb Add support for A8 images to the LZ routines This format is needed to add Render support to the X driver, so we need the ability to compress and decompress it. apparently missed growing the IS_IMAGE_TYPE_* arrays. Hence in lz_encode(), probably - encoder->type == LZ_IMAGE_TYPE_A8, - IS_IMAGE_TYPE_PLT[encoder->type] invokes undefined behavior and happens to evaluate to nonzero, - PLT_PIXELS_PER_BYTE[encoder->type] is reached unexpectedly; it invokes undefined behavior too, evaluates to zero, and traps.
Created attachment 732059 [details] [1/1] supply missing IS_IMAGE_TYPE_* elements for LZ_IMAGE_TYPE_A8 The IS_IMAGE_TYPE_PLT and IS_IMAGE_TYPE_RGB arrays are supposed to decide whether each image type is PLT or RGB. Dependent on the result, one of the PLT_PIXELS_PER_BYTE and RGB_BYTES_PER_PIXEL arrays can be indexed. Commit c0b048eb introduced the LZ_IMAGE_TYPE_A8 enum constant and grew the RGB_BYTES_PER_PIXEL array by one element, but it missed to append a zero to IS_IMAGE_TYPE_PLT, and a one to IS_IMAGE_TYPE_RGB. Do so now. --- common/lz_common.h | 4 ++-- 1 files changed, 2 insertions(+), 2 deletions(-)
(In reply to comment #24) > Created attachment 732059 [details] > [1/1] supply missing IS_IMAGE_TYPE_* elements for LZ_IMAGE_TYPE_A8 Adam, can you please try to update the host-side spice package from <http://koji.fedoraproject.org/koji/taskinfo?taskID=5226716> before starting the VM? Thanks.
("spice-server-0.12.2-5.fc19.x86_64.rpm", that is)
Roger!
(Checking the arrays' contents with objdump --section=.rodata -s libspice-server.so.1.6.0 the offsets originate from the disassemblies, comment 18): - original build <http://koji.fedoraproject.org/koji/buildinfo?buildID=400800>: 678c7: 48 8d 05 f2 b1 07 00 lea 0x7b1f2(%rip),%rax # e2ac0 <spice_server_port_event+0x8e700> ... 678ec: 48 8d 0d 9d b1 07 00 lea 0x7b19d(%rip),%rcx # e2a90 <spice_server_port_event+0x8e6d0> 678f3: 99 cltd 678f4: f7 3c b9 idivl (%rcx,%rdi,4) static const int PLT_PIXELS_PER_BYTE[] = {0, 8, 8, 2, 2, 1}; e2a90 00000000 08000000 08000000 02000000 ................ e2aa0 02000000 01000000 00000000 00000000 ................ e2ab0 00000000 00000000 00000000 00000000 ................ ^^^^^^^^ corresponds to PLT_PIXELS_PER_BYTE[LZ_IMAGE_TYPE_A8], zero divisor static const int IS_IMAGE_TYPE_PLT[] = {0, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0}; e2ac0 00000000 01000000 01000000 01000000 ................ e2ad0 01000000 01000000 00000000 00000000 ................ e2ae0 00000000 00000000 00000000 6d617273 ............mars ^^^^^^^^ corresponds to IS_IMAGE_TYPE_PLT[LZ_IMAGE_TYPE_A8], nonzero e2af0 68616c6c 65722e63 006d2d3e 64617461 haller.c.m->data - patched build from comment 25 <http://koji.fedoraproject.org/koji/taskinfo?taskID=5226716>: 678d7: 48 8d 05 02 b2 07 00 lea 0x7b202(%rip),%rax # e2ae0 <spice_server_port_event+0x8e710> ... 678fc: 48 8d 0d ad b1 07 00 lea 0x7b1ad(%rip),%rcx # e2ab0 <spice_server_port_event+0x8e6e0> 67903: 99 cltd 67904: f7 3c b9 idivl (%rcx,%rdi,4) static const int PLT_PIXELS_PER_BYTE[] = {0, 8, 8, 2, 2, 1}; e2ab0 00000000 08000000 08000000 02000000 ................ e2ac0 02000000 01000000 00000000 00000000 ................ e2ad0 00000000 00000000 00000000 00000000 ................ ^^^^^^^^ unchanged, but access/division unreached static const int IS_IMAGE_TYPE_PLT[] = {0, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0}; e2ae0 00000000 01000000 01000000 01000000 ................ e2af0 01000000 01000000 00000000 00000000 ................ e2b00 00000000 00000000 00000000 00000000 ................ ^^^^^^^^ corresponds to IS_IMAGE_TYPE_PLT[LZ_IMAGE_TYPE_A8], zero e2b10 6d617273 68616c6c 65722e63 006d2d3e marshaller.c.m->
Hm - it still crashes, but the error is gone :( so maybe there are two problems? I'll try and poke it more.
Actually, you know what? The VM is actually running. It's virt-manager that crashes, now. If I run virt-manager and try to interact with that VM at all, virt-manager falls over - but nothing seems to catch the crash, there's nothing in the system logs or in ABRT's GUI, and if I run virt-manager from a console I don't see the error printed to the console. But if I run spicec and connect to localhost:5900, I can interact with the VM just fine. So it looks like that fix does indeed fix the problem in SPICE, and now we're onto some kind of virt-manager issue.
(In reply to comment #32) > The VM is actually running. Good to hear! > It's virt-manager that > crashes, now. If I run virt-manager and try to interact with that VM at all, > virt-manager falls over - but nothing seems to catch the crash, there's > nothing in the system logs or in ABRT's GUI, and if I run virt-manager from > a console I don't see the error printed to the console. > > But if I run spicec and connect to localhost:5900, I can interact with the > VM just fine. So it looks like that fix does indeed fix the problem in > SPICE, and now we're onto some kind of virt-manager issue. I will assume that you've installed "spicec" from the same koji build (comment 25), package "spice-client-0.12.2-5.fc19.x86_64.rpm". I think the problem is that "lz_common.h" *defines* (not only declares) the problematic array with internal linkage. All C files that include this header (directly or recursively) will get a copy, and all accesses to the element at offset LZ_IMAGE_TYPE_A8 are undefined behavior. If you've indeed installed spicec from the koji build, then it works because "lz_common.h" was patched for all sub-packages built from that SRPM. virt-viewer depends on shared libraries from the "spice-gtk" package <http://koji.fedoraproject.org/koji/buildinfo?buildID=402663>. Unfortunately the SRPM includes a separate instance of "lz_common.h". I'll try to build a spice-gtk package as well.
(In reply to comment #24) > Created attachment 732059 [details] > [1/1] supply missing IS_IMAGE_TYPE_* elements for LZ_IMAGE_TYPE_A8 > Oh nice catch, can you send this patch to the spice-devel mailing list? http://lists.freedesktop.org/mailman/listinfo/spice-devel We should try to add some compile time checks for the sizes of these various arrays to prevent this from happening again.
Adam, (In reply to comment #33) > I'll try to build a spice-gtk package as well. can you please install (freshen) the packages from <http://koji.fedoraproject.org/koji/taskinfo?taskID=5231613>, then restart libvirtd & virt-manager? Thanks.
I have no fedpkg push rights so I'll just attach the dist-git patches too.
Created attachment 733172 [details] dist git patch for the spice package
Created attachment 733173 [details] dist git patch for the spice-gtk package
(In reply to comment #34) > (In reply to comment #24) > > Created attachment 732059 [details] > > [1/1] supply missing IS_IMAGE_TYPE_* elements for LZ_IMAGE_TYPE_A8 > > > > Oh nice catch, can you send this patch to the spice-devel mailing list? > http://lists.freedesktop.org/mailman/listinfo/spice-devel > We should try to add some compile time checks for the sizes of these various > arrays to prevent this from happening again. http://lists.freedesktop.org/archives/spice-devel/2013-April/012999.html
"I will assume that you've installed "spicec" from the same koji build (comment 25), package "spice-client-0.12.2-5.fc19.x86_64.rpm"." That is correct. "can you please install (freshen) the packages from <http://koji.fedoraproject.org/koji/taskinfo?taskID=5231613>, then restart libvirtd & virt-manager? Thanks." Will test ASAP, thanks!
Indeed, that fix looks good - I can now boot an F19 live image in a qxl/SPICE UEFI guest all within virt-manager. Awesome job, thanks again.
*** Bug 953480 has been marked as a duplicate of this bug. ***
This is fixed in spice-server-0.12.3, which is part of F-19 GA, closing.