Bug 868216

Summary: core dump when migration to compressed file located on read-only/NO_ENOUGH location
Product: Red Hat Enterprise Linux 6 Reporter: Sibiao Luo <sluo>
Component: qemu-kvmAssignee: Juan Quintela <quintela>
Status: CLOSED WORKSFORME QA Contact: Virtualization Bugs <virt-bugs>
Severity: medium Docs Contact:
Priority: medium    
Version: 6.4CC: acathrow, areis, bsarathy, chayang, cwei, dyuan, flang, juzhang, lnovich, michen, mkenneth, mzhan, owasserm, qzhang, shuang, shu, sluo, virt-maint, weizhan, zhpeng
Target Milestone: rcKeywords: Regression
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2013-09-04 19:13:08 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 896690, 960054    

Description Sibiao Luo 2012-10-19 09:49:49 UTC 
Description of problem:
core dump when migration to compressed file located on read-only nfs server.

Version-Release number of selected component (if applicable):
# uname -r && rpm -q qemu-kvm
2.6.32-331.el6.x86_64
qemu-kvm-0.12.1.2-2.327.el6.x86_64
# rpm -q seabios
seabios-0.6.1.2-25.el6.x86_64
# rpm -q spice-server
spice-server-0.12.0-1.el6.x86_64
guest info:
# uname -r
2.6.32-331.el6.x86_64

How reproducible:
100%

Steps to Reproduce:
1.prepare a readonly nfs server 
# cat /etc/exports 
/home *(ro,no_root_squash,sync)
2.mount readonly nfs server to local
# mount <ip>:/home /mnt
3.start guest on src host.
eg: # /usr/libexec/qemu-kvm -M rhel6.4.0 -cpu SandyBridge -enable-kvm -m 4096 -smp 4,sockets=2,cores=2,threads=1 -usb -device usb-tablet,id=input0 -name sluo_migration -uuid 990ea161-6b67-47b2-b803-19fb01d30d30 -rtc base=localtime,clock=host,driftfix=slew -device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x3 -drive file=/dev/vg-90.100-sluo/lv-90.100-migration-macvtap,if=none,id=drive-virtio-disk0,format=qcow2,cache=none,werror=stop,rerror=stop -device virtio-blk-pci,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -netdev tap,id=hostnet0,vhost=on,fd=6 6<>/dev/tap6 -device virtio-net-pci,netdev=hostnet0,id=virtio-net-pci0,mac=08:2E:5F:0A:0D:B1,bus=pci.0,addr=0x5 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -chardev spicevmc,id=charchannel0,name=vdagent -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=com.redhat.spice.0 -spice port=5931,disable-ticketing,seamless-migration=on -vga qxl -global qxl-vga.vram_size=67108864 -device intel-hda,id=sound0,bus=pci.0,addr=0x6 -device hda-duplex -device usb-ehci,id=ehci,addr=0x7 -chardev spicevmc,name=usbredir,id=usbredirchardev1 -device usb-redir,chardev=usbredirchardev1,id=usbredirdev1,bus=ehci.0,debug=3 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x8 -drive file=/dev/vg-90.100-sluo/lv-90.100-data-disk,if=none,id=data-disk,format=qcow2,cache=none,werror=stop,rerror=stop -device virtio-blk-pci,bus=pci.0,addr=0x9,drive=data-disk,id=sluo-disk -nodefaults -serial unix:/tmp/ttyS0,server,nowait -qmp tcp:0:4444,server,nowait -boot menu=on -monitor stdio
4.migration to compressed file located on the read-only nfs server.
(qemu) __com.redhat_spice_migrate_info 10.66.11.229 5931
main_channel_migrate_src_complete: 
main_channel_client_handle_migrate_connected: client 0x7ffff975df80 connected: 1 seamless 1
(qemu) migrate -d "exec:gzip -c > /mnt/sluo.gz"
  
Actual results:
after step 4, migration failed with prompts that "/mnt/sluo.gz: Read-only file system", but core dump. I will paste the core dump log later. 

Expected results:
migration should failed should prompts sth like "no permission to write"

Additional info:
I will check the old qemu-kvm version whether is regression issue.
Comment 1 Sibiao Luo 2012-10-19 09:51:26 UTC 
(In reply to comment #0)
>   
> Actual results:
> after step 4, migration failed with prompts that "/mnt/sluo.gz: Read-only
> file system", but core dump. I will paste the core dump log later. 
> 
(qemu) migrate -d "exec:gzip -c > /mnt/sluo.gz"
Detaching after fork from child process 10440.
sh: /mnt/sluo.gz: Read-only file system

Program received signal SIGPIPE, Broken pipe.
0x00007ffff77474ed in write () from /lib64/libpthread.so.0

(gdb) bt
#0  0x00007ffff77474ed in write () from /lib64/libpthread.so.0
#1  0x00007ffff7e6e46c in file_write (s=<value optimized out>, buf=<value optimized out>, size=<value optimized out>) at migration-exec.c:41
#2  0x00007ffff7e5ef8c in migrate_fd_put_buffer (opaque=0x7ffff9d7dec0, data=0x7ffff9fe4f60, size=32768) at migration.c:355
#3  0x00007ffff7e5ed5d in buffered_put_buffer (opaque=0x7ffff9aaad30, buf=0x7ffff9fe4f60 "QEVM", pos=0, size=32768) at buffered_file.c:136
#4  0x00007ffff7e658f8 in qemu_fflush (f=0x7ffff9fe4f10) at savevm.c:446
#5  0x00007ffff7e67477 in qemu_put_buffer (f=0x7ffff9fe4f10, buf=0x7ffed3e0ce90 "", size=368) at savevm.c:519
#6  0x00007ffff7de4a38 in ram_save_block (f=0x7ffff9fe4f10) at /usr/src/debug/qemu-kvm-0.12.1.2/vl.c:2820
#7  0x00007ffff7de4b5c in ram_save_live (mon=<value optimized out>, f=0x7ffff9fe4f10, stage=1, opaque=<value optimized out>) at /usr/src/debug/qemu-kvm-0.12.1.2/vl.c:2927
#8  0x00007ffff7e6764b in qemu_savevm_state_begin (mon=0x0, f=0x7ffff9fe4f10, blk_enable=<value optimized out>, shared=<value optimized out>) at savevm.c:1524
#9  0x00007ffff7e5f39f in migrate_fd_connect (s=0x7ffff9d7dec0) at migration.c:380
#10 0x00007ffff7e6e416 in exec_start_outgoing_migration (mon=0x7ffff8a6a940, command=<value optimized out>, bandwidth_limit=33554432, detach=1, blk=0, inc=0)
    at migration-exec.c:109
#11 0x00007ffff7e5f8ea in do_migrate (mon=0x7ffff8a6a940, qdict=<value optimized out>, ret_data=<value optimized out>) at migration.c:118
#12 0x00007ffff7decbc0 in monitor_call_handler (mon=0x7ffff8a6a940, cmd=0x7ffff82bf180, params=<value optimized out>) at /usr/src/debug/qemu-kvm-0.12.1.2/monitor.c:4206
#13 0x00007ffff7df1f0f in handle_user_command (mon=0x7ffff8a6a940, cmdline=<value optimized out>) at /usr/src/debug/qemu-kvm-0.12.1.2/monitor.c:4243
#14 0x00007ffff7df204a in monitor_command_cb (mon=0x7ffff8a6a940, cmdline=<value optimized out>, opaque=<value optimized out>)
    at /usr/src/debug/qemu-kvm-0.12.1.2/monitor.c:4877
#15 0x00007ffff7e4c72d in readline_handle_byte (rs=0x7ffff9e9e2c0, ch=<value optimized out>) at readline.c:369
#16 0x00007ffff7df2270 in monitor_read (opaque=<value optimized out>, buf=0x7fffffffb760 "\r", size=1) at /usr/src/debug/qemu-kvm-0.12.1.2/monitor.c:4863
#17 0x00007ffff7e627bb in qemu_chr_read (opaque=0x7ffff86da770) at qemu-char.c:180
#18 fd_chr_read (opaque=0x7ffff86da770) at qemu-char.c:688
#19 0x00007ffff7de540f in main_loop_wait (timeout=1000) at /usr/src/debug/qemu-kvm-0.12.1.2/vl.c:3994
#20 0x00007ffff7e071ba in kvm_main_loop () at /usr/src/debug/qemu-kvm-0.12.1.2/qemu-kvm.c:2244
#21 0x00007ffff7de7f65 in main_loop (argc=20, argv=<value optimized out>, envp=<value optimized out>) at /usr/src/debug/qemu-kvm-0.12.1.2/vl.c:4206
#22 main (argc=20, argv=<value optimized out>, envp=<value optimized out>) at /usr/src/debug/qemu-kvm-0.12.1.2/vl.c:6443
(gdb)
Comment 3 Sibiao Luo 2012-10-19 11:27:02 UTC 
(In reply to comment #0)
>
> Additional info:
> I will check the old qemu-kvm version whether is regression issue.

I check it on qemu-kvm-0.12.1.2-2.295.el6.x86_64, it also hit this issue. so this isn't a regression bug.

(qemu) migrate -d "exec:gzip -c > /mnt/sluo.gz"
Detaching after fork from child process 5320.
(qemu) sh: /mnt/sluo.gz: Read-only file system

Program received signal SIGPIPE, Broken pipe.
0x00007ffff77514ed in write () from /lib64/libpthread.so.0
(gdb) bt
#0  0x00007ffff77514ed in write () from /lib64/libpthread.so.0
#1  0x00007ffff7e7187c in file_write (s=<value optimized out>, buf=<value optimized out>, size=<value optimized out>) at migration-exec.c:41
#2  0x00007ffff7e62c5c in migrate_fd_put_buffer (opaque=0x7ffff8b35930, data=0x7ffffc4bccb0, size=32768) at migration.c:346
#3  0x00007ffff7e6281a in buffered_flush (s=0x7ffff89b9630) at buffered_file.c:88
#4  0x00007ffff7e629bf in buffered_put_buffer (opaque=0x7ffff89b9630, buf=0x0, pos=0, size=0) at buffered_file.c:128
#5  0x00007ffff7e62eee in migrate_fd_put_notify (opaque=0x7ffff8b35930) at migration.c:330
#6  0x00007ffff7dec336 in main_loop_wait (timeout=1000) at /usr/src/debug/qemu-kvm-0.12.1.2/vl.c:3995
#7  0x00007ffff7e0da4a in kvm_main_loop () at /usr/src/debug/qemu-kvm-0.12.1.2/qemu-kvm.c:2244
#8  0x00007ffff7deecec in main_loop (argc=20, argv=<value optimized out>, envp=<value optimized out>) at /usr/src/debug/qemu-kvm-0.12.1.2/vl.c:4202
#9  main (argc=20, argv=<value optimized out>, envp=<value optimized out>) at /usr/src/debug/qemu-kvm-0.12.1.2/vl.c:6427
(gdb)
Comment 5 Qunfang Zhang 2012-10-22 08:13:02 UTC 
Also hit the issue when migrate guest to compressed file located on NO Enough Space disk.

gzip: stdout: No space left on device

Program received signal SIGPIPE, Broken pipe.
0x00007ffff77474ed in write () from /lib64/libpthread.so.0
(gdb) bt
#0  0x00007ffff77474ed in write () from /lib64/libpthread.so.0
#1  0x00007ffff7e6e46c in file_write (s=<value optimized out>, buf=<value optimized out>, 
    size=<value optimized out>) at migration-exec.c:41
#2  0x00007ffff7e5ef8c in migrate_fd_put_buffer (opaque=0x7ffff911f3c0, data=
    0x7ffff8783400, size=32768) at migration.c:355
#3  0x00007ffff7e5eb4a in buffered_flush (s=0x7ffff89d1f40) at buffered_file.c:88
#4  0x00007ffff7e5ecef in buffered_put_buffer (opaque=0x7ffff89d1f40, buf=0x0, pos=0, 
    size=0) at buffered_file.c:128
#5  0x00007ffff7e5f21e in migrate_fd_put_notify (opaque=0x7ffff911f3c0) at migration.c:339
#6  0x00007ffff7de55a6 in main_loop_wait (timeout=1000)
    at /usr/src/debug/qemu-kvm-0.12.1.2/vl.c:3999
#7  0x00007ffff7e071ba in kvm_main_loop ()
    at /usr/src/debug/qemu-kvm-0.12.1.2/qemu-kvm.c:2244
#8  0x00007ffff7de7f65 in main_loop (argc=20, argv=<value optimized out>, 
    envp=<value optimized out>) at /usr/src/debug/qemu-kvm-0.12.1.2/vl.c:4206
#9  main (argc=20, argv=<value optimized out>, envp=<value optimized out>)
    at /usr/src/debug/qemu-kvm-0.12.1.2/vl.c:6443
(gdb)
Comment 6 Sibiao Luo 2012-10-24 07:12:27 UTC 
Hi all,

   I tried the rhel6.3GA host with the same steps as comments 0 without any core dump. migration failed with prompting like "Read-only file system" when migration to compressed file located on read-only nfs server, that's just we expected.

host info:
# uname -r && rpm -q qemu-kvm
2.6.32-279.el6.x86_64
qemu-kvm-0.12.1.2-2.295.el6.x86_64

(qemu) migrate -d "exec:gzip -c > /tmp/zxc/sluo.gz"
sh: /tmp/zxc/sluo.gz: Read-only file system
spice_server_migrate_end: 
reds_mig_finished: 
reds_mig_finished: no peer connected
spice_server_migrate_end: 
reds_mig_finished: 
reds_mig_finished: no peer connected
(qemu) 
(qemu) info migrate 
Migration status: failed

   Base on above, this is regression bug, we'd better fix it in rhel6.4 version.

Best Regards.
sluo
Comment 7 juzhang 2012-10-24 07:29:11 UTC 
According to comment6, add keyword "Regression". From KVM QE pov, suggest to fix this issue on rhel6.4.
Comment 10 Juan Quintela 2013-05-20 17:29:16 UTC 
Upstream just don't detect the error.

Will be fixed in both places.

Thanks for the report.
Comment 17 Juan Quintela 2013-09-03 14:51:58 UTC 
See comment #6.

The problem here is gdb.  If you run qemu without gdb, I am not able to reproduce it.  Problem only happens when run inside gdb.  Launching outside of gdb, I get the same value that #6.

Could you reproduce without gdb?

Otherwise I would close this as no error (from qemu prespective).

I have tested, and you can reproduce the behaviour with a simple program.
Comment 18 Sibiao Luo 2013-09-04 05:10:30 UTC 
(In reply to Juan Quintela from comment #17)
> See comment #6.
> 
> The problem here is gdb.  If you run qemu without gdb, I am not able to
> reproduce it.  Problem only happens when run inside gdb.  Launching outside
> of gdb, I get the same value that #6.
maybe i cann't remember it if it with gdb or not.
> Could you reproduce without gdb?
no, i also cann't reproduce it without gdb, only hit it with gdb now.
> Otherwise I would close this as no error (from qemu prespective).
> 
> I have tested, and you can reproduce the behaviour with a simple program.
- without gdb:
1.readonly location.
(qemu) migrate -d "exec:gzip -c > /mnt/sluo.gz"
(qemu) sh: /mnt/sluo.gz: Read-only file system

2.no_enough location.
(qemu) migrate -d "exec:gzip -c > /mnt/sluo.gz"

gzip: stdout: No space left on device

- with gdb:
1.readonly location:
(qemu) migrate -d "exec:gzip -c > /mnt/sluo.gz"
Detaching after fork from child process 7787.
(qemu) sh: /mnt/sluo.gz: Read-only file system

Program received signal SIGPIPE, Broken pipe.
0x00007ffff77096fd in write () from /lib64/libpthread.so.0

(gdb) bt
#0  0x00007ffff77096fd in write () from /lib64/libpthread.so.0
#1  0x00007ffff7e607bc in file_write (s=<value optimized out>, buf=<value optimized out>, size=<value optimized out>)
    at /usr/src/debug/qemu-kvm-0.12.1.2/migration-exec.c:42
#2  0x00007ffff7e4fa6c in migrate_fd_put_buffer (opaque=0x7ffff8a5d590, data=0x7ffff94d7a80, size=32768)
    at /usr/src/debug/qemu-kvm-0.12.1.2/migration.c:389
#3  0x00007ffff7e4f62a in buffered_flush (s=0x7ffff8a5d720) at /usr/src/debug/qemu-kvm-0.12.1.2/buffered_file.c:88
#4  0x00007ffff7e4f7cf in buffered_put_buffer (opaque=0x7ffff8a5d720, buf=0x0, pos=0, size=0)
    at /usr/src/debug/qemu-kvm-0.12.1.2/buffered_file.c:128
#5  0x00007ffff7e4fd3e in migrate_fd_put_notify (opaque=0x7ffff8a5d590)
    at /usr/src/debug/qemu-kvm-0.12.1.2/migration.c:373
#6  0x00007ffff7dc9e86 in main_loop_wait (timeout=1000) at /usr/src/debug/qemu-kvm-0.12.1.2/vl.c:4058
#7  0x00007ffff7decd3a in kvm_main_loop () at /usr/src/debug/qemu-kvm-0.12.1.2/qemu-kvm.c:2245
#8  0x00007ffff7dcccf9 in main_loop (argc=69, argv=<value optimized out>, envp=<value optimized out>)
    at /usr/src/debug/qemu-kvm-0.12.1.2/vl.c:4266
#9  main (argc=69, argv=<value optimized out>, envp=<value optimized out>) at /usr/src/debug/qemu-kvm-0.12.1.2/vl.c:6644
(gdb) 

2.no_enough location.
(qemu) 
gzip: stdout: No space left on device

Program received signal SIGPIPE, Broken pipe.
0x00007ffff77096fd in write () from /lib64/libpthread.so.0
(gdb) bt
#0  0x00007ffff77096fd in write () from /lib64/libpthread.so.0
#1  0x00007ffff7e607bc in file_write (s=<value optimized out>, buf=<value optimized out>, size=<value optimized out>)
    at /usr/src/debug/qemu-kvm-0.12.1.2/migration-exec.c:42
#2  0x00007ffff7e4fa6c in migrate_fd_put_buffer (opaque=0x7ffff8a5d590, data=0x7ffff94afa60, size=32768)
    at /usr/src/debug/qemu-kvm-0.12.1.2/migration.c:389
#3  0x00007ffff7e4f83d in buffered_put_buffer (opaque=0x7ffff8a5d720, buf=0x7ffff94afa60 "", pos=229376, size=32768)
    at /usr/src/debug/qemu-kvm-0.12.1.2/buffered_file.c:136
#4  0x00007ffff7e57b58 in qemu_fflush (f=0x7ffff94afa10) at /usr/src/debug/qemu-kvm-0.12.1.2/savevm.c:446
#5  0x00007ffff7e58548 in qemu_put_be32 (f=0x7ffff94afa10, v=0) at /usr/src/debug/qemu-kvm-0.12.1.2/savevm.c:680
#6  0x00007ffff7e585ad in qemu_put_be64 (f=0x7ffff94afa10, v=6918178) at /usr/src/debug/qemu-kvm-0.12.1.2/savevm.c:687
#7  0x00007ffff7dc92ba in ram_save_block (f=0x7ffff94afa10) at /usr/src/debug/qemu-kvm-0.12.1.2/vl.c:2792
#8  0x00007ffff7dc93ac in ram_save_live (mon=<value optimized out>, f=0x7ffff94afa10, stage=2, 
    opaque=<value optimized out>) at /usr/src/debug/qemu-kvm-0.12.1.2/vl.c:2914
#9  0x00007ffff7e5866e in qemu_savevm_state_iterate (mon=0x0, f=0x7ffff94afa10)
    at /usr/src/debug/qemu-kvm-0.12.1.2/savevm.c:1565
#10 0x00007ffff7e4fdbd in migrate_fd_put_ready (opaque=0x7ffff8a5d590)
    at /usr/src/debug/qemu-kvm-0.12.1.2/migration.c:436
#11 0x00007ffff7e4f8ef in buffered_put_buffer (opaque=0x7ffff8a5d720, buf=0x0, pos=0, size=0)
    at /usr/src/debug/qemu-kvm-0.12.1.2/buffered_file.c:165
#12 0x00007ffff7e4fd3e in migrate_fd_put_notify (opaque=0x7ffff8a5d590)
    at /usr/src/debug/qemu-kvm-0.12.1.2/migration.c:373
#13 0x00007ffff7dc9e86 in main_loop_wait (timeout=1000) at /usr/src/debug/qemu-kvm-0.12.1.2/vl.c:4058
#14 0x00007ffff7decd3a in kvm_main_loop () at /usr/src/debug/qemu-kvm-0.12.1.2/qemu-kvm.c:2245
#15 0x00007ffff7dcccf9 in main_loop (argc=69, argv=<value optimized out>, envp=<value optimized out>)
    at /usr/src/debug/qemu-kvm-0.12.1.2/vl.c:4266
#16 main (argc=69, argv=<value optimized out>, envp=<value optimized out>) at /usr/src/debug/qemu-kvm-0.12.1.2/vl.c:6644
(gdb)

Best Regards,
sluo
Comment 19 Ademar Reis 2013-09-04 19:13:08 UTC 
(In reply to Sibiao Luo from comment #18)
> (In reply to Juan Quintela from comment #17)
> > See comment #6.
> > 
> > The problem here is gdb.  If you run qemu without gdb, I am not able to
> > reproduce it.  Problem only happens when run inside gdb.  Launching outside
> > of gdb, I get the same value that #6.
>
> maybe i cann't remember it if it with gdb or not.
> 
> > Could you reproduce without gdb?
> 
> no, i also cann't reproduce it without gdb, only hit it with gdb now.
> 
> > Otherwise I would close this as no error (from qemu prespective).

Closing.