Bug 1412183 - VM fails to access disk after migration
Summary: VM fails to access disk after migration
Keywords:
Status: CLOSED NOTABUG
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: qemu-kvm-rhev
Version: 7.4
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: rc
: ---
Assignee: Virtualization Maintenance
QA Contact: Virtualization Bugs
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-01-11 13:22 UTC by Andrei Stepanov
Modified: 2017-01-11 13:42 UTC (History)
5 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2017-01-11 13:42:37 UTC
Target Upstream Version:


Attachments (Terms of Use)

Description Andrei Stepanov 2017-01-11 13:22:33 UTC
After local TCP migration VM cannot work with its disk.

Host info:
kernel: 3.10.0-514.el7.x86_64
qemu-kvm-rhev-2.6.0-28.el7_3.3.x86_64


* Wed Jan 04 2017 Miroslav Rezanina <mrezanin@redhat.com> - rhev-2.6.0-28.el7_3.3
- kvm-pc_piix-fix-compat-props-typo-for-RHEL6-machine-type.patch [bz#1408122]
- kvm-net-don-t-poke-at-chardev-internal-QemuOpts.patch [bz#1410200]
- Resolves: bz#1408122
  (Opteron_G4 CPU model broken in QEMU 2.6 with RHEL 6 machine type)
- Resolves: bz#1410200
  (qemu gets SIGSEGV when hot-plug a vhostuser network)



How reproducible: always


Steps to Reproduce:
0. Disable firewall (if you have).
1. Have a qcow2 image with OS. (I have RHEL6.9 32bit)

2. Create first QEMU instance (Source) to migrate from:

export SRCPORT=3001
export GUEST_IMAGE_PATH=/local_storage/rhel6932.img

/usr/libexec/qemu-kvm -m 1024 -spice port=$SRCPORT,disable-ticketing,seamless-migration=on -vga qxl -global qxl-vga.vram_size=67108864 -device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x8 -chardev spicevmc,id=charchannel0,name=vdagent -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=com.redhat.spice.0 $GUEST_IMAGE_PATH -monitor stdio

3. Create secondary QEMU instance (Destination) to migrate to:

export DSTPORT=3001
export GUEST_IMAGE_PATH=/local_storage/rhel6932.img
export DSTMIGPORT=4444
export DSTIPADDR=0.0.0.0

/usr/libexec/qemu-kvm -m 1024 -spice port=$DSTPORT,disable-ticketing,seamless-migration=on -vga qxl -global qxl-vga.vram_size=67108864 -device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x8 -chardev spicevmc,id=charchannel0,name=vdagent -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=com.redhat.spice.0 $GUEST_IMAGE_PATH -monitor stdio -incoming tcp:$DSTIPADDR:$DSTMIGPORT

4. Connect with remote-viewer to VM (first QEMU instance):

remote-viewer spice://<IP>:3001

5. Migrate VM to the second VM instance:

(qemu) client_migrate_info spice <IP> 3002
(qemu) main_channel_client_handle_migrate_connected: client 0x7fa82fa71a80 connected: 1 seamless 1
(qemu) migrate -d tcp:127.0.0.1:4444
(qemu) red_client_migrate: migrate client with #channels 4
red_dispatcher_cursor_migrate: channel type 4 id 0
red_dispatcher_display_migrate: channel type 2 id 0
red_channel_client_disconnect: rcc=0x7fa8329fc000 (channel=0x7fa82fb42000 type=3 id=0)
red_channel_client_disconnect: rcc=0x7fa832466000 (channel=0x7fa8305e6600 type=2 id=0)
red_channel_client_disconnect: rcc=0x7fa8305f7000 (channel=0x7fa832006000 type=4 id=0)
red_channel_client_disconnect: rcc=0x7fa8305f2000 (channel=0x7fa82fb3a000 type=1 id=0)
main_channel_client_on_disconnect: rcc=0x7fa8305f2000
red_client_destroy: destroy client 0x7fa82fa71a80 with #channels=4
red_dispatcher_disconnect_cursor_peer: 
red_dispatcher_disconnect_display_peer: 


After this step VM fails to access its disk:

Log from VM:

ata1.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x6
ata1.00: BMDMA stat 0x5
ata1.00: failed command: WRITE DMA
ata1.00: cmd ca/00:38:b0:1e:c4/00:00:00:00:00/e1 tag 0 dma 28672 out
         res 00/00:00:00:00:00/00:00:00:00:00/00 Emask 0x2 (HSM violation)
ata1: soft resetting link
ata1.00: revalidation failed (errno=-2)
ata1: soft resetting link
ata1.00: revalidation failed (errno=-2)
ata1: soft resetting link
ata1.00: revalidation failed (errno=-2)
ata1.00: disabled
ata1: soft resetting link
ata1: EH complete
sd 0:0:0:0: [sda]  Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
sd 0:0:0:0: [sda] CDB: Write(10): 2a 00 01 c4 1e b0 00 00 38 00
end_request: I/O error, dev sda, sector 29630128
Aborting journal on device sda1-8.
EXT4-fs (sda1): delayed block allocation failed for inode 278698 at logical offset 0 with max blocks 8 with error -30
sd 0:0:0:0: [sda]  Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
sd 0:0:0:0: [sda] CDB: Write(10): 2a 00 01 c4 08 00 00 00 08 00
end_request: I/O error, dev sda, sector 29624320
Buffer I/O error on device sda1, logical block 3702784
lost page write due to I/O error on sda1

This should not happen!!  Data will be lost
EXT4-fs error (device sda1) in ext4_reserve_inode_write: Journal has aborted
JBD2: I/O error detected when updating journal superblock for sda1-8.
EXT4-fs error (device sda1): ext4_journal_start_sb: Detected aborted journal
EXT4-fs (sda1): Remounting filesystem read-only
EXT4-fs (sda1): ext4_da_writepages: jbd2_start: 1024 pages, ino 278698; err -30

sd 0:0:0:0: [sda]  Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
sd 0:0:0:0: [sda] CDB: Write(10): 2a 00 00 00 08 00 00 00 08 00
end_request: I/O error, dev sda, sector 2048
Buffer I/O error on device sda1, logical block 0
lost page write due to I/O error on sda1
EXT4-fs warning (device sda1): ext4_delete_inode: couldn't mark inode dirty (err -30)
sd 0:0:0:0: [sda]  Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
sd 0:0:0:0: [sda] CDB: Write(10): 2a 00 00 01 51 08 00 00 08 00
end_request: I/O error, dev sda, sector 86280
Buffer I/O error on device sda1, logical block 10529
lost page write due to I/O error on sda1
JBD2: Detected IO errors while flushing file data on sda1-8
sd 0:0:0:0: [sda]  Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
sd 0:0:0:0: [sda] CDB: Read(10): 28 00 01 44 63 d0 00 00 08 00
end_request: I/O error, dev sda, sector 21259216
sd 0:0:0:0: [sda]  Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
sd 0:0:0:0: [sda] CDB: Read(10): 28 00 01 44 63 d0 00 00 08 00
end_request: I/O error, dev sda, sector 21259216
sd 0:0:0:0: [sda]  Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
sd 0:0:0:0: [sda] CDB: Read(10): 28 00 03 1a 4c 40 00 00 10 00
end_request: I/O error, dev sda, sector 52055104
sd 0:0:0:0: [sda]  Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
sd 0:0:0:0: [sda] CDB: Read(10): 28 00 03 1a 4c 40 00 00 08 00
end_request: I/O error, dev sda, sector 52055104
sd 0:0:0:0: [sda]  Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
sd 0:0:0:0: [sda] CDB: Read(10): 28 00 03 1a 4c 40 00 00 08 00
end_request: I/O error, dev sda, sector 52055104
sd 0:0:0:0: [sda]  Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
sd 0:0:0:0: [sda] CDB: Read(10): 28 00 01 44 63 d0 00 00 08 00
end_request: I/O error, dev sda, sector 21259216
sd 0:0:0:0: [sda]  Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
sd 0:0:0:0: [sda] CDB: Read(10): 28 00 03 1a 4c 50 00 00 18 00
end_request: I/O error, dev sda, sector 52055120
sd 0:0:0:0: [sda]  Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
sd 0:0:0:0: [sda] CDB: Read(10): 28 00 03 1a 4c 50 00 00 08 00
end_request: I/O error, dev sda, sector 52055120
sd 0:0:0:0: [sda]  Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
sd 0:0:0:0: [sda] CDB: Read(10): 28 00 01 47 2a 98 00 00 08 00
end_request: I/O error, dev sda, sector 21441176
sd 0:0:0:0: [sda]  Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
sd 0:0:0:0: [sda] CDB: Read(10): 28 00 01 47 2a 98 00 00 08 00
end_request: I/O error, dev sda, sector 21441176
sd 0:0:0:0: [sda]  Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
sd 0:0:0:0: [sda] CDB: Read(10): 28 00 03 07 cc 00 00 00 20 00
end_request: I/O error, dev sda, sector 50842624
sd 0:0:0:0: [sda]  Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
sd 0:0:0:0: [sda] CDB: Read(10): 28 00 03 07 cc 00 00 00 08 00
end_request: I/O error, dev sda, sector 50842624
Core dump to |/usr/libexec/abrt-hook-ccpp 6 0 2339 0 0 1484136191 e pipe failed
sd 0:0:0:0: [sda]  Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
sd 0:0:0:0: [sda] CDB: Read(10): 28 00 01 44 63 d0 00 00 08 00
end_request: I/O error, dev sda, sector 21259216
sd 0:0:0:0: [sda]  Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
sd 0:0:0:0: [sda] CDB: Read(10): 28 00 01 47 2a 98 00 00 08 00
end_request: I/O error, dev sda, sector 21441176
sd 0:0:0:0: [sda]  Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
sd 0:0:0:0: [sda] CDB: Read(10): 28 00 03 07 cc 00 00 00 08 00
end_request: I/O error, dev sda, sector 50842624
Core dump to |/usr/libexec/abrt-hook-ccpp 6 0 2575 0 0 1484136780 e pipe failed

VM is broken and doesn't work as expected.

Comment 1 Andrei Stepanov 2017-01-11 13:42:37 UTC
Okay, it was my mistake. Secondary QEMU instance had bad cmdline.


Note You need to log in before you can comment on or make changes to this bug.