RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 602209 - Core dumped during Guest installation
Summary: Core dumped during Guest installation
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Red Hat Enterprise Linux 6
Classification: Red Hat
Component: qemu-kvm
Version: 6.0
Hardware: All
OS: Linux
medium
high
Target Milestone: rc
: ---
Assignee: Kevin Wolf
QA Contact: Virtualization Bugs
URL:
Whiteboard:
: 604221 (view as bug list)
Depends On:
Blocks: 618521
TreeView+ depends on / blocked
 
Reported: 2010-06-09 12:07 UTC by Mike Cao
Modified: 2013-01-09 22:42 UTC (History)
11 users (show)

Fixed In Version: qemu-kvm-0.12.1.2-2.97.el6
Doc Type: Bug Fix
Doc Text:
Clone Of:
: 618521 (view as bug list)
Environment:
Last Closed: 2010-07-27 07:31:48 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)
Screen dumped (124.69 KB, image/png)
2010-07-22 05:08 UTC, Mike Cao
no flags Details

Description Mike Cao 2010-06-09 12:07:29 UTC
Description of problem:

Core dumped during Guest installation
Version-Release number of selected component (if applicable):
qemu-kvm-0.12.1.2-2.71.el6.x86_64

How reproducible:
always

Steps to Reproduce:
1.dd if=/dev/zero of=/SCSI bs=1M count=10240
2.qemu-img create -f qcow2 /SCSI 10G
3.Add /SCSI target device.
eg:
dd if=/dev/zero of=/SCSI bs=1M count=10240
tgtadm --lld iscsi --op new --mode target --tid 1 T iqn.redhat.com:storage1
tgtadm --lld iscsi --op new --mode logicalunit --tid 1 --lun 1 -b /SCSI
tgtadm --lld iscsi --op bind --mode target --tid 1 -I ALL
4.use initiator to connect it .
iscsiadm --mode node --targetname iqn.redhat.com:storage1 --portal <ip> --login
5.find /dev/sdb by using #fdisk -l
6.check /SCSI and /dev/sdb by using #qemu-img check.
7.install RHEL6 Guest in /dev/sdb
#/usr/libexec/qemu-kvm -m 16G -smp 8 -name RHEL6_64 -uuid `uuidgen` -rtc-td-hack -boot dc -cdrom -drive file=/dev/sdb,if=none,id=drive-virtio0-0-0,boot=on,format=qcow2 -device virtio-blk-pci,drive=drive-virtio0-0-0,id=virtio0-0-0 -net nic,macaddr=20:40:50:12:23:21,model=virtio,vlan=0 -net tap,script=/etc/qemu-ifup,vlan=0 -spice port=5930,disable-ticketing -vga qxl -balloon virtio -monitor stdio
8.Repeat Step 6.

Actual results:

After step6,
# qemu-img check /dev/sdb
No errors were found on the image.
# qemu-img check /ISCSI
No errors were found on the image.

After step7,
Core dumped during installation 
(gdb) bt
#0  0x000000000048baa3 in qcow_aio_setup (bs=0x2ab9010, sector_num=20971496, qiov=0x7fa38005f040, 
    nb_sectors=24, cb=<value optimized out>, opaque=<value optimized out>, is_write=1)
    at block/qcow2.c:501
#1  0x000000000048be30 in qcow_aio_writev (bs=<value optimized out>, 
    sector_num=<value optimized out>, qiov=<value optimized out>, nb_sectors=<value optimized out>, 
    cb=<value optimized out>, opaque=<value optimized out>) at block/qcow2.c:651
#2  0x0000000000477cd3 in bdrv_aio_writev (bs=0x2ab9010, sector_num=20971496, qiov=0x7fa38005f040, 
    nb_sectors=24, cb=<value optimized out>, opaque=<value optimized out>) at block.c:1832
#3  0x0000000000478a9b in bdrv_aio_multiwrite (bs=0x2ab9010, reqs=0x7fa387f975d0, num_reqs=2)
    at block.c:2023
#4  0x000000000041d3be in do_multiwrite (bs=<value optimized out>, blkreq=0x7fa387f975d0, 
    num_writes=2) at /usr/src/debug/qemu-kvm-0.12.1.2/hw/virtio-blk.c:235
#5  0x000000000041da68 in virtio_blk_handle_output (vdev=0x2bb3a00, vq=<value optimized out>)
    at /usr/src/debug/qemu-kvm-0.12.1.2/hw/virtio-blk.c:362
#6  0x000000000042a3d1 in kvm_handle_io (env=0x2b00200)
    at /usr/src/debug/qemu-kvm-0.12.1.2/kvm-all.c:538
#7  kvm_run (env=0x2b00200) at /usr/src/debug/qemu-kvm-0.12.1.2/qemu-kvm.c:969
#8  0x000000000042a479 in kvm_cpu_exec (env=<value optimized out>)
    at /usr/src/debug/qemu-kvm-0.12.1.2/qemu-kvm.c:1652
#9  0x000000000042b09f in kvm_main_loop_cpu (_env=0x2b00200)
    at /usr/src/debug/qemu-kvm-0.12.1.2/qemu-kvm.c:1894
#10 ap_main_loop (_env=0x2b00200) at /usr/src/debug/qemu-kvm-0.12.1.2/qemu-kvm.c:1944
#11 0x00000033fe007761 in start_thread () from /lib64/libpthread.so.0
#12 0x00000033fd8e14fd in clone () from /lib64/libc.so.6

After step 8
# qemu-img check /dev/sdb 
ERROR: invalid cluster offset=0x40000
ERROR: invalid cluster offset=0x50000
ERROR: I/O error in check_refcounts_l1
ERROR: I/O error in check_refcounts_l1
qemu-img: An error occurred during the check
# qemu-img check /ISCSI
ERROR: invalid cluster offset=0x40000
ERROR: invalid cluster offset=0x50000
2 errors were found on the image.


Expected results:


Additional info:

Comment 2 RHEL Program Management 2010-06-09 12:33:38 UTC
This request was evaluated by Red Hat Product Management for inclusion in a Red
Hat Enterprise Linux major release.  Product Management has requested further
review of this request by Red Hat Engineering, for potential inclusion in a Red
Hat Enterprise Linux Major release.  This request is not yet committed for
inclusion.

Comment 3 Alex Williamson 2010-06-09 17:46:39 UTC
Hmm, this looks bogus to me...

(In reply to comment #0)
> Steps to Reproduce:
> 1.dd if=/dev/zero of=/SCSI bs=1M count=10240

So no /SCSI is a 10G raw image

> 2.qemu-img create -f qcow2 /SCSI 10G

But now you've overwritten it with a qcow2 image

> 3.Add /SCSI target device.
> eg:
> dd if=/dev/zero of=/SCSI bs=1M count=10240

But now you've overwritten it again as a raw image

> tgtadm --lld iscsi --op new --mode target --tid 1 T iqn.redhat.com:storage1
> tgtadm --lld iscsi --op new --mode logicalunit --tid 1 --lun 1 -b /SCSI
> tgtadm --lld iscsi --op bind --mode target --tid 1 -I ALL
> 4.use initiator to connect it .
> iscsiadm --mode node --targetname iqn.redhat.com:storage1 --portal <ip> --login
> 5.find /dev/sdb by using #fdisk -l
> 6.check /SCSI and /dev/sdb by using #qemu-img check.

And now you've verified that /dev/sdb is the iSCSI device backed by /SCSI, and /dev/sdb is clearly a raw device.

> 7.install RHEL6 Guest in /dev/sdb
> #/usr/libexec/qemu-kvm -m 16G -smp 8 -name RHEL6_64 -uuid `uuidgen`
> -rtc-td-hack -boot dc -cdrom -drive
> file=/dev/sdb,if=none,id=drive-virtio0-0-0,boot=on,format=qcow2 -device

And here you tell qemu it's format=qcow2.  The only way I know of to export a qcow2 image over the network is nbd, and even then the guest would still use it as a raw image.  iSCSI is always going to look like a raw device to the guest.  Re-open if you can still reproduce a problem with format=raw.

Comment 4 Alex Williamson 2010-06-17 15:58:49 UTC
Re-opening.  Note the submitter has since clarified that step 3 was actually:

3.Add /SCSI target device .
eg:
tgtadm --lld iscsi --op new --mode target --tid 1 T iqn.redhat.com:storage1
tgtadm --lld iscsi --op new --mode logicalunit --tid 1 --l

The qcow2 image was not dd'd over.  Still not sure how this is supposed to work, but assigning to someone to provide another opinion.

Comment 5 Kevin Wolf 2010-06-23 13:00:29 UTC
Do you still have the core dump around so that I could take a look?

Comment 7 lihuang 2010-06-29 15:46:28 UTC
*** Bug 604221 has been marked as a duplicate of this bug. ***

Comment 13 Mike Cao 2010-07-22 05:06:14 UTC
Retest this in qemu-kvm-0.12.1.2-2.97.el6 . 
#uname -r
2.6.32-44.1.el6.x86_64

steps:
1.dd if=/dev/zero of=/home/ISCSI2 bs=1M count=15000
2.qemu-img create -f qcow2 /home/ISCSI2 15G
3.Add /home/ISCSI2 target device.
eg:
tgtadm --lld iscsi --op new --mode target --tid 1 T iqn.redhat.com:storage1
tgtadm --lld iscsi --op new --mode logicalunit --tid 1 --lun 1 -b /home/ISCSI2
tgtadm --lld iscsi --op bind --mode target --tid 1 -I ALL
4.use initiator to connect it .
iscsiadm --mode node --targetname iqn.redhat.com:storage1 --portal <ip> --login
5.find /dev/sdd by using #fdisk -l
6.check info  /home/SCSI2 and /dev/sdd by using #qemu-img check.
7.install RHEL6 Guest in /dev/sdd
CLI: /usr/libexec/qemu-kvm -m 4G -smp 4 -name RHEL5 -uuid `uuidgen` -rtc base=localtime -boot dc -cdrom /home/RHEL5.5-Server-20100322.0-x86_64-DVD.iso -drive file=/dev/sdd,if=none,id=mike,boot=on,format=qcow2 -device virtio-blk-pci,drive=mike -net nic,macaddr=20:30:10:38:32:12,model=virtio,vlan=0 -net tap,script=/etc/qemu-ifup,downscript=no,vlan=0 -vnc :3 -balloon virtio -chardev socket,id=monitor,path=/home/ttttttttttttt.monitor,server,nowait -mon chardev=monitor,mode=control

Actual Results:

After step6,
#qemu-img info /home/ISCSI2
image: /home/ISCSI2
file format: qcow2
virtual size: 16G (17179869184 bytes)
disk size: 136K
cluster_size: 65536

#qemu-img info /dev/sdd
image: /dev/sdd
file format: qcow2
virtual size: 16G (17179869184 bytes)
disk size: 0
cluster_size: 65536

#qemu-img check /dev/sdd
No errors were found on the image.

#qemu-img check /home/ISCSI2
No errors were found on the image.

Note /dev/sdd and /home/ISCSI2 are the same image.

After step7,
When the step of guest installation is to initialize drive .Guest turns to stop status and can not be resumed.

{"execute":"cont"}
{"timestamp": {"seconds": 1279773679, "microseconds": 431056}, "event": "RESUME"}
{"return": {}}
{"timestamp": {"seconds": 1279773679, "microseconds": 434683}, "event": "BLOCK_IO_ERROR", "data": {"device": "mike", "__com.redhat_reason": "enospc", "operation": "write", "action": "stop"}}
{"timestamp": {"seconds": 1279773679, "microseconds": 434909}, "event": "STOP"}

Above all ,Re-assign this bug for it has not fixed yet .

Comment 14 Mike Cao 2010-07-22 05:08:49 UTC
Created attachment 433581 [details]
Screen dumped

During this step of installation,guest paused and can not be resumed.

Comment 15 Dor Laor 2010-07-22 08:42:35 UTC
Are you sure you have enough space? Seems that you get ENOSPACE error and when that happens qemu automatically stops. IF you won't enlarge the storage, it will stop again once you 'cont' it.

Also, according to above:
1.dd if=/dev/zero of=/home/ISCSI2 bs=1M count=15000
2.qemu-img create -f qcow2 /home/ISCSI2 15G
3.Add /home/ISCSI2 target device.


Seems like step 2 erases what step 1 did. Probably the same for step 3 and 2.

Comment 16 Mike Cao 2010-07-22 12:05:22 UTC
(In reply to comment #15)
> Are you sure you have enough space? Seems that you get ENOSPACE error and when
> that happens qemu automatically stops. IF you won't enlarge the storage, it
> will stop again once you 'cont' it.

According to comment #13 .dd if=/dev/zero of=/home/ISCSI2 bs=1M count=15000 ,15G is enough for RHEL5.5 Guest installation. and according to comment #14, Guest stop before packages install into virtual disk and even before the image to be formatted.


> Also, according to above:
> 1.dd if=/dev/zero of=/home/ISCSI2 bs=1M count=15000
> 2.qemu-img create -f qcow2 /home/ISCSI2 15G
> 3.Add /home/ISCSI2 target device.
> 
> 
> Seems like step 2 erases what step 1 did. Probably the same for step 3 and 2.    

Step1 is to make image to be iscsi-target's Backing store.Because I don't know whether the raw image made by #qemu-img create ...  Can be iscsi-target's Backing store.
Step2 is to make /home/ISCIS2 be qcow2 disk. 
Step3:eg.
tgtadm --lld iscsi --op new --mode target --tid 1 T iqn.redhat.com:storage1
tgtadm --lld iscsi --op new --mode logicalunit --tid 1 --lun 1 -b /home/ISCSI2
tgtadm --lld iscsi --op bind --mode target --tid 1 -I ALL ,
It will not influence step 1 and step 2.

Based on above ,I don't think this issue has been fixed.

Comment 17 Kevin Wolf 2010-07-23 09:21:09 UTC
First of all, this bug is about a crash (core dumped). You don't get such a crash any more, so this is fixed. If anything, you have found a new problem.

But second, as Dor already said, ENOSPC means that your device is too small to hold a fully allocated image and that you need to extend it before you can resume the VM. Note that 15000M is less than 15G and that qcow2 always needs more space than raw because it contains metadata.

Comment 18 Mike Cao 2010-07-23 10:55:47 UTC
(In reply to comment #17)

> But second, as Dor already said, ENOSPC means that your device is too small to
> hold a fully allocated image and that you need to extend it before you can
> resume the VM. Note that 15000M is less than 15G and that qcow2 always needs
> more space than raw because it contains metadata.    

1.create a 100G file by using 'dd',and make it to be 90G qcow2 images for RHEL5.5 Guest installation.
1.1
# dd if=/dev/zero of=/SCSI/RHEL5 bs=1M count=102400
102400+0 records in
102400+0 records out
107374182400 bytes (107 GB) copied, 1365.03 seconds, 78.7 MB/s
1.2.
# qemu-img info /SCSI/RHEL5
image: /SCSI/RHEL5
file format: raw
virtual size: 100G (107374182400 bytes)
disk size: 100G
1.3.
#qemu-img create -f qcow2 /SCSI/RHEL5 90G
# qemu-img info /SCSI/RHEL5
image: /SCSI/RHEL5
file format: qcow2
virtual size: 90G (96636764160 bytes)
disk size: 136K
cluster_size: 65536

2.Add /SCSI/RHEL5 target device to scsi-target's
Backing store. find /dev/sdc by using fdisk -l
# qemu-img info /dev/sdc 
image: /dev/sdc
file format: qcow2
virtual size: 90G (96636764160 bytes)
disk size: 0
cluster_size: 65536


the other steps are same with comment #13.

Acutal Results:

Same Results in comment #13,When the step of guest installation is to initialize drive .Guest turns to stop
status and can not be resumed.

It is obviously that 90G images is fully enough for RHEL5.5 Guest installation. What else may cause this issue.?

Comment 19 Amit Shah 2010-07-23 11:34:34 UTC
Mike,

Can you start your testing from step 1.3 in comment 18 instead of starting with 1.1?

Also, please open a new bug report since this bug was concerned with a core dump and there's no core dump with the new packages.

Comment 20 Mike Cao 2010-07-27 07:17:30 UTC
(In reply to comment #19)
> Mike,
> 
> Can you start your testing from step 1.3 in comment 18 instead of starting with
> 1.1?

Retested ,still hit this issue.

# uname -r
2.6.32-52.el6.x86_64
# rpm -q qemu-kvm
qemu-kvm-0.12.1.2-2.99.el6.x86_64
steps 
1
#qemu-img create -f qcow2 RHEL5 40G
# qemu-img info /home/RHEL5
image: /home/RHEL5
file format: qcow2
virtual size: 40G (42949672960 bytes)
disk size: 136K
cluster_size: 65536
2. Add this image to Iscsi-target and connected by initiator.
#fdisk -l
Disk /dev/sdc: 0 MB, 262144 bytes
1 heads, 1 sectors/track, 512 cylinders, total 512 sectors
Units = cylinders of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

Disk /dev/sdc doesn't contain a valid partition table

# qemu-img info /dev/sdc
image: /dev/sdc
file format: qcow2
virtual size: 40G (42949672960 bytes)
disk size: 0
cluster_size: 65536

3, start VM by using -drive file=/dev/sdc...

Actual Results .
Still paused and can not be continued as discribed in comment #13

Comment 21 Mike Cao 2010-07-27 07:31:12 UTC
Based on comment #19, the original core dump disappeared, so close this one, open a new bug about bz618521


Note You need to log in before you can comment on or make changes to this bug.