Bug 490846 - The fix of Bug 488698 cause a slowdown especially on i386 arch when writing to scsi disk
The fix of Bug 488698 cause a slowdown especially on i386 arch when writing t...
Status: CLOSED CURRENTRELEASE
Product: Red Hat Enterprise Linux 5
Classification: Red Hat
Component: xen (Show other bugs)
5.3
All Linux
low Severity medium
: rc
: ---
Assigned To: Rik van Riel
Virtualization Bugs
:
Depends On: 465116
Blocks: 514499
  Show dependency treegraph
 
Reported: 2009-03-18 06:15 EDT by lihuang
Modified: 2010-11-09 08:26 EST (History)
5 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2010-04-12 05:20:50 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description lihuang 2009-03-18 06:15:46 EDT
Description of problem:
During verify the Bug 488698. I found it need much more time to finish the copy.
(copy a 2.8G file to scsi disk )

especially on i386 arch.
with xen-3.0.3-80.el5 it only need less than 5 minutes,
while with xen-3.0.3-80.el5_3.2.i386 it need more than 90 minutes.

on x86_64 arch, the performance is better.
the copy could finished in 15minute with the new package .


Host info:
[root@han ~]# uname -a
Linux han 2.6.18-128.el5xen #1 SMP Wed Dec 17 12:22:24 EST 2008 i686 athlon i386 GNU/Linux



Guest info:
name = "rbug"
uuid = "8e5f813f-505d-ae16-bedb-d14d43e505c2"
maxmem = 4096
memory = 4096
vcpus = 1
builder = "hvm"
kernel = "/usr/lib/xen/boot/hvmloader"
boot = "c"
pae = 1
acpi = 1
apic = 1
localtime = 0
on_poweroff = "destroy"
on_reboot = "restart"
on_crash = "restart"
device_model = "/usr/lib/xen/bin/qemu-dm"
sdl = 0
vnc = 1
vncunused = 0
keymap = "en-us"
vncdisplay = "7"
disk = [ "file:/var/lib/xen/images/rbug.img,hda,w", ",hdc:cdrom,r", "file:/var/lib/xen/images/Newrbug.img,sda,w" ]
vif = [ "mac=00:16:3e:0b:16:93,bridge=xenbr0,script=vif-bridge" ]
serial = "pty"

[root@dhcp-66-83-226 ~]# uname -a
Linux dhcp-66-83-226.nay.redhat.com 2.6.18-92.el5PAE #1 SMP Tue Apr 29 13:31:02 EDT 2008 i686 athlon i386 GNU/Linux




Version-Release number of selected component (if applicable):
xen-3.0.3-80.el5_3.2.i386

How reproducible:
Every Time 


Steps to Reproduce:
1. Create vm
2. add a scsi disk to the guest ( a image of another VM. 20G, with 2 partion
"/boot" and "/".
disk = [ "file:/var/lib/xen/images/rbug.img,hda,w", ",hdc:cdrom,r",
"file:/var/lib/xen/images/Newrbug.img,sdb,w" ]
3. mount a partion on the guest: mount /dev/sda2 /mnt
4. run ' cp ~/Desktop/rhel-5.1-server-i386-dvd.iso /mnt/.

  
Actual results:


Expected results:


Additional info:
Comment 1 Rik van Riel 2009-03-18 14:13:15 EDT
Waiting for disk IO to complete before submitting the next IO will always be slower than submitting all IO to dom0 and hoping the data will be written to disk at some point in the future.

Maybe the AIO backport (which I am working on now) will improve this a little in the future, but most of the slowdown is probably a feature, not a bug :)
Comment 2 Daniel Berrange 2009-03-19 06:20:33 EDT
I've seen SCSI vs IDE benchmarks for QEMU, but they were always done with the AIO based impl of QEMU. I'm not surprised the non-AIO impl has slowed down, and doubt there's anything we can do about that, while still ensuring safety.
Comment 3 Daniel Berrange 2009-03-19 06:21:12 EDT
Rik, do you have a BZ number for the AIO backport ?  If so, can you make this ticket depend on that BZ.
Comment 4 Rik van Riel 2009-03-19 10:48:35 EDT
The AIO backport is an attempt at fixing bugs like bug #465116 which appear to be caused due to the HVM guest spending too much time in qemu.  This will especially be true for virtual scsi disks, which have no DMA thread.
Comment 5 Rik van Riel 2009-03-26 13:31:40 EDT
I have backported AIO to qemu's IDE and SCSI emulation.  Could you try out the test RPMs on http://people.redhat.com/riel/.xen-aio/ and let me know if IO is still horribly slow with those?
Comment 6 lihuang 2009-04-01 23:49:53 EDT
lihuang --> Riel:

Hi Riel. I still failed to mount the disk with the new packages...

following is my steps:
1.  Create a New VM.
2.  Add the  scsi disk to the guest. ( A copy of the new vm's img )
3.  run mount /dev/sda2 /mnt
    ==> passed. and the data is shown in /mnt
4.  scp the iso file from host to Guest.  
5.  ls /mnt  ==> not data 
6.  run 'umount /mnt' and  re-mount 'mount /dev/sda2 /mnt'
    ==> failed 
[root@dhcp-66-70-17 ~]# mount /dev/sda2 /mnt/
mount: wrong fs type, bad option, bad superblock on /dev/sda2,
       missing codepage or other error
       In some cases useful info is found in syslog - try
       dmesg | tail  or so

output of dmesg:
kjournald starting.  Commit interval 5 seconds
EXT3 FS on sda2, internal journal
EXT3-fs: recovery complete.
EXT3-fs: mounted filesystem with ordered data mode.
SELinux: initialized (dev sda2, type ext3), uses xattr
EXT3-fs error (device sda2): htree_dirblock_to_tree: bad entry in directory #2: rec_len % 4 != 0 - offset=0, inode=4294967295, rec_len=65535, name_len=255
Aborting journal on device sda2.
ext3_abort called.
EXT3-fs error (device sda2): ext3_journal_start_sb: Detected aborted journal
Remounting filesystem read-only
EXT3-fs error (device sda2): htree_dirblock_to_tree: bad entry in directory #2: rec_len % 4 != 0 - offset=0, inode=4294967295, rec_len=65535, name_len=255
EXT3-fs error (device sda2): htree_dirblock_to_tree: bad entry in directory #2: rec_len % 4 != 0 - offset=0, inode=4294967295, rec_len=65535, name_len=255
EXT3-fs error (device sda2): htree_dirblock_to_tree: bad entry in directory #2: rec_len % 4 != 0 - offset=0, inode=4294967295, rec_len=65535, name_len=255
EXT3-fs error (device sda2): htree_dirblock_to_tree: bad entry in directory #2: rec_len % 4 != 0 - offset=0, inode=4294967295, rec_len=65535, name_len=255
EXT3-fs error (device sda2): htree_dirblock_to_tree: bad entry in directory #2: rec_len % 4 != 0 - offset=0, inode=4294967295, rec_len=65535, name_len=255
EXT3-fs error (device sda2): ext3_check_descriptors: Inode table for group 0 not in group (block 1238641059)!
EXT3-fs: group descriptors corrupted!
EXT3-fs error (device sda2): ext3_check_descriptors: Inode table for group 0 not in group (block 1238641059)!
EXT3-fs: group descriptors corrupted!
EXT3-fs error (device sda2): ext3_check_descriptors: Inode table for group 0 not in group (block 1238641059)!
EXT3-fs: group descriptors corrupted!
EXT3-fs error (device sda2): ext3_check_descriptors: Inode table for group 0 not in group (block 1238641059)!
EXT3-fs: group descriptors corrupted!
[root@dhcp-66-70-17 ~]# 

what might be wrong ?




Thanks
Comment 7 lihuang 2009-04-02 01:02:31 EDT
Host info:
[root@dhcp-66-70-55 images]# uname -a
Linux dhcp-66-70-55.nay.redhat.com 2.6.18-128.el5xen #1 SMP Wed Dec 17 12:22:24 EST 2008 i686 i686 i386 GNU/Linux
[root@dhcp-66-70-55 images]# rpm -qa | grep xen
kmod-gnbd-xen-0.1.5-2.el5
xen-3.0.3-83.el5.aio2
kernel-xen-2.6.18-128.el5
kmod-cmirror-xen-0.1.21-10.el5
kernel-xen-devel-2.6.18-128.el5
xen-debuginfo-3.0.3-83.el5.aio2
xen-devel-3.0.3-83.el5.aio2
kmod-gfs-xen-0.1.31-3.el5
xen-libs-3.0.3-83.el5.aio2


Guest Info:
[root@dhcp-66-70-55 images]# cat /etc/xen/
auto/                    xend-config.sxp          xmexample2
qemu-ifup                xend-pci-permissive.sxp  xmexample.hvm
scripts/                 xend-pci-quirks.sxp      xmexample.vti
TestAio2                 xmexample1               
[root@dhcp-66-70-55 images]# cat /etc/xen/TestAio2 
name = "TestAio2"
uuid = "3288376f-8189-b6bc-542e-0ee7b94ef4a1"
maxmem = 1024
memory = 1024
vcpus = 1
builder = "hvm"
kernel = "/usr/lib/xen/boot/hvmloader"
boot = "c"
pae = 1
acpi = 1
apic = 1
localtime = 0
on_poweroff = "destroy"
on_reboot = "restart"
on_crash = "restart"
device_model = "/usr/lib/xen/bin/qemu-dm"
sdl = 0
vnc = 1
vncunused = 1
keymap = "en-us"
disk = [ "file:/var/lib/xen/images/TestAio2.img,hda,w", ",hdc:cdrom,r", "file:/var/lib/xen/images/scsi.img,sda,w" ]
vif = [ "mac=00:16:3e:43:1d:58,bridge=xenbr0" ]
serial = "pty"
Comment 8 Rik van Riel 2009-06-02 10:58:32 EDT
2.  Add the  scsi disk to the guest. ( A copy of the new vm's img )

Do you mean you are trying to access the same disk image twice from the guest?

That looks like something that could confuse the filesystem code.

Please try it with a newly created (empty) virtual scsi disk, running some tests on that (mkfs, mount, etc).
Comment 9 lihuang 2009-06-02 11:58:31 EDT
(In reply to comment #8)
> 2.  Add the  scsi disk to the guest. ( A copy of the new vm's img )
> 
> Do you mean you are trying to access the same disk image twice from the guest?

no. the virtual scsi disk is copy of the vm image.
in comment #7:
disk = [ "file:/var/lib/xen/images/TestAio2.img,hda,w", ",hdc:cdrom,r",
"file:/var/lib/xen/images/scsi.img,sda,w" ]



> 
> That looks like something that could confuse the filesystem code.
> 
> Please try it with a newly created (empty) virtual scsi disk, running some
> tests on that (mkfs, mount, etc).  

ok.  I will test again.
Comment 12 lihuang 2010-03-16 10:11:16 EDT
> > That looks like something that could confuse the filesystem code.
> > 
> > Please try it with a newly created (empty) virtual scsi disk, running some
> > tests on that (mkfs, mount, etc).  
> 
> ok.  I will test again.    

Just re-check the rpms in comment 5. 
have re-installed a RHEL5U3.i386 Host and FV Guest.
and updated xen to the .aio2 version, restart xend.

in the guest:

create partition.
make ext3 file system.
mount.
--> FAILED !!
SCSI device sda: 41984000 512-byte hdwr sectors (21496 MB)
sda: Write Protect is off
sda: Mode Sense: 13 00 00 00
SCSI device sda: drive cache: write back
 sda: sda1
SCSI device sda: 41984000 512-byte hdwr sectors (21496 MB)
sda: Write Protect is off
sda: Mode Sense: 13 00 00 00
SCSI device sda: drive cache: write back
 sda: sda1
EXT3-fs error (device sda1): ext3_check_descriptors: Block bitmap for group 1 not in group (block 0)!
EXT3-fs: group descriptors corrupted!


maybe the test build is out of date...
and maybe the aio patches are aready in 5.4/5.5
if so.
I will check the the problem (mount and performance in 5.5 latest tree) later.
if not,i am ready to test the new scratch build, just needinfo me. :)

Thanks
Lijun Huang
Comment 13 Michal Novotny 2010-03-16 12:31:22 EDT
(In reply to comment #12)
> > > That looks like something that could confuse the filesystem code.
> > > 
> > > Please try it with a newly created (empty) virtual scsi disk, running some
> > > tests on that (mkfs, mount, etc).  
> > 
> > ok.  I will test again.    
> 
> Just re-check the rpms in comment 5. 
> have re-installed a RHEL5U3.i386 Host and FV Guest.
> and updated xen to the .aio2 version, restart xend.
> 
> in the guest:
> 
> create partition.
> make ext3 file system.
> mount.
> --> FAILED !!
> SCSI device sda: 41984000 512-byte hdwr sectors (21496 MB)
> sda: Write Protect is off
> sda: Mode Sense: 13 00 00 00
> SCSI device sda: drive cache: write back
>  sda: sda1
> SCSI device sda: 41984000 512-byte hdwr sectors (21496 MB)
> sda: Write Protect is off
> sda: Mode Sense: 13 00 00 00
> SCSI device sda: drive cache: write back
>  sda: sda1
> EXT3-fs error (device sda1): ext3_check_descriptors: Block bitmap for group 1
> not in group (block 0)!
> EXT3-fs: group descriptors corrupted!
> 
> 
> maybe the test build is out of date...
> and maybe the aio patches are aready in 5.4/5.5
> if so.
> I will check the the problem (mount and performance in 5.5 latest tree) later.
> if not,i am ready to test the new scratch build, just needinfo me. :)
> 
> Thanks
> Lijun Huang    

Oh, thanks for testing but could you try retesting using the latest kernel-xen and xen shipped with RHEL 5.4 or the newest versions (kernel-xen-2.6.18-191.el5 and xen-3.0.3-105.el5) ?

Michal
Comment 14 lihuang 2010-03-17 00:40:40 EDT
Done :
====
Host
====
[root@amd-9600b-4-4 ~]# uname -a
Linux amd-9600b-4-4 2.6.18-192.el5xen #1 SMP Tue Mar 9 12:12:11 EST 2010 i686 athlon i386 GNU/Linux
[root@amd-9600b-4-4 ~]# rpm -q xen
xen-3.0.3-105.el5

=====
Guest
=====
[root@virtlab-66-85-36 ~]# uname -a
Linux virtlab-66-85-36.englab.nay.redhat.com 2.6.18-128.el5 #1 SMP Wed Dec 17 11:42:39 EST 2008 i686 athlon i386 GNU/Linux

[root@virtlab-66-85-36 ~]# fdisk -l

Disk /dev/hda: 21.4 GB, 21495808000 bytes
255 heads, 63 sectors/track, 2613 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

   Device Boot      Start         End      Blocks   Id  System
/dev/hda1   *           1          13      104391   83  Linux
/dev/hda2              14        2613    20884500   8e  Linux LVM

Disk /dev/sda: 10.7 GB, 10737418240 bytes
64 heads, 32 sectors/track, 10240 cylinders
Units = cylinders of 2048 * 512 = 1048576 bytes

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1               1       10240    10485744   83  Linux

======
result
======
1. mount with out error.
2. the copy operation is finished with a reasonable time

[root@virtlab-66-85-36 ~]# time cp RHEL5.3-Server-20090106.0-i386-DVD.iso /mnt/ide-to-scsi

real    6m23.443s
user    0m0.817s
sys     2m9.934s
[root@virtlab-66-85-36 ~]# time cp RHEL5.3-Server-20090106.0-i386-DVD.iso ide-to-ide

real    5m33.815s
user    0m0.840s
sys     0m44.541s

[root@virtlab-66-85-36 ~]# dd if=/dev/zero of=/mnt/scsi bs=1M count=3072
3072+0 records in
3072+0 records out
3221225472 bytes (3.2 GB) copied, 186.793 seconds, 17.2 MB/s

[root@virtlab-66-85-36 ~]# dd if=/dev/zero of=ide bs=1M count=3072
3072+0 records in
3072+0 records out
3221225472 bytes (3.2 GB) copied, 212.73 seconds, 15.1 MB/s

[root@virtlab-66-85-36 ~]# dd if=/mnt/RHEL5.3-Server-20090106.0-i386-DVD.iso of=/dev/null 
5886444+0 records in
5886444+0 records out
3013859328 bytes (3.0 GB) copied, 182.073 seconds, 16.6 MB/s

[root@virtlab-66-85-36 ~]# dd if=RHEL5.3-Server-20090106.0-i386-DVD.iso of=/dev/null 
5886444+0 records in
5886444+0 records out
3013859328 bytes (3.0 GB) copied, 199.909 seconds, 15.1 MB/s



Thanks 
Lijun Huang.

Note You need to log in before you can comment on or make changes to this bug.