Description of problem: During verify the Bug 488698. I found it need much more time to finish the copy. (copy a 2.8G file to scsi disk ) especially on i386 arch. with xen-3.0.3-80.el5 it only need less than 5 minutes, while with xen-3.0.3-80.el5_3.2.i386 it need more than 90 minutes. on x86_64 arch, the performance is better. the copy could finished in 15minute with the new package . Host info: [root@han ~]# uname -a Linux han 2.6.18-128.el5xen #1 SMP Wed Dec 17 12:22:24 EST 2008 i686 athlon i386 GNU/Linux Guest info: name = "rbug" uuid = "8e5f813f-505d-ae16-bedb-d14d43e505c2" maxmem = 4096 memory = 4096 vcpus = 1 builder = "hvm" kernel = "/usr/lib/xen/boot/hvmloader" boot = "c" pae = 1 acpi = 1 apic = 1 localtime = 0 on_poweroff = "destroy" on_reboot = "restart" on_crash = "restart" device_model = "/usr/lib/xen/bin/qemu-dm" sdl = 0 vnc = 1 vncunused = 0 keymap = "en-us" vncdisplay = "7" disk = [ "file:/var/lib/xen/images/rbug.img,hda,w", ",hdc:cdrom,r", "file:/var/lib/xen/images/Newrbug.img,sda,w" ] vif = [ "mac=00:16:3e:0b:16:93,bridge=xenbr0,script=vif-bridge" ] serial = "pty" [root@dhcp-66-83-226 ~]# uname -a Linux dhcp-66-83-226.nay.redhat.com 2.6.18-92.el5PAE #1 SMP Tue Apr 29 13:31:02 EDT 2008 i686 athlon i386 GNU/Linux Version-Release number of selected component (if applicable): xen-3.0.3-80.el5_3.2.i386 How reproducible: Every Time Steps to Reproduce: 1. Create vm 2. add a scsi disk to the guest ( a image of another VM. 20G, with 2 partion "/boot" and "/". disk = [ "file:/var/lib/xen/images/rbug.img,hda,w", ",hdc:cdrom,r", "file:/var/lib/xen/images/Newrbug.img,sdb,w" ] 3. mount a partion on the guest: mount /dev/sda2 /mnt 4. run ' cp ~/Desktop/rhel-5.1-server-i386-dvd.iso /mnt/. Actual results: Expected results: Additional info:
Waiting for disk IO to complete before submitting the next IO will always be slower than submitting all IO to dom0 and hoping the data will be written to disk at some point in the future. Maybe the AIO backport (which I am working on now) will improve this a little in the future, but most of the slowdown is probably a feature, not a bug :)
I've seen SCSI vs IDE benchmarks for QEMU, but they were always done with the AIO based impl of QEMU. I'm not surprised the non-AIO impl has slowed down, and doubt there's anything we can do about that, while still ensuring safety.
Rik, do you have a BZ number for the AIO backport ? If so, can you make this ticket depend on that BZ.
The AIO backport is an attempt at fixing bugs like bug #465116 which appear to be caused due to the HVM guest spending too much time in qemu. This will especially be true for virtual scsi disks, which have no DMA thread.
I have backported AIO to qemu's IDE and SCSI emulation. Could you try out the test RPMs on http://people.redhat.com/riel/.xen-aio/ and let me know if IO is still horribly slow with those?
lihuang --> Riel: Hi Riel. I still failed to mount the disk with the new packages... following is my steps: 1. Create a New VM. 2. Add the scsi disk to the guest. ( A copy of the new vm's img ) 3. run mount /dev/sda2 /mnt ==> passed. and the data is shown in /mnt 4. scp the iso file from host to Guest. 5. ls /mnt ==> not data 6. run 'umount /mnt' and re-mount 'mount /dev/sda2 /mnt' ==> failed [root@dhcp-66-70-17 ~]# mount /dev/sda2 /mnt/ mount: wrong fs type, bad option, bad superblock on /dev/sda2, missing codepage or other error In some cases useful info is found in syslog - try dmesg | tail or so output of dmesg: kjournald starting. Commit interval 5 seconds EXT3 FS on sda2, internal journal EXT3-fs: recovery complete. EXT3-fs: mounted filesystem with ordered data mode. SELinux: initialized (dev sda2, type ext3), uses xattr EXT3-fs error (device sda2): htree_dirblock_to_tree: bad entry in directory #2: rec_len % 4 != 0 - offset=0, inode=4294967295, rec_len=65535, name_len=255 Aborting journal on device sda2. ext3_abort called. EXT3-fs error (device sda2): ext3_journal_start_sb: Detected aborted journal Remounting filesystem read-only EXT3-fs error (device sda2): htree_dirblock_to_tree: bad entry in directory #2: rec_len % 4 != 0 - offset=0, inode=4294967295, rec_len=65535, name_len=255 EXT3-fs error (device sda2): htree_dirblock_to_tree: bad entry in directory #2: rec_len % 4 != 0 - offset=0, inode=4294967295, rec_len=65535, name_len=255 EXT3-fs error (device sda2): htree_dirblock_to_tree: bad entry in directory #2: rec_len % 4 != 0 - offset=0, inode=4294967295, rec_len=65535, name_len=255 EXT3-fs error (device sda2): htree_dirblock_to_tree: bad entry in directory #2: rec_len % 4 != 0 - offset=0, inode=4294967295, rec_len=65535, name_len=255 EXT3-fs error (device sda2): htree_dirblock_to_tree: bad entry in directory #2: rec_len % 4 != 0 - offset=0, inode=4294967295, rec_len=65535, name_len=255 EXT3-fs error (device sda2): ext3_check_descriptors: Inode table for group 0 not in group (block 1238641059)! EXT3-fs: group descriptors corrupted! EXT3-fs error (device sda2): ext3_check_descriptors: Inode table for group 0 not in group (block 1238641059)! EXT3-fs: group descriptors corrupted! EXT3-fs error (device sda2): ext3_check_descriptors: Inode table for group 0 not in group (block 1238641059)! EXT3-fs: group descriptors corrupted! EXT3-fs error (device sda2): ext3_check_descriptors: Inode table for group 0 not in group (block 1238641059)! EXT3-fs: group descriptors corrupted! [root@dhcp-66-70-17 ~]# what might be wrong ? Thanks
Host info: [root@dhcp-66-70-55 images]# uname -a Linux dhcp-66-70-55.nay.redhat.com 2.6.18-128.el5xen #1 SMP Wed Dec 17 12:22:24 EST 2008 i686 i686 i386 GNU/Linux [root@dhcp-66-70-55 images]# rpm -qa | grep xen kmod-gnbd-xen-0.1.5-2.el5 xen-3.0.3-83.el5.aio2 kernel-xen-2.6.18-128.el5 kmod-cmirror-xen-0.1.21-10.el5 kernel-xen-devel-2.6.18-128.el5 xen-debuginfo-3.0.3-83.el5.aio2 xen-devel-3.0.3-83.el5.aio2 kmod-gfs-xen-0.1.31-3.el5 xen-libs-3.0.3-83.el5.aio2 Guest Info: [root@dhcp-66-70-55 images]# cat /etc/xen/ auto/ xend-config.sxp xmexample2 qemu-ifup xend-pci-permissive.sxp xmexample.hvm scripts/ xend-pci-quirks.sxp xmexample.vti TestAio2 xmexample1 [root@dhcp-66-70-55 images]# cat /etc/xen/TestAio2 name = "TestAio2" uuid = "3288376f-8189-b6bc-542e-0ee7b94ef4a1" maxmem = 1024 memory = 1024 vcpus = 1 builder = "hvm" kernel = "/usr/lib/xen/boot/hvmloader" boot = "c" pae = 1 acpi = 1 apic = 1 localtime = 0 on_poweroff = "destroy" on_reboot = "restart" on_crash = "restart" device_model = "/usr/lib/xen/bin/qemu-dm" sdl = 0 vnc = 1 vncunused = 1 keymap = "en-us" disk = [ "file:/var/lib/xen/images/TestAio2.img,hda,w", ",hdc:cdrom,r", "file:/var/lib/xen/images/scsi.img,sda,w" ] vif = [ "mac=00:16:3e:43:1d:58,bridge=xenbr0" ] serial = "pty"
2. Add the scsi disk to the guest. ( A copy of the new vm's img ) Do you mean you are trying to access the same disk image twice from the guest? That looks like something that could confuse the filesystem code. Please try it with a newly created (empty) virtual scsi disk, running some tests on that (mkfs, mount, etc).
(In reply to comment #8) > 2. Add the scsi disk to the guest. ( A copy of the new vm's img ) > > Do you mean you are trying to access the same disk image twice from the guest? no. the virtual scsi disk is copy of the vm image. in comment #7: disk = [ "file:/var/lib/xen/images/TestAio2.img,hda,w", ",hdc:cdrom,r", "file:/var/lib/xen/images/scsi.img,sda,w" ] > > That looks like something that could confuse the filesystem code. > > Please try it with a newly created (empty) virtual scsi disk, running some > tests on that (mkfs, mount, etc). ok. I will test again.
> > That looks like something that could confuse the filesystem code. > > > > Please try it with a newly created (empty) virtual scsi disk, running some > > tests on that (mkfs, mount, etc). > > ok. I will test again. Just re-check the rpms in comment 5. have re-installed a RHEL5U3.i386 Host and FV Guest. and updated xen to the .aio2 version, restart xend. in the guest: create partition. make ext3 file system. mount. --> FAILED !! SCSI device sda: 41984000 512-byte hdwr sectors (21496 MB) sda: Write Protect is off sda: Mode Sense: 13 00 00 00 SCSI device sda: drive cache: write back sda: sda1 SCSI device sda: 41984000 512-byte hdwr sectors (21496 MB) sda: Write Protect is off sda: Mode Sense: 13 00 00 00 SCSI device sda: drive cache: write back sda: sda1 EXT3-fs error (device sda1): ext3_check_descriptors: Block bitmap for group 1 not in group (block 0)! EXT3-fs: group descriptors corrupted! maybe the test build is out of date... and maybe the aio patches are aready in 5.4/5.5 if so. I will check the the problem (mount and performance in 5.5 latest tree) later. if not,i am ready to test the new scratch build, just needinfo me. :) Thanks Lijun Huang
(In reply to comment #12) > > > That looks like something that could confuse the filesystem code. > > > > > > Please try it with a newly created (empty) virtual scsi disk, running some > > > tests on that (mkfs, mount, etc). > > > > ok. I will test again. > > Just re-check the rpms in comment 5. > have re-installed a RHEL5U3.i386 Host and FV Guest. > and updated xen to the .aio2 version, restart xend. > > in the guest: > > create partition. > make ext3 file system. > mount. > --> FAILED !! > SCSI device sda: 41984000 512-byte hdwr sectors (21496 MB) > sda: Write Protect is off > sda: Mode Sense: 13 00 00 00 > SCSI device sda: drive cache: write back > sda: sda1 > SCSI device sda: 41984000 512-byte hdwr sectors (21496 MB) > sda: Write Protect is off > sda: Mode Sense: 13 00 00 00 > SCSI device sda: drive cache: write back > sda: sda1 > EXT3-fs error (device sda1): ext3_check_descriptors: Block bitmap for group 1 > not in group (block 0)! > EXT3-fs: group descriptors corrupted! > > > maybe the test build is out of date... > and maybe the aio patches are aready in 5.4/5.5 > if so. > I will check the the problem (mount and performance in 5.5 latest tree) later. > if not,i am ready to test the new scratch build, just needinfo me. :) > > Thanks > Lijun Huang Oh, thanks for testing but could you try retesting using the latest kernel-xen and xen shipped with RHEL 5.4 or the newest versions (kernel-xen-2.6.18-191.el5 and xen-3.0.3-105.el5) ? Michal
Done : ==== Host ==== [root@amd-9600b-4-4 ~]# uname -a Linux amd-9600b-4-4 2.6.18-192.el5xen #1 SMP Tue Mar 9 12:12:11 EST 2010 i686 athlon i386 GNU/Linux [root@amd-9600b-4-4 ~]# rpm -q xen xen-3.0.3-105.el5 ===== Guest ===== [root@virtlab-66-85-36 ~]# uname -a Linux virtlab-66-85-36.englab.nay.redhat.com 2.6.18-128.el5 #1 SMP Wed Dec 17 11:42:39 EST 2008 i686 athlon i386 GNU/Linux [root@virtlab-66-85-36 ~]# fdisk -l Disk /dev/hda: 21.4 GB, 21495808000 bytes 255 heads, 63 sectors/track, 2613 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Device Boot Start End Blocks Id System /dev/hda1 * 1 13 104391 83 Linux /dev/hda2 14 2613 20884500 8e Linux LVM Disk /dev/sda: 10.7 GB, 10737418240 bytes 64 heads, 32 sectors/track, 10240 cylinders Units = cylinders of 2048 * 512 = 1048576 bytes Device Boot Start End Blocks Id System /dev/sda1 1 10240 10485744 83 Linux ====== result ====== 1. mount with out error. 2. the copy operation is finished with a reasonable time [root@virtlab-66-85-36 ~]# time cp RHEL5.3-Server-20090106.0-i386-DVD.iso /mnt/ide-to-scsi real 6m23.443s user 0m0.817s sys 2m9.934s [root@virtlab-66-85-36 ~]# time cp RHEL5.3-Server-20090106.0-i386-DVD.iso ide-to-ide real 5m33.815s user 0m0.840s sys 0m44.541s [root@virtlab-66-85-36 ~]# dd if=/dev/zero of=/mnt/scsi bs=1M count=3072 3072+0 records in 3072+0 records out 3221225472 bytes (3.2 GB) copied, 186.793 seconds, 17.2 MB/s [root@virtlab-66-85-36 ~]# dd if=/dev/zero of=ide bs=1M count=3072 3072+0 records in 3072+0 records out 3221225472 bytes (3.2 GB) copied, 212.73 seconds, 15.1 MB/s [root@virtlab-66-85-36 ~]# dd if=/mnt/RHEL5.3-Server-20090106.0-i386-DVD.iso of=/dev/null 5886444+0 records in 5886444+0 records out 3013859328 bytes (3.0 GB) copied, 182.073 seconds, 16.6 MB/s [root@virtlab-66-85-36 ~]# dd if=RHEL5.3-Server-20090106.0-i386-DVD.iso of=/dev/null 5886444+0 records in 5886444+0 records out 3013859328 bytes (3.0 GB) copied, 199.909 seconds, 15.1 MB/s Thanks Lijun Huang.