Hide Forgot
Created attachment 527444 [details] RHEL6.1GA vs 6.2-SNAP1 Description of problem: I see performance drop on operation read on SW RAID (dmraid1) device on SAS drives, ext4 filesystem. Description of test and system: Test = /performance/iozone_devel_with_library/certification Hostname = ibm-x3650m3-01.lab.eng.brq.redhat.com Arch = x86_64 Distro = RHEL6.2-20111005.1 Kernel = 2.6.32-206.el6.x86_64 SElinux mode = Permissive CPU count : speeds MHz = 12 : 12 @ 2793.000 CPU model name = Intel(R) Xeon(R) CPU X5650 @ 2.67GHz CPU cache size = 12288 KB BIOS Information Version : Date = -D6E145FUS-1.07- : 04/26/2010 Total Memory = 11899 (MB) NUMA is Enabled. # of nodes = 1 nodes (0) Tuned profile = default RHTS information RHTS JOBID = 140176 RHTS RECIPEID = 290547 RHTS RECIPETESTID = 3201452 FILESYSTEM configuration Filesystems to test (requested) = ext4 Filesystems to test (actual) = ext4 Mount point of filesystem under test = /RHTSspareLUN1 LUN for filesystem under test = /dev/md0 readahead for LUN above = 128 KB Speed test: hdparm -tT /dev/md0 Timing buffered disk reads => 134.26 MB/sec Timing cached reads => 8311.08 MB/sec Free Diskspace on /RHTSspareLUN1 = 275GB get_scheduler_info: DEVICEMOUNT seems to be raid array I/O scheduler on testing device = cfq IOZONE version = 3.327 Page Size [smallest file to work on] = 4 (KB) 90% of Free disk space available = 239601 (MB) Command line options = -r 3 --rsync --eatmem --swap --iozone_umount --iozone_rec_size --incache --directio --outofcache -f ext4 In Cache test maximum file size = 4096 (MB) Out of Cache test minimum file size = 512 (MB) Out of Cache test maximum file size = 2048 (MB) Free memory after running eatmem = 512 (MB) Direct I/O test maximum file size = 4096 (MB) Number of sequential runs = 3 Anaconda kickstart - partitions section: #[sdb] 974608384 512-byte logical blocks: (498 GB/464 GiB) #[sda] 583983104 512-byte logical blocks: (298 GB/278 GiB) #[sdc] 60545024 512-byte logical blocks: (30.9 GB/28.8 GiB) #[sdd] 583983104 512-byte logical blocks: (298 GB/278 GiB) part /boot --fstype ext2 --size=1024 --asprimary --label=BOOT --ondisk=sdc part /mnt/tests --fstype=ext4 --size=4096 --asprimary --label=MNT --ondisk=sdc part / --fstype=ext4 --size=1 --grow --asprimary --label=ROOT --ondisk=sdc part raid.01 --size=1 --grow --ondisk=sda part raid.02 --size=1 --grow --ondisk=sdd part /RHTSspareLUN2 --fstype=ext4 --size=1 --grow --asprimary --label=sdb_rest --ondisk=sdb raid /RHTSspareLUN1 --fstype=ext4 --level=1 --device=md0 raid.01 raid.02 I don't see regression just on ext4 file system, I don't see regression as well on RAID0, RAID5 configuration just in RAID1 configuration. Compared results are attached.
This request was evaluated by Red Hat Product Management for inclusion in the current release of Red Hat Enterprise Linux. Because the affected component is not scheduled to be updated in the current release, Red Hat is unfortunately unable to address this request at this time. Red Hat invites you to ask your support representative to propose this request, if appropriate and relevant, in the next release of Red Hat Enterprise Linux. If you would like it considered as an exception in the current release, please ask your support representative.
Are you referring ti the correct component here (dmraid)? I'm assuming this must be mdraid. Please clarify.
Kamil, Could you please confirm if this is on md-raid or dm-raid, per Heinz's question on 2011-10-11. In addition, do you only see this drop on RAID and not on native disks? Last, are you really running on 3 different drive sizes/models like it is indicated in the kickstart output? Thanks, Jes
No replies to request for info from reporter. Closing
Jes, I don't see any drops regarding to native drive, file system or io scheduler. And I'm testing it each build. This is an issue of mdraid. Tests for SW raid I'm running on only this machine. Soon I will get another one box for RAID testing so I will update this with new results. Sorry for late response and confusing. Thanks Kamil
Kamil, Please provide data on the system, partition layout, raid config, output of /proc/mdstat, iozone parameters, how you run the test on top of the raid.
As you can see from comment 1: part /boot --fstype ext2 --size=1024 --asprimary --label=BOOT --ondisk=sdc part /mnt/tests --fstype=ext4 --size=4096 --asprimary --label=MNT --ondisk=sdc part / --fstype=ext4 --size=1 --grow --asprimary --label=ROOT --ondisk=sdc part raid.01 --size=1 --grow --ondisk=sda part raid.02 --size=1 --grow --ondisk=sdd part /RHTSspareLUN2 --fstype=ext4 --size=1 --grow --asprimary --label=sdb_rest --ondisk=sdb raid /RHTSspareLUN1 --fstype=ext4 --level=1 --device=md0 raid.01 raid.02 So I'm installing mdraid using anaconda kickstart. iozone -U /RHTSspareLUN1 -a -f /RHTSspareLUN1/ext3 -n 4k -g 4096m iozone -U /RHTSspareLUN1 -a -f /RHTSspareLUN1/ext3 -I -g 4096m -i 0 -i 1 -i 2 -i 3 -i 4 -i 5 iozone -U /RHTSspareLUN1 -a -f /RHTSspareLUN1/ext3 -n 512m -g 2048m It mean I'm running test on /RHTSspareLUN1 (raid1). Each this instance of iozone I'm running 3 times and doing averages. Second and third run instance is running with shortage of memory (to omit cache effect). I don't have now /proc/mdstat file saved. Next run when I will run it I will save it for you and attach here.
I really need the real partition layout, the kickstart stuff doesn't mean anything to me. It means I have to guess how things may end up looking on the real system. /proc/mdstat is absolutely vital for any md raid related bug report. Thanks, Jes
$ df -l Filesystem 1K-blocks Used Available Use% Mounted on /dev/sdc3 24237240 1951304 21034876 9% / /dev/sdb1 472038708 202800 447470928 1% /RHTSspareLUN2 /dev/sdc2 4061572 89248 3762676 3% /mnt/tests /dev/sdc1 1019208 7760 958840 1% /boot tmpfs 6136504 0 6136504 0% /dev/shm /dev/md0 287407328 195616 272612248 1% /RHTSspareLUN1 During running iozone testing - resync $ cat /proc/mdstat Personalities : [raid1] md0 : active raid1 sdd1[1] sda1[0] 291989312 blocks [2/2] [UU] [===========>.........] resync = 55.2% (161389312/291989312) finish=2102.0min speed=1032K/sec unused devices: <none> It is all you need?
I need the partition table covering sdd1 and sda1, so /proc/partitions output or the output from parted, please.
/proc/partitions 8 0 291991552 sda 8 1 291989376 sda1 8 16 487304192 sdb 8 17 487299613 sdb1 8 32 30272512 sdc 8 33 1052226 sdc1 8 34 4192965 sdc2 8 35 25021237 sdc3 8 48 291991552 sdd 8 49 291989376 sdd1 9 0 291989312 md0
Kamil, Any data as to whether the situation improved in 6.3? Jes
No reply to my last question for > 1 month, so I am assuming this is no longer a problem. If wanting to verify, please check against latest 6.4 kernel.