Description of problem: Building Xen guest nodes on Dell 860 and SC1435 hardware with PERC 5 controllers takes 4-5 hours using virt-install. Process begins to hang on filesystem formatting, eventually finishes. virt-install standard 512MB guest, no full virtualization and installation over nfs The problem is can be reproduced every single time. The problem does not seem to occur on older Dell 1850 (Perc 4) hardeware. Module Size Used by backupdriver 305848 0 nfs 224961 0 lockd 58697 1 nfs fscache 20449 1 nfs nfs_acl 7617 1 nfs xt_physdev 6993 10 iptable_filter 7105 1 ip_tables 17029 1 iptable_filter x_tables 17349 2 xt_physdev,ip_tables bridge 53725 1 xt_physdev netloop 10817 0 netbk 78017 0 [permanent] blktap 385125 22 [permanent] blkbk 21089 0 [permanent] autofs4 23749 2 hidp 23105 2 rfcomm 42457 0 l2cap 29505 10 hidp,rfcomm bluetooth 53925 5 hidp,rfcomm,l2cap sunrpc 142973 4 nfs,lockd,nfs_acl ipv6 250369 14 video 19269 0 sbs 18533 0 i2c_ec 9025 1 sbs button 10705 0 battery 13637 0 asus_acpi 19289 0 ac 9157 0 parport_pc 29157 0 lp 15849 0 parport 37641 2 parport_pc,lp joydev 13185 0 sg 35933 0 pcspkr 7105 0 tg3 101317 0 i2c_i801 11341 0 i2c_core 23745 2 i2c_ec,i2c_i801 serial_core 23617 0 serio_raw 10693 0 dm_snapshot 20581 0 dm_zero 6209 0 dm_mirror 29841 0 dm_mod 56537 8 dm_snapshot,dm_zero,dm_mirror mptsas 29001 2 mptscsih 26177 1 mptsas mptbase 53089 2 mptsas,mptscsih scsi_transport_sas 29249 1 mptsas sd_mod 22977 3 scsi_mod 130893 5 sg,mptsas,mptscsih,scsi_transport_sas,sd_mod ext3 123081 2 jbd 56553 1 ext3 ehci_hcd 33229 0 ohci_hcd 23645 0 uhci_hcd 25677 0 Version-Release number of selected component (if applicable): How reproducible: Steps to Reproduce: 1. build vps node virt-install 2. create new file system and select packages 3. wait for file system and packages to be installed (several hours) Actual results: This process takes several hours. Expected results: On a Dell 1850 Perc4 this process takes 5-10 minutes. Additional info: Perc 5 controller card on these newer generation dell systems.
Changing component to virt-manager
Please re-run with --debug argument & attached all the debuging output it generates, and the full command line used to invoke virt-install.
No output is provided, the process just sits there "Formatting / file system..." The pid sits in a wait status: F S UID PID PPID C PRI NI ADDR SZ WCHAN TTY TIME CMD 4 S 0 22301 22283 0 75 0 - 3665 wait pts/9 0:00 /usr/bin/python -tt /usr/sbin/virt-install --debug Process 22301 attached - interrupt to quit waitpid(22419, Process 22419 attached - interrupt to quit select(6, [0 5], NULL, NULL, NULL
Over a 15 minute window it went from 36 to 38% with the following activity on the console strace: strace -p 22419 Process 22419 attached - interrupt to quit select(6, [0 5], NULL, NULL, NULL) = 1 (in [5]) read(5, "\33[13;32H \33[7C7\33[13;68H", 512) = 22 write(1, "\33[13;32H \33[7C7\33[13;68H", 22) = 22 select(6, [0 5], NULL, NULL, NULL) = 1 (in [5]) read(5, "\33[13;40H8\33[13;68H", 512) = 17 write(1, "\33[13;40H8\33[13;68H", 17) = 17 select(6, [0 5], NULL, NULL, NULL
This extends to IO availability for Xen guests -- try a mysql import of a database or a yum update and watch iowait sit at 100% on one of these PERC/5 enabled systems.
Ok, this sounds like its nothing todo with virt-install then, rather the kernel I/O system. Changing to kernel-xen
Thank you, we have live systems available to provide any debugging information necessary. Just let us know what you need. We've tested this through to 2.6.18-8.1.8 Another side effect of the problem is that when any type of load is put on dom0, and you try to create a guest `xm create /etc/xen/something` the process: 1) is extremely slow 2) typically will return a kernel panic and require several additional attempts: "Kernel panic - not syncing: Attempted to kill init!" Again this is reproduceable on a Dell 1435SC with the PERC/5 controller card. Thank you for anything you can do.
Has any progress been made on this matter?
This request was evaluated by Red Hat Product Management for inclusion, but this component is not scheduled to be updated in the current Red Hat Enterprise Linux release. If you would like this request to be reviewed for the next minor release, ask your support representative to set the next rhel-x.y flag to "?".
I believe this to be an absurd response but it is your distribution to do what you will with it. -Thomas
Hi, I had the same problem with big slowdown on filesystem formatting. I noticed that this happened only when using sparse image files. Launching virt-install with "--nonsparse" or creating a non sparse file with "dd" workarounded this problem. I hope this can help finding the real problem. Bye!
Installing a guest to a sparse file saves disk space, but (because no space has been allocated to the file) ends up turning every asynchronous IO into a synchronous one. This will cause the slowdown you have seen. Using nonsparse files (now the default, I believe) will get you running at a normal speed. The default was changed to nonsparse a while ago under another bug number.