Bug 241708 - virt-install on dell 860/sc1435 takes several hours to complete
virt-install on dell 860/sc1435 takes several hours to complete
Status: CLOSED WORKSFORME
Product: Red Hat Enterprise Linux 5
Classification: Red Hat
Component: kernel-xen (Show other bugs)
5.0
i686 Linux
medium Severity medium
: ---
: ---
Assigned To: Xen Maintainance List
Martin Jenner
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2007-05-29 14:03 EDT by Thomas
Modified: 2009-01-12 12:14 EST (History)
4 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2009-01-12 12:14:34 EST
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Thomas 2007-05-29 14:03:04 EDT
Description of problem:

Building Xen guest nodes on Dell 860 and SC1435 hardware with PERC 5 controllers
takes 4-5 hours using virt-install.

Process begins to hang on filesystem formatting, eventually finishes.

virt-install standard 512MB guest, no full virtualization and installation over nfs

The problem is can be reproduced every single time.  The problem does not seem
to occur on older Dell 1850 (Perc 4) hardeware.

Module                  Size  Used by
backupdriver          305848  0 
nfs                   224961  0 
lockd                  58697  1 nfs
fscache                20449  1 nfs
nfs_acl                 7617  1 nfs
xt_physdev              6993  10 
iptable_filter          7105  1 
ip_tables              17029  1 iptable_filter
x_tables               17349  2 xt_physdev,ip_tables
bridge                 53725  1 xt_physdev
netloop                10817  0 
netbk                  78017  0 [permanent]
blktap                385125  22 [permanent]
blkbk                  21089  0 [permanent]
autofs4                23749  2 
hidp                   23105  2 
rfcomm                 42457  0 
l2cap                  29505  10 hidp,rfcomm
bluetooth              53925  5 hidp,rfcomm,l2cap
sunrpc                142973  4 nfs,lockd,nfs_acl
ipv6                  250369  14 
video                  19269  0 
sbs                    18533  0 
i2c_ec                  9025  1 sbs
button                 10705  0 
battery                13637  0 
asus_acpi              19289  0 
ac                      9157  0 
parport_pc             29157  0 
lp                     15849  0 
parport                37641  2 parport_pc,lp
joydev                 13185  0 
sg                     35933  0 
pcspkr                  7105  0 
tg3                   101317  0 
i2c_i801               11341  0 
i2c_core               23745  2 i2c_ec,i2c_i801
serial_core            23617  0 
serio_raw              10693  0 
dm_snapshot            20581  0 
dm_zero                 6209  0 
dm_mirror              29841  0 
dm_mod                 56537  8 dm_snapshot,dm_zero,dm_mirror
mptsas                 29001  2 
mptscsih               26177  1 mptsas
mptbase                53089  2 mptsas,mptscsih
scsi_transport_sas     29249  1 mptsas
sd_mod                 22977  3 
scsi_mod              130893  5 sg,mptsas,mptscsih,scsi_transport_sas,sd_mod
ext3                  123081  2 
jbd                    56553  1 ext3
ehci_hcd               33229  0 
ohci_hcd               23645  0 
uhci_hcd               25677  0 



Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1.  build vps node virt-install
2.  create new file system and select packages
3.  wait for file system and packages to be installed (several hours)
  
Actual results:

This process takes several hours.

Expected results:

On a Dell 1850 Perc4 this process takes 5-10 minutes.

Additional info:  Perc 5 controller card on these newer generation dell systems.
Comment 2 Michael Hideo 2007-07-09 01:53:04 EDT
Changing component to virt-manager
Comment 3 Daniel Berrange 2007-07-09 10:01:39 EDT
Please re-run  with  --debug argument & attached all the debuging output it
generates, and the full command line used to invoke  virt-install.

Comment 4 Thomas 2007-07-17 09:30:53 EDT
No output is provided, the process just sits there "Formatting / file system..."

The pid sits in a wait status:

F S   UID   PID  PPID  C PRI  NI ADDR SZ WCHAN  TTY        TIME CMD
4 S     0 22301 22283  0  75   0 -  3665 wait   pts/9      0:00 /usr/bin/python
-tt /usr/sbin/virt-install --debug

Process 22301 attached - interrupt to quit
waitpid(22419, 

Process 22419 attached - interrupt to quit
select(6, [0 5], NULL, NULL, NULL
Comment 5 Thomas 2007-07-17 09:32:10 EDT
Over a 15 minute window it went from 36 to 38% with the following activity on
the console strace:

strace -p 22419
Process 22419 attached - interrupt to quit
select(6, [0 5], NULL, NULL, NULL)      = 1 (in [5])
read(5, "\33[13;32H \33[7C7\33[13;68H", 512) = 22
write(1, "\33[13;32H \33[7C7\33[13;68H", 22) = 22
select(6, [0 5], NULL, NULL, NULL)      = 1 (in [5])
read(5, "\33[13;40H8\33[13;68H", 512)   = 17
write(1, "\33[13;40H8\33[13;68H", 17)   = 17
select(6, [0 5], NULL, NULL, NULL
Comment 6 Thomas 2007-08-30 00:43:32 EDT
This extends to IO availability for Xen guests -- try a mysql import of a
database or a yum update and watch iowait sit at 100% on one of these PERC/5
enabled systems.
Comment 7 Daniel Berrange 2007-08-30 08:32:10 EDT
Ok, this sounds like its nothing todo with virt-install then, rather the kernel
I/O system. Changing to kernel-xen
Comment 8 Thomas 2007-08-30 08:41:53 EDT
Thank you, we have live systems available to provide any debugging information
necessary.  Just let us know what you need.

We've tested this through to 2.6.18-8.1.8

Another side effect of the problem is that when any type of load is put on dom0,
and you try to create a guest `xm create /etc/xen/something` the process:

1)  is extremely slow
2)  typically will return a kernel panic and require several additional attempts:

"Kernel panic - not syncing: Attempted to kill init!"

Again this is reproduceable on a Dell 1435SC with the PERC/5 controller card.

Thank you for anything you can do.
Comment 9 Thomas 2007-10-22 22:12:29 EDT
Has any progress been made on this matter?
Comment 10 RHEL Product and Program Management 2008-02-01 17:41:38 EST
This request was evaluated by Red Hat Product Management for
inclusion, but this component is not scheduled to be updated in
the current Red Hat Enterprise Linux release. If you would like
this request to be reviewed for the next minor release, ask your
support representative to set the next rhel-x.y flag to "?".
Comment 11 Thomas 2008-02-01 17:45:52 EST
I believe this to be an absurd response but it is your distribution to do what
you will with it.

-Thomas
Comment 12 Simone Gotti 2008-08-08 07:01:16 EDT
Hi,

I had the same problem with big slowdown on filesystem formatting. I noticed that this happened only when using sparse image files.
Launching virt-install with "--nonsparse" or creating a non sparse file with "dd" workarounded this problem.

I hope this can help finding the real problem.

Bye!
Comment 13 Rik van Riel 2009-01-12 12:14:34 EST
Installing a guest to a sparse file saves disk space, but (because no space has been allocated to the file) ends up turning every asynchronous IO into a synchronous one.

This will cause the slowdown you have seen.  Using nonsparse files (now the default, I believe) will get you running at a normal speed.  The default was changed to nonsparse a while ago under another bug number.

Note You need to log in before you can comment on or make changes to this bug.