I'm attempting to reserve the machine now, and will see what happens when I attempt to install manually.

The install is hanging when formatting the disk. dmesg on dom0 shows that the
blktap driver isn't happy

blktap: ring-ref 770, event-channel 7, protocol 1 (x86_64-abi)
blk_tap: invalid kernel buffer -- could not remap it
blk_tap: invalid user buffer -- could not remap it
blk_tap: invalid kernel buffer -- could not remap it
blk_tap: invalid user buffer -- could not remap it
blk_tap: invalid kernel buffer -- could not remap it
blk_tap: invalid user buffer -- could not remap it
blk_tap: invalid kernel buffer -- could not remap it
blk_tap: invalid user buffer -- could not remap it
blk_tap: Reached Fail_flush
... and more ...

and xm dmesg shows the hypervisor is having trouble with several memory pages.
xm dmesg also shows messages repeated like this

(XEN) sysctl.c:51: Allowing physinfo call with newer ABI version

I'm not sure if the ABI stuff is related or even an issue, but this is all
interesting stuff.

I poked in the code and don't see how the ABI logs could be related, but there's certainly a lot on this machine.

I've also found this is reproducible with the latest RHEL5 (-178). Both dom0 and domU were updated. The logs shown above appeared much less frequently though.

I've also found it's not really hung. It's just super, super slow. Probably the RHTS workflow fails because some watchdog trips before it completes. The formatting takes around 15-20 minutes, then the install takes up to 2 hours. The same results for both 64b and 32b guests.

I've also seen a log like this

(XEN) grant_table.c:154:d0 Increased maptrack size to 3 frames.

in xm dmesg. This looks fine, like it's doing what it should, but it's probably from the same hypervisor call that was returning the maps that blktap is complaining about in the other logs.

I'm going to get some disk I/O metrics on bare-metal, dom0, and the guest.

It's definitely something funny with blktap. If I ensure that I'm not using it by seting up my image file to be on a loopback device and then passing that in for the installation, so I have a phy device in my config, then installs are back to normal speed. So the question is if the problem is in the blktap driver, or the hypervisor grant tables, or a combo of them both.

The question is also how this particular machine is exposing this problem. Since we don't generally see this issue.

I got this machine back and started looking at this again. I installed a vm quickly by using a loop device for its disk. Then I attached another disk using blktap to the vm and compared the performance for both using fs_mark.

#### blktap #####

FSUse%        Count         Size    Files/sec     App Overhead
     8         1000        10240         37.6            14742

CREAT (Min/Avg/Max)        WRITE (Min/Avg/Max)        FSYNC (Min/Avg/Max)
16     33     7130         21       31     58         18293  26481  63295

SYNC (Min/Avg/Max)        CLOSE (Min/Avg/Max)       UNLINK (Min/Avg/Max)
0      0        0          2        3     10         15       17     52

------------------------------------------------------------------------------

#### blkback #####

FSUse%        Count         Size    Files/sec     App Overhead
    17         1000        10240       3478.1            14343

CREAT (Min/Avg/Max)        WRITE (Min/Avg/Max)        FSYNC (Min/Avg/Max)
14     26       44         21     28      51          187   214     4138

SYNC (Min/Avg/Max)        CLOSE (Min/Avg/Max)       UNLINK (Min/Avg/Max)
0      0        0          2     4       15          15     16       42


The numbers are quite similar for most measurements, but the FSYNC times are on average 120 times longer for blktap than blkback, which causes it to be 100 times slower (files/sec) than blkback.

I currently have -183 kernel-xen and -102 xen on the host. The guest is -182. I don't get any logs in dmesg or xm dmesg with these revs.

*** Bug 553719 has been marked as a duplicate of this bug. ***

This has popped up on two machines from different vendors. The common denominator is they both have Intel SATA IDE controllers (which uses the ata_piix module). To test that, I reserved a third machine from yet another vendor that has a similar controller, and sure enough it reproduced again.

So it looks like something with this controller causes a serious slowdown with blktap.

It is no surprise that blkback is faster, since it caches all the data in memory while blktap flushes it to disk.  For bare-metal dell-pe1950-01 I'm getting

[root@dell-pe1950-01 ~]# fs_mark -d test -d test2 -s 51200 -n 4096

#  fs_mark  -d  test  -d  test2  -s  51200  -n  4096 
#	Version Version 3.2, 2 thread(s) starting at Tue Feb 23 12:18:01 2010
#	Sync method: INBAND FSYNC: fsync() per file in write loop.
#	Directories:  no subdirectories used
#	File names: 40 bytes long, (16 initial bytes of time stamp with 24 random bytes at end of name)
#	Files info: size 51200 bytes, written with an IO size of 16384 bytes per write
#	App overhead is time in microseconds spent in the test not doing file writing related system calls.

FSUse%        Count         Size    Files/sec     App Overhead
     7         8192        51200         39.0            73261

so the controller is slow and blkback is working around the slowness at the cost of safety.

What bare-metal kernel/drivers did you use? We should check the numbers with the latest drivers we can find for this controller, which would ensure the controller doesn't just look slow because the drivers don't know how to use it.

We also shouldn't lose sight of the data points we have in comment 2. I don't think blktap would or should complain this way just because the controller is slow. So in the least we may need to improve our handling of these types of controllers.

In any respect, we should answer a few more questions better, and this bug is as good as any place to do it.

Technical note added. If any revisions are required, please edit the "Technical Notes" field
accordingly. All revisions will be proofread by the Engineering Content Services team.

New Contents:
Under certain, yet to be determined circumstances, which are likely SATA controller related, blktap will not function properly. The primary symptom is very, very slow disk I/O making it appear that the guest is hung. To work around this issue all guests should be installed using a physical disk, such as a real partition or a logical volume.

Technical note updated. If any revisions are required, please edit the "Technical Notes" field
accordingly. All revisions will be proofread by the Engineering Content Services team.

Diffed Contents:
@@ -1 +1 @@
-Under certain, yet to be determined circumstances, which are likely SATA controller related, blktap will not function properly. The primary symptom is very, very slow disk I/O making it appear that the guest is hung. To work around this issue all guests should be installed using a physical disk, such as a real partition or a logical volume.+blktap may not function as expected, resulting in slow disk I/O causing the guest to operate slowly also. To work around this issue guests should be installed using a physical disk (i.e. a real partition or a logical volume). (BZ#545692)

I think it's just that the hardware is bad?  blktap is less than 10% slower than bare-metal (37.6 vs 39 on fs_mark).  I'm closing this as NOTABUG.

Clearing out old flags for reporting purposes.

Chris Lalancette

*** Bug 501162 has been marked as a duplicate of this bug. ***