This service will be undergoing maintenance at 00:00 UTC, 2016-08-01. It is expected to last about 1 hours

Bug 526549

Summary: rawhide/i386 kvm host corrupts data of guests
Product: [Fedora] Fedora Reporter: James Laska <jlaska>
Component: kernelAssignee: chellwig <chellwig>
Status: CLOSED RAWHIDE QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: high Docs Contact:
Priority: high    
Version: rawhideCC: agk, bmarzins, bmr, clalance, dougsland, drjones, dwysocha, gansalmon, heinzm, itamar, jforbes, jturner, kernel-maint, lvm-team, markmc, mbroz, msnitzer, notting, prajnoha, prockai, tcallawa, vanmeeuwen+fedora, virt-maint
Target Milestone: ---Keywords: Reopened
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2009-11-04 14:33:07 EST Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Bug Depends On:    
Bug Blocks: 498968, 498969    
Attachments:
Description Flags
updates.img
none
anacdump.txt (using updates.img)
none
autoqa-rats_install-wedge.patch
none
rats-logs.tgz (anaconda-12.39) none

Description James Laska 2009-09-30 15:52:10 EDT
Description of problem:

All i386 and x86_64 rats_install kickstart tests are timing out because anaconda is prompting for some swap operation.  This seems related to commit 86330e68477dd4e89f462a966ef900b38b2b5ab6.

Version-Release number of selected component (if applicable):

 * anaconda-12.32

How reproducible:

 * 100%

Steps to Reproduce:
1. git clone git://git.fedorahosted.org/autoqa.git
2. cd autoqa/tests/rats_install
3. python install.py -a i386 http://download.fedoraproject.org/pub/fedora/linux/development/i386/os
  
Actual results:

       ┌───────────────────────────┤ Error ├───────────────────────────┐
       │                                                               │
       │ The swap device:                                              │
       │                                                               │
       │      /dev/mapper/VolGroup-lv_swap                             │
       │                                                               │        
       │ does not contain a support swap volume.  In order to continue │        
       │ installation, you will need to format the device or skip it.  │        
       │                                                               │        
       │      ┌──────┐            ┌────────┐            ┌──────┐       │        
       │      │ (jlaska@localhost)-(2/pts/5)-(09/30/2009 03:22pm)-     │        
(~)>   │      └──────┘            └────────┘            └──────┘       │
       │                                                               │
       │                                                               │
       └───────────────────────────────────────────────────────────────┘

Expected results:

 * kickstart install with 'autopart' should not prompt for feedback?

Additional info:

 * ks.cfg - http://git.fedorahosted.org/git/?p=autoqa.git;a=blob;f=tests/rats_install/ks.cfg;h=0fa5954c08431ca3562605eb8ca2a2bfcd5e00e4;hb=HEAD
Comment 1 David Cantrell 2009-09-30 22:54:25 EDT
So I will agree that prompting during a kickstart install like this is annoying, but I'm not sure if the kickstart and autopart conditions should skip over these warnings.  We generally take kickstart commands and prompt if necessary.  Personally, I think the prompt should be there, but should only be suppressed if you're doing a cmdline install, which would not be interactive.  We still consider kickstart installs interactive, after all.

Thoughts?
Comment 2 James Laska 2009-10-01 08:23:32 EDT
I don't mind the prompt for an exceptional condition ... you are right, that makes sense.  

This test doesn't seem to be too unusual ... just a small @base virt kvm install.  If there's something awkward about our test environment, we can probably tweak to avoid the problem, but there seems to be something going wrong under the covers that shouldn't happen.  For example, why is it asking me to format swap ... didn't I already request that with 'autopart + clearpart --all --initlabel'?  

Is there something additional we need to add to our kickstart to quiesce the prompt?  

Is our virt environment contributing to the frequency of this failure?  I *only* see it in our rats_install tests ... I'm unclear why.

Do you see this when you run the rats_install command on a kvm-capable system (see 'steps to reproduce')?
Comment 3 Chris Lumens 2009-10-01 10:20:33 EDT
We will definitely hear from RHEL6 customers if there's not a way to automatically skip this warning during a kickstart install.
Comment 4 David Cantrell 2009-10-01 20:56:20 EDT
James,

I have a hunch this might be related to some of the odd disk errors we see on virt systems in general.  I'm trying to fire up rats_install, but I don't think I have all the pieces in place to run it.  Here's the last error I get:

$ sudo python install.py -a i386 http://tenon.honolulu.burdell.org/rawhide/i386
== installer image sanity test ==
.treeinfo                                                | 1.4 kB     00:00     
CHECK: [general] section in .treeinfo: OK
CHECK: arch = i386 in [general]: OK
CHECK: [images-i386] section: OK
CHECK: kernel: OK
CHECK: initrd: OK
CHECK: mainimage item in [stage2] section: OK
stage2 mainimage: install.img
vmlinuz                                                  | 3.5 MB     00:00     
CHECK: vmlinuz non-empty: OK
initrd.img                                               |  22 MB     00:00     
CHECK: initrd.img non-empty: OK
boot.iso                                                 | 201 MB     00:08     
CHECK: boot.iso non-empty: OK
install.img                                              | 113 MB     00:04     
CHECK: install.img non-empty: OK
CHECK: [checksum] section: OK
CHECK: boot.iso sha256: OK
CHECK: vmlinuz sha256: OK
CHECK: install.img sha256: OK
CHECK: initrd.img sha256: OK
sanity check complete.
tree timestamp: Thu Oct  1 02:46:17 2009 (local time)
TEST RESULT: images: OK
prep:
  uncompressing initrd
  saving original initrd
  writing kickstart
  adding kickstart to initrd
  recompressing initrd
  new guest name: RATS
starting virt guest
ERROR    Prompting disabled, but yes/no was requested. Try --force to force 'yes' for such prompts. Prompt was: The requested volume capacity will exceed the available pool space when the volume is fully allocated. (8192 M requested capacity > 2646 M available) Do you really want to use the disk (yes or no)?
error: failed to get domain 'RATS'
error: Domain not found

error: failed to get domain 'RATS'
error: Domain not found

error: failed to get vol '/var/lib/libvirt/images/RATS.img'
error: invalid storage volume pointer in no storage vol with matching path

Traceback (most recent call last):
  File "install.py", line 310, in <module>
    guest = create_guest(rats.scratchdir, url=baseuri)
  File "install.py", line 152, in create_guest
    guest.create(location=imgdir)
  File "/home/dcantrel/autoqa/tests/rats_install/virtguest.py", line 168, in create
    raise RuntimeError, "virt-install failed: %i" % retval
RuntimeError: virt-install failed: 1

I'd like to reproduce what you're seeing.  It is something that shouldn't be happening, but I want to find the root cause.
Comment 5 James Laska 2009-10-12 11:21:28 EDT
Sorry for delay David.  So it seems like rats_install is asking for a 8G disk for the guest, but only ~ 2G are available?  Is that right?

If that's the case, you can try manually adjusting the disksize to 2G ... http://pastie.org/651589

If that's not the case, you can try adding the '--force' argument to virt-install by making this change: http://pastie.org/651587
Comment 6 David Cantrell 2009-10-13 23:47:37 EDT
Whoops, my bad.  Available space in /var/lib/libvirt/images.  Of course.  I added another pool for storage on an external 500GB disk and things got a little farther.  Now I see this:

----------

$ sudo python install.py -a i386 http://tenon.honolulu.burdell.org/rawhide/i386
== installer image sanity test ==
.treeinfo                                                | 1.4 kB     00:00     
CHECK: [general] section in .treeinfo: OK
CHECK: arch = i386 in [general]: OK
CHECK: [images-i386] section: OK
CHECK: kernel: OK
CHECK: initrd: OK
CHECK: mainimage item in [stage2] section: OK
stage2 mainimage: install.img
vmlinuz                                                  | 3.5 MB     00:00     
CHECK: vmlinuz non-empty: OK
initrd.img                                               |  22 MB     00:00     
CHECK: initrd.img non-empty: OK
boot.iso                                                 | 201 MB     00:09     
CHECK: boot.iso non-empty: OK
install.img                                              | 113 MB     00:08     
CHECK: install.img non-empty: OK
CHECK: [checksum] section: OK
CHECK: boot.iso sha256: OK
CHECK: vmlinuz sha256: OK
CHECK: install.img sha256: OK
CHECK: initrd.img sha256: OK
sanity check complete.
tree timestamp: Tue Oct 13 01:05:01 2009 (local time)
TEST RESULT: images: OK
prep:
  uncompressing initrd
  saving original initrd
  writing kickstart
  adding kickstart to initrd
  recompressing initrd
  new guest name: RATS
starting virt guest
ERROR    Error with storage parameters: There is not enough free space to create the disk. 8192 M requested > 2044 M available
error: failed to get domain 'RATS'
error: Domain not found

error: failed to get domain 'RATS'
error: Domain not found

error: failed to get vol '/var/lib/libvirt/images/RATS.img'
error: invalid storage volume pointer in no storage vol with matching path

Traceback (most recent call last):
  File "install.py", line 310, in <module>
    guest = create_guest(rats.scratchdir, url=baseuri)
  File "install.py", line 152, in create_guest
    guest.create(location=imgdir)
  File "/home/dcantrel/autoqa/tests/rats_install/virtguest.py", line 168, in create
    raise RuntimeError, "virt-install failed: %i" % retval
RuntimeError: virt-install failed: 1

----------

I added another storage pool in virt-manager for localhost.  I added a directory on an external 500GB USB disk, but I still got the failure above.  rats_install is still trying to use /var/lib/libvirt/images/RATS.img instead of another location in the pool space.  I made the following change and things started to work:

diff --git a/tests/rats_install/virtguest.py b/tests/rats_install/virtguest.py
index 84d3671..86605fe 100644
--- a/tests/rats_install/virtguest.py
+++ b/tests/rats_install/virtguest.py
@@ -48,7 +48,7 @@ class VirtGuest(object):
     '''dumb little class to hold data about virt guests'''
     def __init__(self, name, disk=None, wait=60, network='default'):
         self.name = name
-        self.disk = disk or '/var/lib/libvirt/images/%s.img' % name
+        self.disk = disk or '/media/passport/virt/%s.img' % name
         self.wait = wait
         self.network = network
         self.birthdate = 0

----------

Probably something for the rats_install TODO list.  Working with the virt storage pools would be cool.
Comment 7 David Cantrell 2009-10-13 23:58:58 EDT
Created attachment 364681 [details]
updates.img

James,

Can you try this updates.img with the latest rawhide tree and see if it fixes up this prompting issue?

Here's what I did:

diff --git a/storage/__init__.py b/storage/__init__.py
index 9a3752e..25db6cf 100644
--- a/storage/__init__.py
+++ b/storage/__init__.py
@@ -1640,6 +1640,11 @@ class FSSet(object):
 
     def turnOnSwap(self, anaconda, upgrading=None):
         def swapErrorDialog(msg, device):
+            if (anaconda.isKickstart or flags.cmdline.has_key("cmdline")) and \
+               anaconda.id.storage.doAutoPart:
+                device.format.create(force=True)
+                return True
+
             if not anaconda.intf:
                 sys.exit(0)
Comment 8 James Laska 2009-10-15 14:29:56 EDT
Created attachment 364974 [details]
anacdump.txt (using updates.img)

Thanks for the updates.img David.  This leads to the original problem ...

After about 3 minutes of the following, it eventually tracebacks.

14:24:57 DEBUG   : SwapSpace.create: device: /dev/mapper/VolGroup-lv_swap ; status: False ; type: swap ;
14:24:57 DEBUG   : LVMLogicalVolumeDevice.setup: VolGroup-lv_swap ; status: True ;
14:24:57 DEBUG   : SwapSpace.setup: device: /dev/mapper/VolGroup-lv_swap ; status: False ; type: swap ;
14:24:57 DEBUG   : SwapSpace.setup: device: /dev/mapper/VolGroup-lv_swap ; status: False ; type: swap ;
14:24:57 DEBUG   : SwapSpace.create: device: /dev/mapper/VolGroup-lv_swap ; status: False ; type: swap ;
14:24:57 DEBUG   : SwapSpace.teardown: device: /dev/mapper/VolGroup-lv_swap ; status: False ; type: swap ;
14:24:57 DEBUG   : SwapSpace.create: device: /dev/mapper/VolGroup-lv_swap ; status: False ; type: swap ;
14:24:57 DEBUG   : LVMLogicalVolumeDevice.setup: VolGroup-lv_swap ; status: True ;
14:24:57 DEBUG   : SwapSpace.setup: device: /dev/mapper/VolGroup-lv_swap ; status: False ; type: swap ;
14:24:57 DEBUG   : SwapSpace.setup: device: /dev/mapper/VolGroup-lv_swap ; status: False ; type: swap ;
14:24:57 DEBUG   : SwapSpace.create: device: /dev/mapper/VolGroup-lv_swap ; status: False ; type: swap ;
14:24:57 DEBUG   : SwapSpace.teardown: device: /dev/mapper/VolGroup-lv_swap ; status: False ; type: swap ;
14:24:57 DEBUG   : SwapSpace.create: device: /dev/mapper/VolGroup-lv_swap ; status: False ; type: swap ;
14:24:57 DEBUG   : LVMLogicalVolumeDevice.setup: VolGroup-lv_swap ; status: True ;
14:24:57 DEBUG   : SwapSpace.setup: device: /dev/mapper/VolGroup-lv_swap ; status: False ; type: swap ;
14:24:57 DEBUG   : SwapSpace.setup: device: /dev/mapper/VolGroup-lv_swap ; status: False ; type: swap ;
14:24:57 INFO    : Running kickstart %%traceback script(s)
14:24:57 INFO    : All kickstart %%traceback script(s) have been run
14:24:57 CRITICAL: anaconda 12.38 exception report
Traceback (most recent call first):
  File "/tmp/updates/storage/devicelibs/swap.py", line 91, in swapon
    raise SwapError("swapon failed for '%s'" % device)
  File "/tmp/updates/storage/formats/swap.py", line 119, in setup
    swap.swapon(self.device, priority=self.priority)
  File "/tmp/updates/storage/__init__.py", line 1682, in turnOnSwap
    device.format.setup()
  File "/tmp/updates/storage/__init__.py", line 1047, in turnOnSwap
    self.fsset.turnOnSwap(self.anaconda, upgrading=upgrading)
  File "/usr/lib/anaconda/packages.py", line 189, in turnOnFilesystems
    anaconda.id.storage.turnOnSwap()
  File "/usr/lib/anaconda/dispatch.py", line 200, in moveStep
    rc = stepFunc(self.anaconda)
  File "/usr/lib/anaconda/dispatch.py", line 123, in gotoNext
    self.moveStep()
  File "/usr/lib/anaconda/dispatch.py", line 222, in currentStep
    self.gotoNext()
  File "/usr/lib/anaconda/text.py", line 439, in run
    (step, instance) = anaconda.dispatch.currentStep()
  File "/usr/bin/anaconda", line 968, in <module>
    anaconda.intf.run(anaconda)
SwapError: swapon failed for '/dev/mapper/VolGroup-lv_swap'
Comment 9 David Cantrell 2009-10-20 23:33:48 EDT
Created attachment 365456 [details]
autoqa-rats_install-wedge.patch

James,

I had to wedge rats_install to make it work on my system, but the install completed using the updates.img I attached to this bug report.  I just copied it in to the images/ subdirectory of the rawhide tree I was specifying on the rats_install command line and it worked fine.

I removed some of the rats_install code that turns off the VM console so I could see what was happening.  I also hit problems with the timeout stuff, as you'll notice in my patch.

But anaconda installed Fedora without the swap problem once I added the updates.img to the imagess/ subdirectory.  Are you still seeing this problem?
Comment 10 David Cantrell 2009-10-21 21:46:37 EDT
To further add to the confusion, I just did a rats_install using my wedge patch, the current i386 rawhide tree, but did NOT include my patch from comment #7.  Installation worked fine.  No swap errors.

My rats_install patch doesn't really change the way you are exec'ing anaconda, it just changes how you start the virt-install and lets me monitor the GUI console.

I hate to do this, but I have to say it works for me.
Comment 11 James Laska 2009-10-26 13:52:23 EDT
Created attachment 366132 [details]
rats-logs.tgz (anaconda-12.39)

Testing update, Using anaconda-12.39-1 (in rawhide) I'm still able to reproduce this failure.  I've reinstalled the test system in question and the problem remains.

Test log - https://fedorahosted.org/pipermail/autoqa-results/2009-October/001787.html

# cat /tmp/rats/log/stage2.log

Welcome to Fedora for i386



       ┌───────────────────────────┤ Error ├───────────────────────────┐
       │                                                               │
       │ The swap device:                                              │
       │                                                               │
       │      /dev/mapper/VolGroup-lv_swap                             │
       │                                                               │        
       │ does not contain a support swap volume.  In order to continue │        
       │ installation, you will need to format the device or skip it.  │        
       │                                                               │        
       │      ┌──────┐            ┌────────┐            ┌──────┐       │        
       │      │ [root@caius ~]#   │ Format │            │ Exit │       │        
       │      └──────┘            └────────┘            └──────┘       │
       │                                                               │
       │                                                               │
       └───────────────────────────────────────────────────────────────┘

See attached log files...

# tar -ztvf Desktop/rats-logs.tgz 
drwxr-xr-x root/root         0 2009-10-26 13:34 tmp/rats/log/
-rw-r--r-- root/root       561 2009-10-26 13:33 tmp/rats/log/ks.cfg
-rw-r--r-- root/root      5749 2009-10-26 13:34 tmp/rats/log/stage1.log
-rw-r--r-- root/root     14247 2009-10-26 13:34 tmp/rats/log/boot.log
-rw-r--r-- root/root       288 2009-10-26 13:34 tmp/rats/log/create-guest.sh
drwxr-xr-x root/root         0 2009-10-26 13:34 tmp/rats/log/minimon/
-rw-r--r-- root/root     37842 2009-10-26 13:34 tmp/rats/log/minimon/anaconda.log
-rw-r--r-- root/root     32237 2009-10-26 13:34 tmp/rats/log/minimon/storage.log
-rw-r--r-- root/root      5004 2009-10-26 13:34 tmp/rats/log/minimon/program.log
-rw-r--r-- root/root     29184 2009-10-26 13:34 tmp/rats/log/minimon/syslog
-rw-r--r-- root/root      4771 2009-10-26 13:34 tmp/rats/log/stage2.log
Comment 12 James Laska 2009-10-27 09:19:26 EDT
After talking w/ dcantrell on IRC, it appears that using rats_install to test an i386 install on an x86_64 host works fine.  The problem seems to be reproducible only on an i386 host.

I'd like to reopen this issue awaiting further testing with this new detail.
Comment 13 James Laska 2009-10-27 17:03:56 EDT
Just reproduced on an i386 guest on an i386 host without using rats_install/install.py

 1. Start virt-install

$ virt-install --name RATS --ram 512 --vcpus 1 --os-type linux --os-variant fedora11 --extra-args  "serial console=ttyS0 ip=dhcp ksdevice=link lang=en keymap=us vnc" --disk path=/var/lib/libvirt/images/RATS.img,size=8,sparse=false --network network:default --location http://download.fedoraproject.org/pub/fedora/linux/development/i386/os --accelerate --nographics

 2. Connect to the VNC session of the guest
 3. Continue through the install, at the disk partitioning screen, select [X] Encrypt system, and click Next
 4. Dialog presents as noted in comment#0.

 - Choose to 'Format' the drive ... nothing happens.
 - Choose to 'Skip' and the installation fails "An error occured mounting device /dev/mapper/VolGroup-lv_root as /: mount failed (2,None).  This is a fatal error and the install cannot continue"

dcantrell, are you able to reproduce on a i386 host?
Comment 14 Chris Lumens 2009-10-28 16:26:16 EDT
After a lot of investigation, I think I have a bit of a handle on this problem.  You don't need to do encryption to reproduce this, so let's not worry about that.  Then, looking at the /tmp/program.log, I see:

Running... ['lvm', 'lvcreate', '-L', '1024m', '-n', 'lv_swap', 'VolGroup']
  Logical volume "lv_swap" created
Running... ['udevadm', 'settle', '--timeout=10']
Running... ['mkswap', '/dev/mapper/VolGroup-lv_swap']
mkswap: /dev/mapper/VolGroup-lv_swap: warning: don't erase bootbits sectors
        on whole disk. Use -f to force.
Setting up swapspace version 1, size = 1048572 KiB
no label, UUID=883eb8af-653e-4e03-ba6c-74a9bfa2eadb

Well that's weird.  What if I try to swapon what mkswap made on tty2 myself:

sh-4.0# swapon /dev/mapper/VolGroup-lv_swap 
swapon: /dev/mapper/VolGroup-lv_swap: read swap header failed: Invalid argument

Additionally, looking back in the syslog I can also see:

<6>EXT4-fs: mballoc: 0 blocks 0 reqs (0 success)
<6>EXT4-fs: mballoc: 0 extents scanned, 0 goal hits, 0 2^N hits, 0 breaks, 0 lost
<6>EXT4-fs: mballoc: 0 generated and it took 0
<6>EXT4-fs: mballoc: 0 preallocated, 0 discarded
<2>EXT3-fs error (device dm-1): ext3_check_descriptors: Block bitmap for group 0 not in group (block 1976475243)!
<3>EXT3-fs: group descriptors corrupted!

So, something funny is definitely going on in the way that LVs are getting created - either things are overlapping, or lvm is not erroring out when attempting to create one of the LVs that won't fit, or something along those lines.  So, mkswap doesn't make swap in the right place, so attempting to read the swap header fails, so we end up with this UnknownSwapError.  This also explains why none of the buttons work correctly.  Reassigning to lvm for now for additional investigation.
Comment 15 Milan Broz 2009-10-29 06:22:57 EDT
This is neither lvm2 nor anaconda bug, there is some strange problem with underlying device.

See - volumes are properly mapped, but after successful mkswap there is no signature:

sh-4.0# blkid /dev/vda2
/dev/vda2: UUID="YgCYXL-8VMS-axsL-6qmm-osd2-sbIR-9vIbS0" TYPE="LVM2_member" 
sh-4.0# lvm lvs
  LV      VG       Attr   LSize Origin Snap%  Move Log Copy%  Convert
  lv_root VolGroup -wi--- 6.80G                                      
  lv_swap VolGroup -wi--- 1.00G                                      
sh-4.0# lvm vgchange -a y
  2 logical volume(s) in volume group "VolGroup" now active
sh-4.0# blkid /dev/VolGroup/lv_swap
sh-4.0# mkswap /dev/VolGroup/lv_swap
mkswap: /dev/VolGroup/lv_swap: warning: don't erase bootbits sectors
        on whole disk. Use -f to force.                             
Setting up swapspace version 1, size = 1048572 KiB                  
no label, UUID=053d3767-e126-49c9-8eac-ffc197681ff9                 
sh-4.0# blkid /dev/VolGroup/lv_swap


I tried some reproducer (running on the same PV):
#!/bin/bash

lvm lvremove -f VolGroup

for i in 1 2 3 4 ; do
        lvm lvcreate -L 1.80G -n $i VolGroup
        mkswap /dev/VolGroup/$i
done
lvm lvs -o name,vg_name,attr,size,devices
blkid /dev/VolGroup/*


See output:
sh-4.0# /test
  Logical volume "1" successfully removed
  Logical volume "2" successfully removed
  Logical volume "3" successfully removed
  Logical volume "4" successfully removed
  Rounding up size to full physical extent 1.80 GB
  Logical volume "1" created                      
mkswap: /dev/VolGroup/1: warning: don't erase bootbits sectors
        on whole disk. Use -f to force.                       
Setting up swapspace version 1, size = 1888252 KiB            
no label, UUID=f7b74dcb-61fb-44a1-9c73-cab4d9e286b2           
  Rounding up size to full physical extent 1.80 GB            
  Logical volume "2" created                                  
mkswap: /dev/VolGroup/2: warning: don't erase bootbits sectors
        on whole disk. Use -f to force.
Setting up swapspace version 1, size = 1888252 KiB
no label, UUID=469c4782-2479-48ef-91d9-ab1b2f781c87
  Rounding up size to full physical extent 1.80 GB
  Logical volume "3" created
mkswap: /dev/VolGroup/3: warning: don't erase bootbits sectors
        on whole disk. Use -f to force.
Setting up swapspace version 1, size = 1888252 KiB
no label, UUID=4db2bb06-e7fc-4009-a029-fc8e6048d7f3
  Rounding up size to full physical extent 1.80 GB
  Logical volume "4" created
mkswap: /dev/VolGroup/4: warning: don't erase bootbits sectors
        on whole disk. Use -f to force.
Setting up swapspace version 1, size = 1888252 KiB
no label, UUID=5e721e6b-a270-436e-8288-3408e5ad6e1d

  LV   VG       Attr   LSize  Devices
  1    VolGroup -wi-a- 1.80G  /dev/vda2(0)
  2    VolGroup -wi-a- 1.80G  /dev/vda2(461)
  3    VolGroup -wi-a- 1.80G  /dev/vda2(922)
  4    VolGroup -wi-a- 1.80G  /dev/vda2(1383)

/dev/VolGroup/1: UUID="f7b74dcb-61fb-44a1-9c73-cab4d9e286b2" TYPE="swap"
/dev/VolGroup/2: UUID="469c4782-2479-48ef-91d9-ab1b2f781c87" TYPE="swap"
/dev/VolGroup/3: UUID="4db2bb06-e7fc-4009-a029-fc8e6048d7f3" TYPE="swap"

The device /dev/VolGroup/4 apparently have some problem, no signature. I guess there is some problem with sparse file handling or page cache in the virt system.

Note that lvm works correcly - mapping is ok:
sh-4.0# dmsetup table
VolGroup-4: 0 3776512 linear 252:2 11329920
VolGroup-3: 0 3776512 linear 252:2 7553408
VolGroup-2: 0 3776512 linear 252:2 3776896
VolGroup-1: 0 3776512 linear 252:2 384

So in short, for some LVs the underlying device ignores writes:

sh-4.0# mkswap /dev/VolGroup/4
mkswap: /dev/VolGroup/4: warning: don't erase bootbits sectors
        on whole disk. Use -f to force.
Setting up swapspace version 1, size = 1888252 KiB
no label, UUID=a83947dc-ca4f-49c9-9758-635583ff76e1
sh-4.0# blkid /dev/VolGroup/4
Comment 16 Milan Broz 2009-10-29 06:27:55 EDT
(and I have no idea which component this should be assigned to... virt-something?)
Comment 17 James Laska 2009-10-29 08:03:07 EDT
At the suggestion of danpb, I retested with a guest using 800Mb of RAM and using IDE disks (instead of virtio).  The original problem still presented in each case.
Comment 18 Alasdair Kergon 2009-10-29 08:25:14 EDT
Well for starters, can anyone try on real hardware without any virt involved?
Comment 19 Milan Broz 2009-10-30 05:15:41 EDT
well, reproduced it on completely different machine.

I have to upgrade host kernel, because the kernel-PAE-2.6.31.5-96 does not start VM at all here, so for host using kernel-PAE-2.6.32-0.33.rc5.git1.fc13.i686 for rawhide. Guest 2.6.31.5-96.fc12.i686.

Anyway - first problem I noticed is direct io is no working properly.


# wipe disk
sh-4.0# dmsetup table
x: 0 100000 linear 252:2 14200000

sh-4.0# ls -l /dev/vda2
brw-rw----. 1 root disk 252, 2 2009-10-30 10:39 /dev/vda2

# wipe disk
sh-4.0# dd if=/dev/zero of=/dev/mapper/x bs=512 count=4096
4096+0 records in                                         
4096+0 records out
2097152 bytes (2.1 MB) copied, 0.268842 s, 7.8 MB/s
sh-4.0# blkid /dev/mapper/x

# write good swap signature
sh-4.0# dd if=/tst_swap_ok.img of=/dev/mapper/x
4096+0 records in
4096+0 records out
2097152 bytes (2.1 MB) copied, 0.265991 s, 7.9 MB/s

sh-4.0# blkid /dev/mapper/x
/dev/mapper/x: UUID="0b743993-d2c6-47f4-9585-8e3d2cb8cd89" TYPE="swap"

-> ok

# and now repeat with O_DIRECT
sh-4.0# dd if=/dev/zero of=/dev/mapper/x bs=512 count=4096
4096+0 records in
4096+0 records out
2097152 bytes (2.1 MB) copied, 0.281456 s, 7.5 MB/s
sh-4.0# blkid /dev/mapper/x

sh-4.0# dd oflag=direct if=/tst_swap_ok.img of=/dev/mapper/x
4096+0 records in
4096+0 records out
2097152 bytes (2.1 MB) copied, 16.7611 s, 125 kB/s

(Note the speed!)

sh-4.0# blkid /dev/mapper/x
<nothing>
Comment 20 Alasdair Kergon 2009-10-30 11:44:48 EDT
Please would some virtualisation experts take a look at this one?
Comment 21 Mark McLoughlin 2009-10-30 11:59:37 EDT
Any ideas Christoph? Note, this is on F-12 blocker list
Comment 22 chellwig@redhat.com 2009-10-30 12:26:50 EDT
Looking over the BZ I have no idea what it's even talking about.  What is RATS?  Can we extract a matrix where it fails and where not and why it seems virtualization related?  Did anyone try this magic install with real hardware?
Comment 23 James Laska 2009-10-30 14:49:07 EDT
chellwig: read comment#13

This is basically just an i386 rawhide kvm guest install failing on a i386 rawhide host.
Comment 24 chellwig@redhat.com 2009-10-30 14:54:18 EDT
Thanks.  Can anyone of you give me access to an i386 machine running rawhide to debug this?
Comment 25 James Laska 2009-10-30 14:57:41 EDT
(In reply to comment #24)
> Thanks.  Can anyone of you give me access to an i386 machine running rawhide to
> debug this?  

Sure, I'm looking for your on freenode or oftc to provide the access details.  I'm nick:jlaska.
Comment 26 James Laska 2009-10-30 15:48:03 EDT
Discussion with jforbes on #virt on whether i386 guest installs on an i386 host are a supported platform.  Jforbes indicated that RHEL does not support i386 VT hosts, but Fedora may be interested (but it shouldn't block release).  Moving to F12Target.

[Oct 30 15:38:16] <      jlaska> | jforbes: around?
[Oct 30 15:38:26] <    jforbes> | jlaska: I am
[Oct 30 15:38:45] <      jlaska> | jforbes: hey there ... bburns suggested you might be able to answer a quick question
[Oct 30 15:38:55] <      jlaska> | during the F12Blocker review meeting, we discussed ...
[Oct 30 15:38:56] <      jlaska> | 526549 -  rawhide/i386 kvm guest install on rawhide/i386 host fails due to swap device prompt
[Oct 30 15:39:03] <    jforbes> | jlaska: yup, I am in that channel too
[Oct 30 15:39:18] <      jlaska> | oh cool
[Oct 30 15:39:26] <      jlaska> | I would have ping'd you there, sorry
[Oct 30 15:39:40] <      jlaska> | so ... what's the take from the virt realm?
[Oct 30 15:39:55]            <-- | daMaestro has quit (Ping timeout: 480 seconds)
[Oct 30 15:39:55] <      jlaska> | do we care if Fedora12 i386 guest installs don't work on and i386 host?
[Oct 30 15:40:27]            --> | daMaestro [~jon@content.beatport.com] has joined #virt
[Oct 30 15:40:39] <      jlaska> | s/and/an/
[Oct 30 15:40:40] <    jforbes> | jlaska: Umm, not sure on that, RHEL certainly doesnt support i386 KVM hosts.  Fedora theoretically should, but I wouldn't call it a 
                                blocker
[Oct 30 15:41:10] <      jlaska> | alrighty, well ... I feel stupid for adding this to our automated test environment ;)
[Oct 30 15:41:26] <      jlaska> | I'll update the bz and move it to the 'nice to have' list
[Oct 30 15:41:48] <    jforbes> | jlaska: I tend to agree with cebbert that all VT capable hardware is also 64bit capable, and there is no reason to run a 32bit host
[Oct 30 15:42:06] <      jlaska> | jforbes: agreed on that point
[Oct 30 15:42:19] <      jlaska> | jforbes: we're running it in this environment to similate i386 arch testing
[Oct 30 15:42:26]            <-- | PrestonConnors [~pconnors@andc-office-fw.atlantic.net] has left #virt ( )
[Oct 30 15:42:43] <    jforbes> | jlaska: And that can't be done from a 64bit host?
[Oct 30 15:43:20]            --> | rwmjones_lptp [~rjones@87.127.66.208] has joined #virt
[Oct 30 15:43:26] <      jlaska> | jforbes: it can be ... but there were other tests that made us choose the native arch (yum repo stuff) 
[Oct 30 15:43:30]            <-- | mndo has quit (Ping timeout: 480 seconds)
[Oct 30 15:43:39] <      jlaska> | jforbes: but, we'll revisit that now as it seems our quick fix isn't sufficient
Comment 27 Milan Broz 2009-10-30 16:21:50 EDT
If we do not support i386 kvm, please _remove_ it from F12. It is completely unreliable and if there is no chance to fix it before F12 GA we just risk losing user's data.

I reproduced problem on different machine - I am not able install it on file backed storage is >4GB.

After later resising, some commands return random output (mkswap, blkid - repeated blkid for 4 swap LVs differs in time - without any writes to storage!).

...
Comment 28 Alasdair Kergon 2009-10-30 17:05:31 EDT
Proposing for consideration as a blocker:  Either to fix the functionality or to mention in the release notes and remove it.
Comment 29 Chris Lalancette 2009-11-02 03:25:03 EST
For what it's worth, there *are* chips out there that support VTx and not 64-bit (my laptop happens to be one of them).  While we may not support them in RHEL for good reason, we should support them in Fedora if possible.  Previous to F-12, 32-bit KVM worked just fine on this laptop, it would be a shame to remove it.

Chris Lalancette
Comment 30 Justin M. Forbes 2009-11-02 10:36:17 EST
Again, Fedora should support kvm on 32bit hosts.  But is this really something we would hold up the release for?
Comment 31 Chris Lalancette 2009-11-02 11:07:45 EST
(In reply to comment #30)
> Again, Fedora should support kvm on 32bit hosts.  But is this really something
> we would hold up the release for?  

Oh, I should have been more clear.  I was mostly responding to Comments #27 and #28 that were advising to remove KVM from 32-bit machines.  While I think we absolutely should leave KVM enabled and available for 32-bit processors, I also don't think it's important enough to hold up the release.

Chris Lalancette
Comment 32 Justin M. Forbes 2009-11-02 11:15:35 EST
Okay, removing from blockers, but leaving this open because it does need to be fixed (rather quickly even).
Comment 33 Alasdair Kergon 2009-11-02 12:11:09 EST
I fail to see how something that corrupts people's data is not important enough to be dealt with before the release!  If it's not fixed in time, it should be mentioned in the release notes and the functionality disabled so you cannot use kvm on such systems until the cause of the problem is identified and dealt with.

Someone should at least test upgrading an existing system from F11 to F12 - does that destroy people's data too?
Comment 34 Alasdair Kergon 2009-11-02 12:36:43 EST
(In reply to comment #31)

Chris, were your comments interpreted correctly?  You support Fedora shipping software that fails tests in a way that corrupts data without even a mention in release notes?

Justin, can you explain how this does not meet Fedora's definition of a 'blocker'?

I quote from http://fedoraproject.org/wiki/QA/ReleaseCriteria:
"All possible data corruptor bugs MUST be fixed."
Comment 35 Justin M. Forbes 2009-11-02 12:49:46 EST
From what I have seen (and maybe I missed something), there has been no actual corruption of real data.  Installations are failing, but I haven't seen any reports of previously installed VMs with issues.  Talking the the few people I have encountered who are running 32bit on 32bit VMs under rawhide, they never even noticed an issue because they had not tried to install new ones recently.  If there is evidence of existing VMs hitting data corruption, I would agree this should be a blocker, but if not this is self protecting in that you cannot install a new VM.
Comment 36 Chris Lalancette 2009-11-02 13:20:37 EST
(In reply to comment #34)
> (In reply to comment #31)
> 
> Chris, were your comments interpreted correctly?  You support Fedora shipping
> software that fails tests in a way that corrupts data without even a mention in
> release notes?

No, not in the slightest.  I just don't wish to hastily rip something out of the distribution because it is problematic right now.

> 
> Justin, can you explain how this does not meet Fedora's definition of a
> 'blocker'?

I'll leave this determination up to people more knowledgeable than me, since I don't know any of the details of the bug.

Chris Lalancette
Comment 37 Alasdair Kergon 2009-11-02 13:52:21 EST
Comment #19 is data corruption demonstrated within the guest. The counter-argument is that the corruption is so serious that systems fail well before reaching any data the user cares about?  That is surely still guest
machine "data corruptor" and therefore a blocker.

Even if this is not fixed in time, it needs to be a blocker so that it gets mentioned in the release notes (and probably disabled).

> Talking to the few people I
> have encountered who are running 32bit on 32bit VMs under rawhide, they never
> even noticed an issue because they had not tried to install new ones recently. 

Assuming they are running the final proposed f12 set of packages on both host and guest using the same type of virtual machine (backing store, similar size etc.) could they perhaps try simple dd tests for block device data corruption and try a new install (or upgrade) along the lines in this bug and see whether or not it works for them to give us some more data points about the scope of the problem?
Comment 38 Justin M. Forbes 2009-11-02 14:18:50 EST
I am doing some additional testing now to see what the current state is.
Comment 39 James Laska 2009-11-02 16:30:24 EST
(In reply to comment #36)
> > Justin, can you explain how this does not meet Fedora's definition of a
> > 'blocker'?
> 
> I'll leave this determination up to people more knowledgeable than me, since I
> don't know any of the details of the bug.

A blocker bug in Fedora terms is an issue that has been reviewed by devel, release engineering and QA.  And the majority of those groups feel that the issue is of high (or greater) severity [1], an unreasonable workaround that impacts a significant user base, or is just embarrassing from a Fedora fit'n'finish perspective.

There isn't always a clear line for determining blocker bug worthiness.  This issue certainly fits that mold.  For me, this issue is a bit surprising, mainly as I wasn't aware of the current support agreement for i386 host virtualization.  As Alasdair points out, there may be a component of data corrupter for this issue. 

If the group determines that this isn't a priority use case, we certainly will look at add this issue to the Common_F12_Bugs page guiding users who may be affected (https://fedoraproject.org/wiki/Common_F12_bugs).

[1] https://fedoraproject.org/wiki/BugZappers/BugStatusWorkFlow#Severity
Comment 40 Milan Broz 2009-11-03 10:15:27 EST
So facts:

1) Installing Fedora rawhide (guest, exactly the same command here failed) on Fedora 11 i386 as HOST
=> everything works, VM boots, swap is ok

2) for testing, I removed problematic lv_swap and replaced it by lv_data with some testinfg file mounted to /data

# sha256sum /data/tst.img
2c97752fb92dbb074472c3e50395e08dfaca15cacb9b09b929d85925d824f840  /data/tst.img

3) Reinstalled HOST ONLY to fedora rawhide

4a) with the ketrnel kernel-PAE-2.6.31.5-96.fc12.i686 VM starts, but never finishes BIOS phase (IOW - it doesn't work at all)

4b) booted with f11 kernel but new rawhide userspace (kernel-PAE-2.6.30.9-90.fc11.i686)

VM machine starts and seems to run...

5) and now, where are my data?

# mount /dev/VolGroup/lv_data /data/
mount: you must specify the filesystem type

# blkid /dev/VolGroup/lv_data
/dev/VolGroup/lv_data: UUID="be32fee1-3a27-4c20-b302-3e809c791e64" TYPE="ext4"

I lost the whole data volume.

# fsck -f /dev/mapper/VolGroup-lv_data
fsck from util-linux-ng 2.16
e2fsck 1.41.9 (22-Aug-2009)
fsck.ext4: Group descriptors look bad... trying backup blocks...
Superblock has an invalid journal (inode 8).

*** ext3 journal has been deleted - filesystem is now ext2 only ***

One or more block group descriptor checksums are invalid.  Fix<y>? yes

Group descriptor 0 checksum is invalid.  FIXED.
Group descriptor 1 checksum is invalid.  FIXED.
Group descriptor 2 checksum is invalid.  FIXED.
Group descriptor 3 checksum is invalid.  FIXED.
Group descriptor 4 checksum is invalid.  FIXED.
Group descriptor 5 checksum is invalid.  FIXED.
Group descriptor 6 checksum is invalid.  FIXED.
Group descriptor 7 checksum is invalid.  FIXED.
Superblock last mount time is in the future.
        (by less than a day, probably due to the hardware clock being incorrectly set)  Fix<y>? yes

fsck.ext4: A block group is missing an inode table while reading bad blocks inode

This is clean data corruption. Upgrading host from i686 Fedora 11 to rawhide can destroy virtual machine...
Comment 41 Justin M. Forbes 2009-11-03 10:34:40 EST
Thank you for that report.  Adding back to blockers, as this is clearly data corruption.
Comment 42 Justin M. Forbes 2009-11-03 18:19:51 EST
Additional data from my testing:
- This does reproduce with qemu-system-x86 (no kvm kernel modules loaded).
- This does not seem to reproduce with a virtual disk size of 1900MB (perhaps a large file wrapping issue)
Comment 43 Justin M. Forbes 2009-11-04 04:21:41 EST
More notes:
- This does not reproduce with upstream qemu-kvm-0.10.6
- This does reproduce with upstream qemu-kvm-0.11.0-rc1
Comment 44 Mark McLoughlin 2009-11-04 11:59:42 EST
Okay, I just bisected this down on a 32 bit Fedora 11 host to this commit:

  http://git.savannah.gnu.org/cgit/qemu.git/commit/?id=ceb42de899
    native preadv/pwritev support (Christoph Hellwig)

Need to confirm that it's the this commit that introduces the regression when running on a Fedora 12 host too
Comment 45 Mark McLoughlin 2009-11-04 12:31:00 EST
This temporary fix works for me:

--- a/posix-aio-compat.c
+++ b/posix-aio-compat.c
@@ -34,7 +34,7 @@ static int idle_threads = 0;
 static TAILQ_HEAD(, qemu_paiocb) request_list;
 
 #ifdef HAVE_PREADV
-static int preadv_present = 1;
+static int preadv_present = 0;
 #else
 static int preadv_present = 0;

Again, this on a Fedora 11 host, so need to confirm it on a F-12 host
Comment 46 Mark McLoughlin 2009-11-04 12:42:19 EST
Just confirms that the workaround works on F-12 too

Just building this:

* Wed Nov  4 2009 Mark McLoughlin <markmc@redhat.com> - 2:0.11.0-11
- Temporarily disable preadv/pwritev support to fix data corruption (#526549)

http://koji.fedoraproject.org/koji/buildinfo?buildID=139754

We should file a new bug about fixing and re-enabling preadv/pwritev after we close this one
Comment 47 Mark McLoughlin 2009-11-04 12:44:50 EST
For reference, we had a pwritev related blocker before (bug #497429), but that's unrelated
Comment 48 Justin M. Forbes 2009-11-04 14:33:07 EST
Tagged for F12-final
https://fedorahosted.org/rel-eng/ticket/3090
Comment 49 Mark McLoughlin 2009-11-04 15:59:42 EST
danpb has filed bug #533063 describing the underlying glibc issue we need to resolve before re-enabling preadv/pwritev