Bug 966289 - anaconda crashed trying to create LVM over RAID1 mirror
anaconda crashed trying to create LVM over RAID1 mirror
Status: CLOSED NEXTRELEASE
Product: Fedora
Classification: Fedora
Component: lvm2 (Show other bugs)
19
All Linux
medium Severity medium
: ---
: ---
Assigned To: LVM and device-mapper development team
Fedora Extras Quality Assurance
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2013-05-22 19:50 EDT by Jeff Bastian
Modified: 2013-12-05 09:47 EST (History)
21 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2013-12-05 09:46:58 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
anaconda traceback file (904.28 KB, text/plain)
2013-05-22 19:50 EDT, Jeff Bastian
no flags Details
anaconda.log (24.82 KB, text/x-log)
2013-05-22 19:51 EDT, Jeff Bastian
no flags Details
program.log (29.95 KB, text/x-log)
2013-05-22 19:51 EDT, Jeff Bastian
no flags Details
storage.log (149.79 KB, text/x-log)
2013-05-22 19:52 EDT, Jeff Bastian
no flags Details
syslog (77.88 KB, text/plain)
2013-05-22 19:52 EDT, Jeff Bastian
no flags Details
ifcfg.log (567 bytes, text/x-log)
2013-05-22 19:53 EDT, Jeff Bastian
no flags Details
packaging.log (599.92 KB, text/x-log)
2013-05-22 19:53 EDT, Jeff Bastian
no flags Details
X.log (18.66 KB, text/x-log)
2013-05-22 19:53 EDT, Jeff Bastian
no flags Details
strace of lvm (155.21 KB, text/plain)
2013-06-13 13:06 EDT, Jeff Bastian
no flags Details
lvm -vvvv output (40.11 KB, text/plain)
2013-06-13 13:09 EDT, Jeff Bastian
no flags Details
anaconda traceback with blivet -vvvv patch (1.58 MB, text/plain)
2013-06-13 14:54 EDT, Jeff Bastian
no flags Details
anaconda-traceback from ppc64 install (341.29 KB, text/plain)
2013-08-27 13:53 EDT, Richard W.M. Jones
no flags Details

  None (edit)
Description Jeff Bastian 2013-05-22 19:50:22 EDT
Created attachment 751951 [details]
anaconda traceback file

Description of problem:
Anaconda crashed when trying to install to an LVM over RAID1 mirror setup.

anaconda 19.28-1 exception report
Traceback (most recent call first):
  File "/usr/lib/python2.7/site-packages/blivet/devices.py", line 789, in create
    raise DeviceCreateError(str(e), self.name)
  File "/usr/lib/python2.7/site-packages/blivet/deviceaction.py", line 272, in execute
    self.device.create()
  File "/usr/lib/python2.7/site-packages/blivet/devicetree.py", line 237, in processActions
    action.execute()
  File "/usr/lib/python2.7/site-packages/blivet/__init__.py", line 306, in doIt
    self.devicetree.processActions()
  File "/usr/lib/python2.7/site-packages/blivet/__init__.py", line 165, in turnOnFilesystems
    storage.doIt()
  File "/usr/lib64/python2.7/site-packages/pyanaconda/install.py", line 138, in doInstall
    turnOnFilesystems(storage, mountOnly=flags.flags.dirInstall)
  File "/usr/lib64/python2.7/threading.py", line 766, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/usr/lib64/python2.7/site-packages/pyanaconda/threads.py", line 168, in run
    threading.Thread.run(self, *args, **kwargs)
DeviceCreateError: ('lvcreate failed for fedora/root: running lvm lvcreate -L 13792m -n root --config  devices { filter=["r|/loop3$|","r|/loop4$|","r|/loop5$|","r|/loop6$|","r|/loop7$|","r|/vdc$|","r|/vdd$|","r|/vde$|","r|/vdf$|"] }  fedora failed', 'fedora-root')

Version-Release number of selected component (if applicable):
anaconda 19.28-1

How reproducible:
unknown

Steps to Reproduce:
1. create a virtual machine with 6 virtual disks
2. select the first 2 disks for installation and manually review the partitions
3. create the automatic partition layout
4. modify the Volume Group to be RAID1

Actual results:
anaconda crashed

Expected results:
no crashes

Additional info:
anaconda's automatic bug reporter caught this as false dupe of bug 960271
Comment 1 Jeff Bastian 2013-05-22 19:51:10 EDT
Created attachment 751952 [details]
anaconda.log
Comment 2 Jeff Bastian 2013-05-22 19:51:46 EDT
Created attachment 751953 [details]
program.log
Comment 3 Jeff Bastian 2013-05-22 19:52:11 EDT
Created attachment 751954 [details]
storage.log
Comment 4 Jeff Bastian 2013-05-22 19:52:37 EDT
Created attachment 751955 [details]
syslog
Comment 5 Jeff Bastian 2013-05-22 19:53:04 EDT
Created attachment 751956 [details]
ifcfg.log
Comment 6 Jeff Bastian 2013-05-22 19:53:25 EDT
Created attachment 751957 [details]
packaging.log
Comment 7 Jeff Bastian 2013-05-22 19:53:50 EDT
Created attachment 751958 [details]
X.log
Comment 8 Jeff Bastian 2013-05-22 19:54:41 EDT
My 6 virtual disks are LVM volumes on the host system, so they may contain stale data from previous installations for the F19 Test Day.

https://fedoraproject.org/wiki/Test_Day:2013-05-21_AnacondaNewUI_Followup
Comment 9 David Lehman 2013-05-23 12:12:51 EDT
18:42:42,504 INFO program: Running... mdadm --create /dev/md/pv00 --run --level=1 --raid-devices=2 --metadata=default --bitmap=internal /dev/vda2 /dev/vdb1
18:42:42,755 INFO program: mdadm: array /dev/md/pv00 started.
18:42:42,756 DEBUG program: Return code: 0

18:42:42,862 INFO program: Running... lvm pvcreate --config  devices { filter=["r|/loop3$|","r|/loop4$|","r|/loop5$|","r|/loop6$|","r|/loop7$|","r|/vdc$|","r|/vdd$|","r|/vde$|","r|/vdf$|"] }  --dataalignment 1024k /dev/md/pv00
18:42:42,883 INFO program:   Physical volume "/dev/md/pv00" successfully created
18:42:42,884 DEBUG program: Return code: 0

18:42:42,949 INFO program: Running... lvm vgcreate -s 4m --config  devices { filter=["r|/loop3$|","r|/loop4$|","r|/loop5$|","r|/loop6$|","r|/loop7$|","r|/vdc$|","r|/vdd$|","r|/vde$|","r|/vdf$|"] }  fedora /dev/md/pv00
18:42:42,978 INFO program:   Volume group "fedora" successfully created
18:42:42,979 DEBUG program: Return code: 0

18:42:43,011 INFO program: Running... lvm lvcreate -L 13792m -n root --config  devices { filter=["r|/loop3$|","r|/loop4$|","r|/loop5$|","r|/loop6$|","r|/loop7$|","r|/vdc$|","r|/vdd$|","r|/vde$|","r|/vdf$|"] }  fedora
18:42:43,048 INFO program:   device-mapper: create ioctl on fedora-root failed: Device or resource busy
18:42:43,048 INFO program:   Failed to activate new LV.
18:42:43,048 DEBUG program: Return code: 5


Reassigning to lvm to get input on possible causes, although I am wondering if this is related to the md mirror resync. Adding the mdadm maintainer in case he has input.
Comment 10 Zdenek Kabelac 2013-05-23 12:19:06 EDT
(In reply to David Lehman from comment #9)

> 18:42:43,011 INFO program: Running... lvm lvcreate -L 13792m -n root
> --config  devices {
> filter=["r|/loop3$|","r|/loop4$|","r|/loop5$|","r|/loop6$|","r|/loop7$|","r|/
> vdc$|","r|/vdd$|","r|/vde$|","r|/vdf$|"] }  fedora
> 18:42:43,048 INFO program:   device-mapper: create ioctl on fedora-root
> failed: Device or resource busy
> 18:42:43,048 INFO program:   Failed to activate new LV.
> 18:42:43,048 DEBUG program: Return code: 5
> 
> 
> Reassigning to lvm to get input on possible causes, although I am wondering
> if this is related to the md mirror resync. Adding the mdadm maintainer in
> case he has input.


Can we get full trace of  'lvm lvcreate -L 13792m -n root -vvvv .....'

It's probably worth to run 'dmsetup table' before failing command.

This kernel trace looks weird:

23:42:42,697 ERR kernel:[   76.697116] device-mapper: table: 253:1: linear: dm-linear: Device lookup failed
23:42:42,697 WARNING kernel:[   76.697119] device-mapper: ioctl: error adding target to table
23:42:42,698 ERR kernel:[   76.698254] device-mapper: table: 253:2: linear: dm-linear: Device lookup failed
23:42:42,698 WARNING kernel:[   76.698256] device-mapper: ioctl: error adding target to table
Comment 11 Jeff Bastian 2013-06-13 13:06:03 EDT
Created attachment 760839 [details]
strace of lvm

I think I managed to reproduce it again with Fedora 19 TC3 and anaconda-19.30.5-1

[anaconda root@localhost tmp]# dmsetup table
fedora-swap: 
fedora-root: 0 16351232 linear 9:127 2048
fedora-rootfs: 
live-rw: 0 2097152 snapshot 7:1 7:2 P 8
fedora-home: 0 2048000 linear 9:127 16353280

[anaconda root@localhost tmp]# strace -f -tt -o /tmp/lvm.trace lvm lvcreate -L 2080m -n swap --config 'devices { filter=["r|/loop3$|","r|/loop4$|","r|/loop5$|","r|/loop6$|","r|/loop7$|"] }' fedora


See the attached lvm.trace output.
Comment 12 Jeff Bastian 2013-06-13 13:09:32 EDT
Created attachment 760840 [details]
lvm -vvvv output

The lvm command run again with -vvvv flags
Comment 13 Jeff Bastian 2013-06-13 13:12:47 EDT
And the traceback just to be complete:

anaconda 19.30.5-1 exception report
Traceback (most recent call first):
  File "/usr/lib/python2.7/site-packages/blivet/devices.py", line 789, in create
    raise DeviceCreateError(str(e), self.name)
  File "/usr/lib/python2.7/site-packages/blivet/deviceaction.py", line 272, in execute
    self.device.create()
  File "/usr/lib/python2.7/site-packages/blivet/devicetree.py", line 237, in processActions
    action.execute()
  File "/usr/lib/python2.7/site-packages/blivet/__init__.py", line 310, in doIt
    self.devicetree.processActions()
  File "/usr/lib/python2.7/site-packages/blivet/__init__.py", line 169, in turnOnFilesystems
    storage.doIt()
  File "/usr/lib64/python2.7/site-packages/pyanaconda/install.py", line 138, in doInstall
    turnOnFilesystems(storage, mountOnly=flags.flags.dirInstall)
  File "/usr/lib64/python2.7/threading.py", line 764, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/usr/lib64/python2.7/site-packages/pyanaconda/threads.py", line 168, in run
    threading.Thread.run(self, *args, **kwargs)
DeviceCreateError: ('lvcreate failed for fedora/swap: running lvm lvcreate -L 2080m -n swap --config  devices { filter=["r|/loop3$|","r|/loop4$|","r|/loop5$|","r|/loop6$|","r|/loop7$|"] }  fedora failed', 'fedora-swap')

Local variables in innermost frame:
self: non-existent 2080MB lvmlv fedora-swap (24) with non-existent swap
e: lvcreate failed for fedora/swap: running lvm lvcreate -L 2080m -n swap --config  devices { filter=["r|/loop3$|","r|/loop4$|","r|/loop5$|","r|/loop6$|","r|/loop7$|"] }  fedora failed




And the syslog shows similar errors from device-mapper:
16:53:04,311 ERR kernel:[  406.850454] device-mapper: table: 253:1: linear: dm-linear: Device lookup failed
16:53:04,311 WARNING kernel:[  406.850457] device-mapper: ioctl: error adding target to table
16:53:04,312 ERR kernel:[  406.851650] device-mapper: table: 253:2: linear: dm-linear: Device lookup failed
16:53:04,312 WARNING kernel:[  406.851651] device-mapper: ioctl: error adding target to table
Comment 14 Alasdair Kergon 2013-06-13 13:18:37 EDT
#libdm-deptree.c:1798     Creating fedora-swap
#ioctl/libdm-iface.c:1724         dm create fedora-swap LVM-FF1ysA4X2Sx1FYWK3w8BfEmnI4a4K4HUkUg4mckqenX8aRy00JG87yu4UddmM6Db NF   [16384] (*1)
#ioctl/libdm-iface.c:1742   device-mapper: create ioctl on fedora-swap failed: Device or resource busy
Comment 15 Alasdair Kergon 2013-06-13 13:21:29 EDT
Is that the true output from the run that failed, or is it output from a different run afterwards showing a different (consequential) problem and not relevant to the original problem?
Comment 16 Jeff Bastian 2013-06-13 13:42:14 EDT
Sorry, I should have clarified

That's the output from a later run.

When anaconda crashed with the traceback seen in comment 13, I hit CTRL-ALT-F2 to get a shell and then I ran the failed lvm command manually through strace, and then I ran it a 3rd time with the -vvvv flag.

If you want to get the -vvvv output from when anaconda tries to run it, I guess we'll need to patch anaconda with an updates.img
Comment 17 Alasdair Kergon 2013-06-13 13:48:09 EDT
Running some command afterwards isn't going to help us see how it got into that state unless you manage to put the system back into the same state as it was when the failing command ran so you can accurately reproduce it.  For that you'd need at least 'dmsetup remove fedora-swap' and fedora-rootfs and possibly more depending on what other incorrect states 'dmsetup info -c' shows.
Comment 18 Jeff Bastian 2013-06-13 14:26:54 EDT
I think it might be easier to use an Anaconda updates.img to add the -vvvv flag.  (Easier for me, anyway, since I'm not a device-mapper expert.)

The python-blivet library runs the lvcreate command now, so I think this is what I need to patch:
https://git.fedorahosted.org/cgit/blivet.git/tree/blivet/devicelibs/lvm.py#n353
Comment 19 David Lehman 2013-06-13 14:44:43 EDT
blivet has scripts/makeupdates which creates an updates image usable by anaconda
Comment 20 Jeff Bastian 2013-06-13 14:50:32 EDT
Thanks!  Between that script and a quick git branch, I made an updates.img with this patch:

diff --git a/blivet/devicelibs/lvm.py b/blivet/devicelibs/lvm.py
index eaafad1..decdcb1 100644
--- a/blivet/devicelibs/lvm.py
+++ b/blivet/devicelibs/lvm.py
@@ -354,10 +354,12 @@ def lvcreate(vg_name, lv_name, size, pvs=[]):
     args = ["lvcreate"] + \
             ["-L", "%dm" % size] + \
             ["-n", lv_name] + \
+            ["-vvvv"] + \
             _getConfigArgs() + \
             [vg_name] + pvs
 
     try:
+        util.run_program(["dmsetup", "table"])
         lvm(args)
     except LVMError as msg:
         raise LVMError("lvcreate failed for %s/%s: %s" % (vg_name, lv_name, msg
Comment 21 Jeff Bastian 2013-06-13 14:54:29 EDT
Created attachment 760906 [details]
anaconda traceback with blivet -vvvv patch

The 'dmsetup table' and 'lvcreate -vvvv' output starts on line 10577:

13:45:59,332 INFO program: Running... dmsetup table
13:45:59,348 INFO program: fedora-root: 
13:45:59,349 INFO program: live-rw: 0 2097152 snapshot 7:1 7:2 P 8
13:45:59,349 INFO program: fedora-home: 
13:45:59,350 DEBUG program: Return code: 0
13:45:59,350 INFO program: Running... lvm lvcreate -L 12876m -n root -vvvv --config  devices { filter=["r|/loop3$|","r|/loop4$|","r|/loop5$|","r|/loop6$|","r|/loop7$|"] }  fedora
...
Comment 22 Jeff Bastian 2013-06-13 15:22:53 EDT
I've been able to pretty consistently hit this bug with a slightly different set of steps.

Steps to Reproduce:
0. First run only: Create 6 LVM volumes on the host to use as virtual disks:
  for x in a b c d e f ; do
    lvcreate -L16G -n f19tc3_sd${x} /dev/vg_gecko_data
  done

  Change vg_gecko_data to your volume group name!

1. Clear existing LVM and MDRAID metadata on all 6 virtual disks:
  for x in a b c d e f ; do
    dd if=/dev/zero of=/dev/mapper/vg_gecko_data-f19tc3_sd${x} bs=1M count=100
  done

2. Start the virtual machine
  virt-install --name=test-machine101 --ram=1024 --vcpus=2 \
    --graphics=vnc \
    --os-type=linux --os-variant=fedora18 \
    --network=network=default,model=virtio \
    --cdrom=/var/lib/libvirt/images/Fedora-19-TC3-x86_64-netinst.iso \
    --disk=/dev/mapper/vg_gecko_data-f19tc3_sda,bus=virtio \
    --disk=/dev/mapper/vg_gecko_data-f19tc3_sdb,bus=virtio \
    --disk=/dev/mapper/vg_gecko_data-f19tc3_sdc,bus=virtio \
    --disk=/dev/mapper/vg_gecko_data-f19tc3_sdd,bus=virtio \
    --disk=/dev/mapper/vg_gecko_data-f19tc3_sde,bus=virtio \
    --disk=/dev/mapper/vg_gecko_data-f19tc3_sdf,bus=virtio

  NOTE: I'm using VNC instead of Spice due to bug 974198

3. Select Install Fedora 19-TC3 and hit TAB to add
      updates=http://server/updates.img
   to the command line.

   It may also be useful to add
      drm_kms_helper.edid_firmware=edid/1024x768.bin
   See bug 966226

4. Set Date & Time and NTP servers

5. Change Software Selection to Minimal Install

6. In the Storage spoke, select all 6 disks and hit Done
   - choose "I want to review" and hit Continue
   - Click the link to create mount points automatically
   - Change /home to 2GB and hit Update Settings
   - Change swap to 1GB and hit Update Settings
   - Select / and hit Modify button for the Volume Group
     - Change RAID level to RAID1 and hit Save
     - Hit Update Settings again (I tend to forget this step!)
   - Hit Done and Accept Changes to return to hub

7. Begin Installation

After a few minutes anaconda should crash when "Creating mdmember on /dev/..."
Comment 23 Alasdair Kergon 2013-06-13 17:42:40 EDT
Well again, the lvcreate failure is correct because 'fedora-root' is shown in the dmsetup table command to exist in the kernel already *prior* to the lvcreate command that is supposedly creating it!  lvcreate correctly refuses to create a new LV with a name that is already in use in the kernel.

13:45:59,332 INFO program: Running... dmsetup table
13:45:59,348 INFO program: fedora-root: 
13:45:59,349 INFO program: live-rw: 0 2097152 snapshot 7:1 7:2 P 8
13:45:59,349 INFO program: fedora-home: 
13:45:59,350 DEBUG program: Return code: 0
13:45:59,350 INFO program: Running... lvm lvcreate -L 12876m -n root -vvvv 

How did it get there?

You might need to try putting some more 'dmsetup info -c' and 'dmsetup table' commands earlier on to see if you can work out how 'fedora-root' got there *before* it was meant to be created!

(1) Are you certain this is running in a clean kernel environment?
(2) If so, could something outside anaconda possibly be attempting to set up dm devices?  (Any udev rules firing?)
(3) Or could some earlier anaconda code be incidentally using VG fedora LV root in some temporary way and not cleaning up properly afterwards?
Comment 24 Alasdair Kergon 2013-06-13 18:01:20 EDT
I should make it clear that we do NOT believe that these messages:

23:42:42,697 ERR kernel:[   76.697116] device-mapper: table: 253:1: linear: dm-linear: Device lookup failed
23:42:42,697 WARNING kernel:[   76.697119] device-mapper: ioctl: error adding target to table
23:42:42,698 ERR kernel:[   76.698254] device-mapper: table: 253:2: linear: dm-linear: Device lookup failed
23:42:42,698 WARNING kernel:[   76.698256] device-mapper: ioctl: error adding target to table

correspond to the -vvvv trace we are being presented with.

This is because these kernel error messages come from a "load" ioctl system call, but the userspace trace does not show that ioctl being issued!

(Do the time stamps give any clues?)


I also notice this error in the last trace:

18:42:50,025 ERR systemd: /usr/lib/systemd/system-generators/lvm2-activation-generator exited with exit status 127.
Comment 25 Alasdair Kergon 2013-06-13 18:09:09 EDT
Could there be pre-existing metadata on the device?  Some part of systemd/udev tries to activate this but fails; anaconda comes along and doesn't clean up sufficiently from the failed activation attempt before reusing the devices?
Comment 26 Jeff Bastian 2013-06-13 18:24:01 EDT
I'm manually zeroing the first 100MiB of each disk before running virt-install (as seen in comment 2).  I guess I could try wiping the end of the disk too and see if that "fixes" this problem.
Comment 27 Jeff Bastian 2013-06-13 18:31:36 EDT
(In reply to Jeff Bastian from comment #26)
> (as seen in comment 2).

That was supposed to be comment 22
Comment 28 David Lehman 2013-06-13 18:40:23 EDT
(In reply to Jeff Bastian from comment #22)
> 1. Clear existing LVM and MDRAID metadata on all 6 virtual disks:
>   for x in a b c d e f ; do
>     dd if=/dev/zero of=/dev/mapper/vg_gecko_data-f19tc3_sd${x} bs=1M
> count=100
>   done

This is only clearing all metadata from the virtual disks if either a) they are only 100 blocks in size or b) none of them were partitioned inside the virt install. Otherwise, this is leaving all of the metadata in each partition, and also on any devices build on top of the partitions. Anaconda does not randomize partition sectors, so partition layouts are generally reproducible to the sector. We take great pains to prevent stale metadata from jamming up the works, but there may be gaps.
Comment 29 Peter Rajnoha 2013-06-14 02:58:12 EDT
David, this seems to be incarnation of bug #882341 which we haven't covered yet. Would it be possible for you to define an lvm.conf with activation/auto_activation_volume_list=[]. But not passing it as <lvm_command> --config passing as the configuration needs to be read from global config that the pvscan --cache -aay in 69-dm-lvm-metad.rules would honour (I'd backport the patch from bug #882341 comment #3 as it's not in F19 yet).

Otherwise, if using the lvm.conf is problematic (I think you mentioned some cases once), I could add the test for ANACONDA=1 and then I can call "pvscan --cache" instead of "pvscan --cache -aay", so the lvmetad is still updated, but there would be no volume autoactivation.

Thing is that this is not an upstream solution and so this would always need to be added as an extra patch for Fedora (RHEL 7+) which is not quite nice... If you could define the lvm.conf - that would be much better.
Comment 30 David Lehman 2013-06-14 11:19:59 EDT
We are actually using lvmetad in f19 intentionally. We no longer set ANACONDA in the udev environment, so there is no need to backport that patch.

Are you saying we should disable auto-activation of lvm completely because of the possibility that something like this will happen? We just switched to allowing all storage devices to get activated by systemd/udev/whatever during boot so turning back now is not really an option -- especially just for this bug.
Comment 31 Jeff Bastian 2013-06-14 11:35:42 EDT
(In reply to David Lehman from comment #28)
> (In reply to Jeff Bastian from comment #22)
> >     dd if=/dev/zero of=/dev/mapper/vg_gecko_data-f19tc3_sd${x} bs=1M
> > count=100
> 
> This is only clearing all metadata from the virtual disks if either a) they
> are only 100 blocks in size or b) none of them were partitioned inside the
> virt install.


I had a block size of 1 MiB (bs=1M) so it was 100 MiB, not 100 blocks.

In any case, I just realized that's not enough since /boot is 500 MiB and the RAID metadata comes after that.

So, I just modified it to zero out the first 1 GiB of the disk, and the last 100 MiB of the disk:

  # wipe any mdraid and lvm metadata first
  for x in a b c d e f ; do
    # /sys/block/DEVICE/size is 512-byte units
    # 100MiB = 204800 blocks of 512-bytes
    DMDEV=$(lvdisplay /dev/vg_gecko_data/f19tc3_sd${x} | awk '/Block device/{d=$NF; sub(".*:","dm-",d); print d}')
    DMSKIP=$(( $( cat /sys/block/${DMDEV}/size ) - 204800 ))
    date
    # wipe 1 GiB from front of disk
    dd if=/dev/zero of=/dev/${DMDEV} bs=512 count=2097152
    # wipe 100 MiB from end of disk
    dd if=/dev/zero of=/dev/${DMDEV} bs=512 count=204800 skip=${DMSKIP}
    echo "$x done"
  done
  virt-install ....


I tried this 3 times in a row and Anaconda succeeded every time.  This seems to indicate something in the system is seeing the stale metadata on the disk and activating old devices.
Comment 32 Jeff Bastian 2013-06-14 11:41:11 EDT
(In reply to Jeff Bastian from comment #31)
>     # wipe 100 MiB from end of disk
>     dd if=/dev/zero of=/dev/${DMDEV} bs=512 count=204800 skip=${DMSKIP}


Oops, that should have been "seek", not "skip".  I guess the end of the disk didn't matter then.
Comment 33 Peter Rajnoha 2013-06-17 03:52:18 EDT
(In reply to David Lehman from comment #30)
> We are actually using lvmetad in f19 intentionally. We no longer set
> ANACONDA in the udev environment, so there is no need to backport that patch.
> 

Aha, ok then. Sorry, I didn't know - I'll close the bug #882341 then.

> Are you saying we should disable auto-activation of lvm completely because
> of the possibility that something like this will happen? We just switched to
> allowing all storage devices to get activated by systemd/udev/whatever
> during boot so turning back now is not really an option -- especially just
> for this bug.

I don't think We need to disable the lvmetad and accompanying auto-activation. However, in that case, the LVM error message is correct then (comment #23 - #25). To avoid this problem, we either need to provide the activation/auto_activation_volume_list lvm configuration and activate only volumes needed or there needs to be a complete wipe of stale metadata on existing disks...
Comment 34 Peter Rajnoha 2013-06-17 03:57:57 EDT
...or adding a better check for existing volumes? Why does anaconda try to create a volume that already exists?
Comment 35 Peter Rajnoha 2013-08-22 06:36:25 EDT
(In reply to Zdenek Kabelac from comment #10)
> 23:42:42,697 ERR kernel:[   76.697116] device-mapper: table: 253:1: linear:
> dm-linear: Device lookup failed
> 23:42:42,697 WARNING kernel:[   76.697119] device-mapper: ioctl: error
> adding target to table
> 23:42:42,698 ERR kernel:[   76.698254] device-mapper: table: 253:2: linear:
> dm-linear: Device lookup failed
> 23:42:42,698 WARNING kernel:[   76.698256] device-mapper: ioctl: error
> adding target to table

Bug #985638 comment #13 might be related or even the same...
Comment 36 David Lehman 2013-08-22 12:19:31 EDT
Anaconda never sees any indication that a fedora-root lv exists, so it uses that name when creating a new lv. There are no active lvs when anaconda scans storage. There are no block devices reported as lvm pvs by udev/blkid. I'm not sure what else we think anaconda should be doing in this case. It seems to me like if lvm fails to activate a vg then it should clean up by removing the maps. No?

Peter, it's not possible to know a list of the lvs that "need" to be activated since we're installing an OS and therefore any existing lv could be needed or not, depending on the user's plan for the system configuration.
Comment 37 Richard W.M. Jones 2013-08-27 13:53:55 EDT
Created attachment 791134 [details]
anaconda-traceback from ppc64 install

I just hit this error when installing a ppc64 Fedora 19 guest
under qemu-system-ppc64.
Comment 38 Peter Rajnoha 2013-12-05 09:46:58 EST
We've improved the lvcreate code here to wipe any stale signatures before finishing the lvcreate operation and also properly directing udev to not interfere (by temporarily flagging the LV for udev to skip the scanning/autoactivation). Also, we're using libblkid now by default to detect the signatures so that should provide much more coverage now (see blkid -k) than the native lvm2 signature detection code (which was used for pvcreate originally, but not for lvcreate).

However, this functionality will be available in F21+ (maybe a separate update for F20, but that's going to be after the release).

I'm closing this bug with NEXTRELEASE (meaning F21+ here).
Comment 39 Peter Rajnoha 2013-12-05 09:47:59 EST
(See also bug #997223)

Note You need to log in before you can comment on or make changes to this bug.