Bug 959576 - Second Logical Volume fails to mount during upgrade (LVM on mdraid)
Summary: Second Logical Volume fails to mount during upgrade (LVM on mdraid)
Keywords:
Status: CLOSED DUPLICATE of bug 958586
Alias: None
Product: Fedora
Classification: Fedora
Component: fedup
Version: 17
Hardware: x86_64
OS: Linux
unspecified
medium
Target Milestone: ---
Assignee: Will Woods
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On: 903998 958586
Blocks:
TreeView+ depends on / blocked
 
Reported: 2013-05-03 20:34 UTC by Will Woods
Modified: 2013-05-16 21:18 UTC (History)
8 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of: 903998
Environment:
Last Closed: 2013-05-16 21:16:55 UTC
Type: Bug
Embargoed:


Attachments (Terms of Use)
Requested disk info (3.68 KB, text/plain)
2013-05-03 23:03 UTC, Ivor Durham
no flags Details

Description Will Woods 2013-05-03 20:34:32 UTC
--- Comment from Ivor Durham on 2013-05-03 00:38:20 EDT ---

I have just encountered this [...] with fedup-0.7.3-4 as have several other people responding on the Installation forum. Here's the output from "fdisk -l" showing my partition structure:

Disk /dev/sda: 1000.2 GB, 1000204886016 bytes
255 heads, 63 sectors/track, 121601 cylinders, total 1953525168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00082e40

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *        2048     1026047      512000   83  Linux
/dev/sda2         1026048  1953519615   976246784   8e  Linux LVM

Disk /dev/sdb: 1000.2 GB, 1000204886016 bytes
255 heads, 63 sectors/track, 121601 cylinders, total 1953525168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00082e40

   Device Boot      Start         End      Blocks   Id  System
/dev/sdb1   *        2048     1026047      512000   83  Linux
/dev/sdb2         1026048  1953519615   976246784   8e  Linux LVM

Disk /dev/md125: 1000.2 GB, 1000202043392 bytes
2 heads, 4 sectors/track, 244189952 cylinders, total 1953519616 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00082e40

      Device Boot      Start         End      Blocks   Id  System
/dev/md125p1   *        2048     1026047      512000   83  Linux
/dev/md125p2         1026048  1953519615   976246784   8e  Linux LVM

Disk /dev/mapper/vg_clowder-lv_swap: 4227 MB, 4227858432 bytes
255 heads, 63 sectors/track, 514 cylinders, total 8257536 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes


Disk /dev/mapper/vg_clowder-lv_root: 53.7 GB, 53687091200 bytes
255 heads, 63 sectors/track, 6527 cylinders, total 104857600 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes


Disk /dev/mapper/vg_clowder-lv_home: 941.7 GB, 941738688512 bytes
255 heads, 63 sectors/track, 114493 cylinders, total 1839333376 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

When running FC17, /dev/mapper contains:

control
vg_clowder-lv_home -> ../dm-2
vg_clowder-lv_root -> ../dm-1
vg_clowder-lv_swap -> ../dm-0

When the System Upgrade boot drops into Emergency Mode, /dev/mapper only contains:

control
vg_clowder-lv_root -> ../dm-1
vg_clowder-lv_swap -> ../dm-0

and /dev/dm-2 is also missing.

Given that this problem was reported in January and it is now May, I have to ask whether or not this bug has been investigated at all? Is any more information needed? I cannot complete my upgrade until there is at least a work-around. I'd rather avoid having to do a clean install if that can be at all avoided because I can't afford the time at the moment. Alternatively, what's the best way to back out of this failed upgrade attempt?

--- Additional comment from Will Woods on 2013-05-03 12:07:21 EDT ---

This isn't likely to be a fedup problem per se, since fedup itself doesn't do the initial mounting. This likely to be a problem with dracut/systemd not knowing how to handle your weird filesystem layout without special instructions.

Does the upgrade actually start/run/finish?
Is the "System Upgrade (fedup)" item still in the boot configuration?
What kernel(s) are available?
Can you choose an older kernel to boot?
If you edit /etc/fstab and comment out /home temporarily, does the system start?
Is /home encrypted?

--- Additional comment from Ivor Durham on 2013-05-03 15:12:19 EDT ---

The "weird" filesystem was the result of choosing a "Use all" default partition when I installed FC15 with a clean install. The installer ended up partitioning the logical volume into /root and /home anyway; it wasn't by conscious choice on my part.

Other answers in order:

Does the upgrade actually start/run/finish?
  No
Is the "System Upgrade (fedup)" item still in the boot configuration?
  Yes
What kernel(s) are available?
  The boot menu has:
    System Upgrade (fedup)
    Fedora (3.8.8-100.fc17.x86_64)
    Fedora (3.8.4-102.fc17.x86_64)
    Fedora (3.8.3-103.fc17.x86_64)
Can you choose an older kernel to boot?
  Yes. I am running 3.8.8-100
If you edit /etc/fstab and comment out /home temporarily, does the system start?
  At this point I'm somewhat reluctant to experiment and risk ending up with
  the need re-install. I realize that an upgrade is a always a risk, but I
  encountered a problem with RAID on an earlier upgrade to FC15 (See Bug #736386)
  that left me dead in the water for a week.
Is /home encrypted?
  No, no partition is encrypted. Also SELinux is in permissive mode.

Comment 1 Will Woods 2013-05-03 20:46:28 UTC
Are the other failures you mentioned all LVM+mdraid, or are there failures with plain LVM?

Are you trying to upgrade to F18 or F19?

Is your RAID device usually named "/dev/md125"? Do you have an /etc/mdadm.conf? Can you attach/paste that here?

Could you also do:

  cat /proc/mounts /proc/cmdline /etc/fstab > diskinfo.txt 
  sudo blkid >> diskinfo.txt

and attach diskinfo.txt?

Comment 2 Ivor Durham 2013-05-03 23:03:34 UTC
Created attachment 743399 [details]
Requested disk info

Comment 3 Ivor Durham 2013-05-03 23:10:57 UTC
The system I am using has always had RAID 1. It has the Intel Matrix Storage Manager hardware. However, a few core releases ago RAID support was forcibly moved to mdadm.

I am trying to upgrade from FC17 to FC18 (to resolve some Gnome problems).

Until FC16 (I think) the devices were /md126 with /md127 created as a "container". With the FC17 upgrade the device became /md125 instead of /md126 with /md127 as a container. I have made no explicit choices to do with the RAID support outside the automatic upgrade process.

The current mdadm.conf is:

MAILADDR root
AUTO +imsm +1.x -all

I have attached the /proc/mounts, /proc/cmdline, /etc/fstab and blkid as requested (with separator lines for readability). Thanks for investigating.

Comment 4 Eduardo 2013-05-05 05:14:03 UTC
I had device timeouts on a similar configuration when I upgraded my system yesterday. Maybe this can help identify the problem root cause. I was able to workaround the different timeouts I saw:
1. file systems on an imsm raid 1 timed out (those partitions were not part of the main installation partitions so I commented them out on fstab)
2. then partitions with ntfs timed out (also not part of the main installation partitions, so commented out as well)
3. btrfs partitions on LVM for /home /tmp and /opt mounted just fine
4. finally got a timeout on mounting /boot (plain ext4)

I thought that getting a timeout on /boot was very weird because it does not have a strange file system and because it was used for loading the kernel in the first place. The only thing that I could think is different is that I have /boot on a non-primary partition:
==
# fdisk -l /dev/sda
Disk /dev/sda: 256.1 GB, 256060514304 bytes, 500118192 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0xb234ccc0

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1              63       80324       40131   de  Dell Utility
/dev/sda2           81920    19419135     9668608    7  HPFS/NTFS/exFAT
/dev/sda3   *    19419136   284753919   132667392    7  HPFS/NTFS/exFAT
/dev/sda4       284768190   500103449   107667630    5  Extended
/dev/sda5       284768253   285792252      512000   83  Linux
/dev/sda6       285792254   500103164   107155455+  8e  Linux LVM
==

/boot is /dev/sda5

After the timeout happened and I was asked to enter my root password. I was able to mount /boot from that shell without problems.

I worked around the problem by enabling the fedup shell (adding rd.upgrade.debugshell on the fedup kernel line) and commenting out the /boot partition on fstab. The process was able to start and as soon as I could I switched to the shell (ctrl-alt-F2). For some reason that first shell is not able to do much. Exiting allows the process to continue and a new shell starts where I was able to mount /boot while the transaction was been analyzed. After that the upgrade process continued without any issues

I was using fedup-0.7.3-4.fc17.noarch

Comment 5 Sergio Basto 2013-05-08 21:58:19 UTC
(In reply to comment #4)

> I worked around the problem by enabling the fedup shell (adding
> rd.upgrade.debugshell on the fedup kernel line) and commenting out the /boot
> partition on fstab. The process was able to start and as soon as I could I
> switched to the shell (ctrl-alt-F2). For some reason that first shell is not
> able to do much. Exiting allows the process to continue and a new shell
> starts where I was able to mount /boot while the transaction was been
> analyzed. After that the upgrade process continued without any issues
> 
> I was using fedup-0.7.3-4.fc17.noarch

I also have device timeouts on by lvm2 monitor on /dev/sda1 /boot , which is strange, the fact that /dev/sda1 is not an lvm volume, just /dev/sda2 .
I follow yours workaround .
I enter debug-shell and just do logout, do not have mount /boot at all , but system is now upgrading , from F17 to F19

Comment 6 Eduardo 2013-05-08 22:09:38 UTC
I mounted /boot manually. You should make sure that the kernel installed by the fedup lands on the right place and the grub2.cfg is updated.

Comment 7 Sergio Basto 2013-05-08 23:18:03 UTC
(In reply to comment #6)
> I mounted /boot manually. You should make sure that the kernel installed by
> the fedup lands on the right place and the grub2.cfg is updated.

but where do you mount it ? not in / ? 

system upgrade went well, just wrote a new dir /boot , after boot into system with kernel from F17 , I re-add /boot on fstab and moved new boot to old boot. I replace all /boot/grub2 from new one and forgot to do 
grub2-install /dev/sda2 , 
but after enter in rescue mode, I done it and system seems fine now .

Comment 8 Eduardo 2013-05-09 12:33:32 UTC
I am glad that you were able to work it out. If you mount the boot partition manually on /boot before fedup installs the kernel you can avoid those extra steps (in your particular case that would be 'mount /dev/sda1 /boot'). Another way to do it would be to uncomment the boot line on fstab first and then just do 'mount /boot'.

Comment 9 Will Woods 2013-05-16 21:16:55 UTC

*** This bug has been marked as a duplicate of bug 958586 ***


Note You need to log in before you can comment on or make changes to this bug.