Bug 902512 - No boot : Dependency failed for /home (and blkid fails to tell UUID)
Summary: No boot : Dependency failed for /home (and blkid fails to tell UUID)
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: util-linux
Version: 18
Hardware: Unspecified
OS: Unspecified
unspecified
high
Target Milestone: ---
Assignee: Karel Zak
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
: 908310 909564 913198 (view as bug list)
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2013-01-21 20:17 UTC by Couret Charles-Antoine
Modified: 2013-04-20 00:05 UTC (History)
23 users (show)

Fixed In Version:
Clone Of:
Environment:
Last Closed: 2013-04-20 00:05:06 UTC
Type: Bug
Embargoed:


Attachments (Terms of Use)
Yum previous update before the bug. (116.41 KB, text/plain)
2013-01-21 20:17 UTC, Couret Charles-Antoine
no flags Details
Journal "journalctl -xb" during the boot of bug. (206.90 KB, text/plain)
2013-01-21 20:18 UTC, Couret Charles-Antoine
no flags Details
The /etc/fstab copy. (595 bytes, text/plain)
2013-01-21 20:19 UTC, Couret Charles-Antoine
no flags Details
Other results (2.60 KB, text/plain)
2013-01-22 11:35 UTC, Couret Charles-Antoine
no flags Details
Debug Outpout (25.99 KB, text/plain)
2013-01-27 13:53 UTC, Couret Charles-Antoine
no flags Details
udevadm outpout (2.04 KB, text/plain)
2013-01-27 13:53 UTC, Couret Charles-Antoine
no flags Details
blkid output from 2.22.1 and 2.22.2 (10.00 KB, application/x-xz)
2013-02-18 23:54 UTC, Michael Cronenworth
no flags Details
Similar issues raid 1. (25.17 KB, text/plain)
2013-02-19 00:47 UTC, Shawn
no flags Details

Description Couret Charles-Antoine 2013-01-21 20:17:02 UTC
Created attachment 684616 [details]
Yum previous update before the bug.

Description of problem:
After an update, the boot failed at this step :
"Dependency failed for /home"

And I invited to use a console to "mount /home" and launch "systemctl default" for continued the boot correctly. Else, the boot is not completed, and the reboot is not an solution.

If I reboot again, I must mount manually /home again… It's not a fix !

I pushed /etc/fstab, journalctl -xb log and the previous yum update.

Comment 1 Couret Charles-Antoine 2013-01-21 20:18:40 UTC
Created attachment 684617 [details]
Journal "journalctl -xb" during the boot of bug.

Comment 2 Couret Charles-Antoine 2013-01-21 20:19:09 UTC
Created attachment 684618 [details]
The /etc/fstab copy.

Comment 3 Michal Schmidt 2013-01-22 09:17:55 UTC
Are all the filesystems on simple partitions or is there LVM or RAID involved?
Please attach the output of:
lsblk -o +uuid

Comment 4 Couret Charles-Antoine 2013-01-22 09:21:02 UTC
All my partitions are in simple partitions formats, no LVM or RAID...

And the results of command :

[root@DiNozzo ~]#  lsblk -o +uuid
NAME   MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT UUID
sda      8:0    0 698,7G  0 disk            
├─sda1   8:1    0   100M  0 part            94105380105367EE
├─sda2   8:2    0  98,1G  0 part            B0F67D52F67D19B4
├─sda3   8:3    0   500M  0 part /boot      30657114-4e50-46d2-b956-acc9d7c49d57
├─sda4   8:4    0     1K  0 part            
├─sda5   8:5    0 522,5G  0 part /home      
├─sda6   8:6    0  48,8G  0 part /          45e8e554-83e4-487e-bced-fc11af21079c
├─sda7   8:7    0  24,4G  0 part            D4BA5F96BA5F73C8
└─sda8   8:8    0   4,2G  0 part [SWAP]     c201679f-57e8-49ab-9b12-03e1fbac3f66
sr0     11:0    1  1024M  0 rom    

The others partitions are for Windows in dual-boot.

Comment 5 Michal Schmidt 2013-01-22 10:57:57 UTC
(In reply to comment #4)
> ├─sda5   8:5    0 522,5G  0 part /home      

It's significant that there is no UUID seen for the /home partition. We need to find out why.

Please attach the output of:
 blkid -o udev /dev/sda5
 tune2fs -l /dev/sda5

Comment 6 Couret Charles-Antoine 2013-01-22 11:35:58 UTC
Created attachment 685097 [details]
Other results

Comment 7 Couret Charles-Antoine 2013-01-22 11:38:48 UTC
It's strange, the /dev/disk/by-uuid show the link between the SDA5 and this UUID...

I have tested yesterday with a fresh install of Fedora 18, if I update my system with update-testing (the list of packages are in the first attachment), I have this bug again.

Comment 8 Michal Schmidt 2013-01-22 14:24:33 UTC
The util-linux update could be relevant. See if downgrading that makes a difference.

Comment 9 Couret Charles-Antoine 2013-01-22 17:09:42 UTC
Yes, this downgrade fix-it.

And the page : https://admin.fedoraproject.org/updates/FEDORA-2013-0609/util-linux-2.22.2-2.fc18?_csrf_token=6b3ff358a15a31a2f8a72bfa907faa6e07d35e3c explains the bug it's solved in appearance…

Thank you for all. ;)

Comment 10 Karel Zak 2013-01-26 23:21:21 UTC
lsblk(8) reads the UUID from udev db, please try:

   udevadm info --name=sda5

anyway, it does not explain the missing blkid(8) output, I'd like to see:

   LIBBLKID_DEBUG=0xffff blkid -p -o udev /dev/sda5


btw, if manually executed "mount /home" works than mount(8) is able to translate the UUID from fstab to the real device name.

Comment 11 Couret Charles-Antoine 2013-01-27 13:53:02 UTC
Created attachment 688435 [details]
Debug Outpout

Comment 12 Couret Charles-Antoine 2013-01-27 13:53:27 UTC
Created attachment 688436 [details]
udevadm outpout

Comment 13 Karel Zak 2013-01-27 18:57:17 UTC
ERROR: superblocks chain: ambivalent result detected (2 filesystems)!

There is ext4 and also btrfs superblock on the device, please use wipefs(1) to clean up the device ("wipefs -t btrfs /dev/sda5" will list the superblocks, but be careful and read wipefs man page, you can lost data!).

Note that the latest libblkid version checks for btrfs backup superblocks too.

Comment 14 Couret Charles-Antoine 2013-01-27 19:10:40 UTC
Ok, I see… Thank you for your attention. ;)

I supposed the problem is closed because it's not really a bug and the only solution is your command or format the partition ?

Comment 15 Kay Sievers 2013-01-27 19:19:11 UTC
(In reply to comment #14)
> I supposed the problem is closed because it's not really a bug and the only
> solution is your command or format the partition ?

Right, filesystem type guessing is a bit of a dangerous business, and the
default behaviour needs to be as careful as possible because we could risk
the user's data by making the wrong guess.

A problem like this should only be manually resolved like stated above, with
the user giving the needed input for the decision to make.

Comment 16 Karel Zak 2013-01-29 15:34:20 UTC
Yes, we have to be paranoid. The solution is to improve mkfs to use blkid_do_wipe() from libblkid. The same problem has been already reported for xfs.

Note that with btrfs the problem is really tricky because it seems that the filesystem utils are able to silently use the backup superblocks and ignore missing superblock at the begin of the device, see:

https://bugzilla.redhat.com/show_bug.cgi?id=889888
http://www.mail-archive.com/linux-btrfs@vger.kernel.org/msg20947.html

Comment 17 Karel Zak 2013-01-29 15:36:24 UTC
I mean something like:

    #include <blkid/blkid.h>

    fd = open(devname, O_RDWR|O_CLOEXEC);
    blkid_probe_set_device(pr, fd, 0, 0);
 
    blkid_probe_enable_superblocks(pr, 1);
    blkid_probe_set_superblocks_flags(pr, BLKID_SUBLKS_MAGIC);
 
    while (blkid_do_probe(pr) == 0)
        blkid_do_wipe(pr, FALSE);

Comment 18 Eric Sandeen 2013-01-29 16:05:21 UTC
bleah, mke2fs expects to do all IO through its own mechanisms, for various reasons, including the little-known-or-used undo functionality.  I'm afraid that calling out to blkid to _write_ to the filesystem will break that.

I'm afraid we may need to reimplement this wiping behavior for mke2fs.  :(

Comment 19 Karel Zak 2013-01-29 18:44:09 UTC
(In reply to comment #18)
> I'm afraid we may need to reimplement this wiping behavior for mke2fs.  :(

Hmm.. we can extend the library to use any user defined callback instead of write().

Comment 20 Eric Sandeen 2013-02-01 19:45:27 UTC
Karel, that might be nice.  Meanwhile I sent a bog-stupid patch to e2fsprogs but I'd happily rewrite it to use the callback.

-Eric

Comment 21 Karel Zak 2013-02-04 10:57:24 UTC
We (I & Lukas) decided that more usable will be to export from libblkid a function that resets the current probing rather than play any games with callbacks.

https://github.com/karelzak/util-linux/commit/cd0fe5c1659d4144610ffd76500002e98b873e0b

See the commit, docs for the new function blkid_probe_step_back() contains example how to implement your wipe function and how to keep libblkid read-only.

Comment 22 Karel Zak 2013-02-10 09:55:34 UTC
*** Bug 909564 has been marked as a duplicate of this bug. ***

Comment 23 Michael Cronenworth 2013-02-17 03:02:20 UTC
I have a RAID0 as my root and I had to downgrade from util-linux 2.22.2-3.fc18 to 2.22.1-2.1.fc18 to have a system that could boot.

Comment 24 Karel Zak 2013-02-18 11:29:23 UTC
(In reply to comment #23)
> I have a RAID0 as my root and I had to downgrade from util-linux
> 2.22.2-3.fc18 to 2.22.1-2.1.fc18 to have a system that could boot.

Well, that's useless information, please try:

 LIBBLKID_DEBUG=0xffff blkid -p -o udev <dev>

from 2.22.2. Thanks.

Comment 25 Michael Cronenworth 2013-02-18 23:54:53 UTC
Created attachment 699227 [details]
blkid output from 2.22.1 and 2.22.2

Here's the output from both versions. It's a MD RAID0 on two SSDs on an Intel P45 chipset (ICH10) ATA port.

Comment 26 Shawn 2013-02-19 00:47:37 UTC
Created attachment 699250 [details]
Similar issues raid 1.

I have my root on a raid 1. I have a 3.7 kernel that was installed when I still had the down graded versions of util-linux installed and it worked fine even when the 3.8 kernels which were installed with the newer versions all failed to boot properly. Note that 3.7 booted fine even with the newer versions so at least part of the problem would appear to be some thing in the initramfs. 

After downgrading util-linux I was able to boot 3.8.0-0.rc7.git3.1.fc19.x86_64 without generating a new initramfs. When I upgraded util-linux again it stopped working. So I down graded util-linux once more uninstalled kernel-3.8.0-0.rc7.git3.1.fc19.x86_64 and then reinstalled it. When I reboot it works fine. I then upgraded util-linux again and rebooted once more and it still works. So the issue would appear to be some thing at the time that the initramfs is generated rather than some thing detected at run time.

Where it chokes at boot is when systemd-remount-fs tries to run. It is saying that mount can't find UUID=d2175c74-9bb8-4d3b-937f-eb72957998bf then lists the UUID for my / partition. It also says /bin/mount for / exited with status 1. 

Doing a lsblk -f after it fails does show the UUID for / as does df. I did a 
  LIBBLKID_DEBUG=0xffff blkid -p -o udev /dev/mapper/pdc_ebgcdijhgbp5  on one of the times when it failed with util-linux-2.22.2-4.fc19 and have attached it.
  

This is the output from lsblk -f



NAME                        FSTYPE                        LABEL UUID                                 MOUNTPOINT
sda                         promise_fasttrack_raid_member                                            
└─pdc_ebgcdijhgb (dm-0)                                                                              
  ├─pdc_ebgcdijhgbp1 (dm-1)                                                                          
  ├─pdc_ebgcdijhgbp2 (dm-2) ext4                                2c48799f-82da-49dd-b131-f938493bcfb9 /boot
  ├─pdc_ebgcdijhgbp3 (dm-3) swap                                eb540497-613e-4ac2-bb0f-418393b72c77 [SWAP]
  ├─pdc_ebgcdijhgbp4 (dm-4)                                                                          
  └─pdc_ebgcdijhgbp5 (dm-5)                                                                          /
sdb                         promise_fasttrack_raid_member                                            
└─pdc_ebgcdijhgb (dm-0)                                                                              
  ├─pdc_ebgcdijhgbp1 (dm-1)                                                                          
  ├─pdc_ebgcdijhgbp2 (dm-2) ext4                                2c48799f-82da-49dd-b131-f938493bcfb9 /boot
  ├─pdc_ebgcdijhgbp3 (dm-3) swap                                eb540497-613e-4ac2-bb0f-418393b72c77 [SWAP]
  ├─pdc_ebgcdijhgbp4 (dm-4)                                                                          
  └─pdc_ebgcdijhgbp5 (dm-5)                                                                          /
sr0 


These are what work for me.


           Command Line   : downgrade http://kojipkgs.fedoraproject.org//packages/util-linux/2.22.1/1.fc19/x86_64/util-linux-2.22.1-1.fc19.x86_64.rpm http://kojipkgs.fedoraproject.org//packages/util-linux/2.22.1/1.fc19/x86_64/libuuid-2.22.1-1.fc19.x86_64.rpm http://kojipkgs.fedoraproject.org//packages/util-linux/2.22.1/1.fc19/x86_64/libblkid-2.22.1-1.fc19.x86_64.rpm http://kojipkgs.fedoraproject.org//packages/util-linux/2.22.1/1.fc19/x86_64/libmount-2.22.1-1.fc19.x86_64.rpm http://kojipkgs.fedoraproject.org//packages/util-linux/2.22.1/1.fc19/x86_64/libuuid-2.22.1-1.fc19.x86_64.rpm http://kojipkgs.fedoraproject.org//packages/util-linux/2.22.1/1.fc19/x86_64/util-linux-2.22.1-1.fc19.x86_64.rpm http://kojipkgs.fedoraproject.org//packages/util-linux/2.22.1/1.fc19/x86_64/uuidd-2.22.1-1.fc19.x86_64.rpm http://kojipkgs.fedoraproject.org//packages/coreutils/8.17/5.fc18/x86_64/coreutils-8.17-5.fc18.x86_64.rpm

Transaction performed with:
    Installed     rpm-4.11.0.1-1.fc19.x86_64 @rawhide
    Installed     yum-3.4.3-59.fc19.noarch   @rawhide
Packages Altered:
    Downgrade  coreutils-8.17-5.fc18.x86_64    @/coreutils-8.17-5.fc18.x86_64
    Downgraded           8.21-1.fc19.x86_64    @rawhide
    Downgrade  libblkid-2.22.1-1.fc19.x86_64   @/libblkid-2.22.1-1.fc19.x86_64
    Downgraded          2.22.2-4.fc19.x86_64   @rawhide
    Downgrade  libmount-2.22.1-1.fc19.x86_64   @/libmount-2.22.1-1.fc19.x86_64
    Downgraded          2.22.2-4.fc19.x86_64   @rawhide
    Downgrade  libuuid-2.22.1-1.fc19.x86_64    @/libuuid-2.22.1-1.fc19.x86_64
    Downgraded         2.22.2-4.fc19.x86_64    @rawhide
    Downgrade  util-linux-2.22.1-1.fc19.x86_64 @/util-linux-2.22.1-1.fc19.x86_64
    Downgraded            2.22.2-4.fc19.x86_64 @rawhide
    Downgrade  uuidd-2.22.1-1.fc19.x86_64      @/uuidd-2.22.1-1.fc19.x86_64
    Downgraded       2.22.2-4.fc19.x86_64      @rawhide
 
My kernel cmdline is

BOOT_IMAGE=/vmlinuz-3.8.0-0.rc7.git3.1.fc19.x86_64 root=UUID=d2175c74-9bb8-4d3b-937f-eb72957998bf ro rd.dm.uuid=pdc_ebgcdijhgb rd.md=0 rd.lvm=0 LANG=en_US.UTF-8 vconsole.font=latarcyrheb-sun16 vconsole.keymap=us rd.luks=0 rhgb quiet

                                                 
                      
Let me know if there is any thing else I can do to help.

Comment 27 Shawn 2013-02-19 00:52:32 UTC
 Now that I am able to read that  a little clearer I see I wasn't clear that that "lsblk -f" was done when it did boot correctly. Sorry for any confusion.

Comment 28 Karel Zak 2013-02-19 13:54:55 UTC
(In reply to comment #25)
> Created attachment 699227 [details]
> blkid output from 2.22.1 and 2.22.2
> 
> Here's the output from both versions. It's a MD RAID0 on two SSDs on an
> Intel P45 chipset (ICH10) ATA port.

 See comment #13.

Comment 29 Karel Zak 2013-02-19 13:56:32 UTC
Shawn, you're also affected with the problem with bltrfs backup superblocks (comment #13).

Comment 30 Michael Cronenworth 2013-02-19 14:28:52 UTC
(In reply to comment #28)
>  See comment #13.

I can call "wipefs /dev/md0" with the latest util-linux and see the btrfs block, but "wipefs -t btrfs /dev/md0" shows nothing. It also will not run while the FS is mounted so I have to spend quite a bit of time fixing my system that was working before a new feature broke it. You can point the blame at other people all you want, but without handling cases like this people's systems won't boot and that is irresponsible.

Since this update is expected to require manual intervention I propose it be added to CommonBugs. Then when people have to use a second computer to try and fix their system Google can show how to do it.

Comment 31 Karel Zak 2013-02-19 15:23:35 UTC
I'm going to update the libblkid patch -- the backup btrfs superblocks will be visible for wipefs(8) only, but mount/udev will work as expected.

Comment 32 Fedora Update System 2013-02-19 15:47:12 UTC
util-linux-2.22.2-5.fc18 has been submitted as an update for Fedora 18.
https://admin.fedoraproject.org/updates/util-linux-2.22.2-5.fc18

Comment 33 Shawn 2013-02-19 19:37:33 UTC
(In reply to comment #29)
> Shawn, you're also affected with the problem with bltrfs backup superblocks
> (comment #13).

I went through and cleaned up the btrfs superblock which was likely left over from a failed ext4 -> btrfs conversion. The UUID for / does show up now with lsblk -f.

I also did a yum upgrade which grabbed the latest util-linux from rawhide and the 3.8 release kernel and installed it. I had no issues booting into the new kernel so yes at least on my system util-linux is working correctly.

Thanks much for the help.

Comment 34 Michael Cronenworth 2013-02-20 00:29:46 UTC
(In reply to comment #32)
> util-linux-2.22.2-5.fc18 has been submitted as an update for Fedora 18.
> https://admin.fedoraproject.org/updates/util-linux-2.22.2-5.fc18

Is this update supposed to fix the two superblock issue? It did not resolve the issue for me. The blkid output is the same and the kernel was not able to be installed completely (grubby was unable to probe for my root so a kernel install failed as it did on 2.22.2-3).

Comment 35 Karel Zak 2013-02-20 09:06:47 UTC
(In reply to comment #34)
> (In reply to comment #32)
> > util-linux-2.22.2-5.fc18 has been submitted as an update for Fedora 18.
>
> Is this update supposed to fix the two superblock issue?

Yes, but I made a bug in the patch. Ignore the update. Sorry.

Comment 36 Fedora Update System 2013-02-20 10:27:24 UTC
util-linux-2.22.2-6.fc18 has been submitted as an update for Fedora 18.
https://admin.fedoraproject.org/updates/util-linux-2.22.2-6.fc18

Comment 37 Karel Zak 2013-02-20 10:32:39 UTC
The update 2.22.2-6 ignores the backup btrfs superblocks:

# LIBBLKID_DEBUG=0xffff blkid -o udev -p /dev/sdb1 | grep backup
btrfs: found backup superblock, ignore

and correctly reports extN:

# blkid -o udev -p /dev/sdb1
ID_FS_TYPE=ext4
ID_FS_USAGE=filesystem
...

Comment 38 Michael Cronenworth 2013-02-20 13:38:34 UTC
(In reply to comment #37)
> The update 2.22.2-6 ignores the backup btrfs superblocks:

That did the trick. Thank you.

Comment 39 Peter Jones 2013-02-20 19:10:38 UTC
*** Bug 908310 has been marked as a duplicate of this bug. ***

Comment 40 Couret Charles-Antoine 2013-02-21 23:10:31 UTC
The bug is solved for me with the 2.22.2-6 update.

Comment 41 Fedora Update System 2013-02-24 08:56:20 UTC
util-linux-2.22.2-6.fc18 has been pushed to the Fedora 18 stable repository.  If problems still persist, please make note of it in this bug report.

Comment 42 Karel Zak 2013-02-28 13:23:40 UTC
*** Bug 913198 has been marked as a duplicate of this bug. ***

Comment 43 Carlisle Childress 2013-02-28 14:34:09 UTC
Bug 913198 for me has now been fixed. 

I reversed my work-arounds and it boots without any problems. 

Thanks.

Comment 44 Frank McCormick 2013-03-02 20:05:32 UTC
Doesn't work for me. I too revesed my downgrades and I still get errors in the generated grub.cfg

Comment 45 Adam Williamson 2013-04-20 00:05:06 UTC
three other reporters including the original reporter of this bug say it did work, so it sounds like you may have a special case. this bug is supposed to be closed, let's close it and drop the CommonBugs nomination, and Frank, if you're still having trouble, can you file a new bug for your case? thanks.


Note You need to log in before you can comment on or make changes to this bug.