Bug 754877 - kernel 2.6.40.6-0.fc15.x86_64 update missing initrd / initramfs
kernel 2.6.40.6-0.fc15.x86_64 update missing initrd / initramfs
Status: CLOSED WONTFIX
Product: Fedora
Classification: Fedora
Component: grubby (Show other bugs)
15
x86_64 Linux
unspecified Severity high
: ---
: ---
Assigned To: Peter Jones
Fedora Extras Quality Assurance
:
: 823631 (view as bug list)
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2011-11-17 20:46 EST by Mike Pittaro
Modified: 2012-08-07 16:02 EDT (History)
10 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2012-08-07 16:01:58 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
dracut.log for review (529.17 KB, text/plain)
2011-11-18 11:15 EST, Mike Pittaro
no flags Details
dracut log for 2.6.40.6-0 run (62.37 KB, text/x-log)
2011-11-29 11:45 EST, Mike Pittaro
no flags Details
debug dracu log (compressed) (299.93 KB, application/x-gzip)
2011-12-07 14:43 EST, Mike Pittaro
no flags Details
updated dracut log (353.88 KB, application/x-gzip)
2011-12-09 13:53 EST, Mike Pittaro
no flags Details

  None (edit)
Description Mike Pittaro 2011-11-17 20:46:08 EST
Description of problem:

After installing the 2.6.40.6-0.fc15.x86_64update, the system will no longer boot.

Version-Release number of selected component (if applicable):


How reproducible:

100%

Steps to Reproduce:
1. boot

  
Actual results:

Kernel panic on boot

Expected results:



Additional info:

The 2.6.40.6-0.fc15.x86_64 update has two problems:

1. initramfs-2.6.40.6-5.fc15.x86_64.img is not installed in /boot
2. /boot/grub/grub.conf never gets an entry for the initrd.

Temporary fix:
a. select an older kernel from the boot menu
or
b. boot from dvd in rescue mode, edit grub.conf to use a previous kernel.
Comment 1 Josh Boyer 2011-11-18 07:11:03 EST
This isn't really a kernel problem in and of itself.

When you did the yum update, were there error messages?  The initramfs is built after the kernel is installed by dracut in a %posttrans script.
Comment 2 Mike Pittaro 2011-11-18 11:14:18 EST
re: kernel, Agreed - just seemed the best fitting component to start with. If 
dracut is the correct component, just update the bug report.

I encountered the problem with a power hit on October 29th. 
the yum update was earlier, on october 26th. here's what my boot directory looks like:

[root@sc440 boot]# ls -lc /boot
total 50235
-rw-r--r--  1 root root   117475 Sep  6 10:27 config-2.6.40.3-0.fc15.x86_64
-rw-r--r--  1 root root   117475 Sep 13 10:32 config-2.6.40.4-5.fc15.x86_64
-rw-r--r--  1 root root   117475 Oct 26 14:43 config-2.6.40.6-0.fc15.x86_64
drwxr-xr-x. 3 root root     1024 Dec 26  2008 efi
-rw-r--r--  1 root root   178436 Jun 20 11:20 elf-memtest86+-4.20
drwxr-xr-x  2 root root     1024 Jun  2 16:32 extlinux
drwxr-xr-x. 2 root root     1024 Nov 17 20:46 grub
-rw-r--r--  1 root root 15394432 Sep  6 10:29 initramfs-2.6.40.3-0.fc15.x86_64.img
-rw-r--r--  1 root root 15528965 Sep 13 10:35 initramfs-2.6.40.4-5.fc15.x86_64.img
-rw-r--r--. 1 root root   574949 Jun  2 13:59 initrd-plymouth.img
drwx------. 2 root root    12288 Oct 30 03:27 lost+found
-rw-r--r--  1 root root   176760 Jun 20 11:20 memtest86+-4.20
-rw-------  1 root root  2356599 Sep  6 10:27 System.map-2.6.40.3-0.fc15.x86_64
-rw-------  1 root root  2358422 Sep 13 10:32 System.map-2.6.40.4-5.fc15.x86_64
-rw-------  1 root root  2358812 Oct 26 14:43 System.map-2.6.40.6-0.fc15.x86_64
-rwxr-xr-x  1 root root  3970320 Sep  6 10:27 vmlinuz-2.6.40.3-0.fc15.x86_64
-rwxr-xr-x  1 root root  3972400 Sep 13 10:32 vmlinuz-2.6.40.4-5.fc15.x86_64
-rwxr-xr-x  1 root root  3974320 Oct 26 14:43 vmlinuz-2.6.40.6-0.fc15.x86_64

I didn't see / notice any update errors. The yum log shows the following, so I don't think any errors were caught.

...
Oct 26 14:42:46 Updated: 1:qt-mysql-4.7.4-2.fc15.x86_64
Oct 26 14:42:47 Updated: perl-libwww-perl-5.837-4.fc15.noarch
Oct 26 14:43:00 Installed: kernel-devel-2.6.40.6-0.fc15.x86_64
Oct 26 14:43:01 Updated: rsyslog-5.8.5-1.fc15.x86_64
Oct 26 14:43:09 Updated: kernel-doc-2.6.40.6-0.fc15.noarch
Oct 26 14:43:11 Updated: kernel-headers-2.6.40.6-0.fc15.x86_64
Oct 26 14:43:22 Installed: kernel-2.6.40.6-0.fc15.x86_64
Oct 27 10:48:39 Erased: perl-libwww-perl
Oct 27 10:48:39 Erased: kernel-devel
Oct 27 10:48:42 Erased: php-common
...

in the main syslog, here's the relevant sections from the install timeframe:

Oct 26 14:42:00 sc440 yum[14511]: Updated: perl-Test-Harness-3.17-162.fc15.noarch
Oct 26 14:42:01 sc440 yum[14511]: Updated: perl-CPAN-1.9402-162.fc15.noarch
Oct 26 14:42:02 sc440 yum[14511]: Updated: httpd-tools-2.2.21-1.fc15.x86_64
Oct 26 14:42:12 sc440 systemd[1]: Reloading.
Oct 26 14:42:13 sc440 systemd[1]: mysqld.service: Supervising process 1534 which is not our child. We'll most likely not notice when it exits.
Oct 26 14:42:13 sc440 yum[14511]: Updated: httpd-2.2.21-1.fc15.x86_64
Oct 26 14:42:14 sc440 yum[14511]: Updated: foomatic-filters-4.0.8-3.fc15.x86_64
Oct 26 14:42:15 sc440 yum[14511]: Updated: foomatic-4.0.8-3.fc15.x86_64
Oct 26 14:42:16 sc440 yum[14511]: Updated: php-5.3.8-3.fc15.x86_64
Oct 26 14:42:18 sc440 dbus-daemon: [system] Reloaded configuration
Oct 26 14:42:31 sc440 yum[14511]: Updated: 6:kdelibs-4.6.5-6.fc15.x86_64
Oct 26 14:42:31 sc440 yum[14511]: Updated: php-mysql-5.3.8-3.fc15.x86_64
Oct 26 14:42:43 sc440 yum[14511]: Updated: kdelibs3-3.5.10-31.fc15.x86_64
Oct 26 14:42:43 sc440 yum[14511]: Updated: rpm-build-4.9.1.2-1.fc15.x86_64
Oct 26 14:42:45 sc440 yum[14511]: Updated: rpm-python-4.9.1.2-1.fc15.x86_64
Oct 26 14:42:47 sc440 yum[14511]: Updated: perl-libwww-perl-5.837-4.fc15.noarch
Oct 26 14:43:00 sc440 yum[14511]: Installed: kernel-devel-2.6.40.6-0.fc15.x86_64
Oct 26 14:43:01 sc440 yum[14511]: Updated: rsyslog-5.8.5-1.fc15.x86_64
Oct 26 14:43:09 sc440 yum[14511]: Updated: kernel-doc-2.6.40.6-0.fc15.noarch
Oct 26 14:43:11 sc440 rsyslogd: imuxsock: recvfrom UNIX: Resource temporarily unavailable
Oct 26 14:43:22 sc440 yum[14511]: Installed: kernel-2.6.40.6-0.fc15.x86_64
Oct 26 14:59:23 sc440 dbus-daemon: [system] Activating service name='org.freedesktop.PackageKit' (using servicehelper)
Oct 26 14:59:24 sc440 dbus-daemon: [system] Successfully activated service 'org.freedesktop.PackageKit'
Oct 26 15:32:48 sc440 dbus-daemon: [system] Activating service name='org.freedesktop.PackageKit' (using servicehelper)
Oct 26 15:32:48 sc440 dbus-daemon: [system] Successfully activated service 'org.freedesktop.PackageKit'
Oct 26 15:35:55 sc440 dnsmasq-dhcp[10977]: DHCPREQUEST(eth0) 10.100.30.9 38:60:77:06:27:85
Oct 26 15:35:55 sc440 dnsmasq-dhcp[10977]: DHCPACK(eth0) 10.100.30.9 38:60:77:06:27:85 blade1
Oct 26 16:09:34 sc440 dnsmasq-dhcp[10977]: DHCPDISCOVER(eth0) 58:55:ca:12:fa:79
Oct 26 16:09:34 sc440 dnsmasq-dhcp[10977]: DHCPOFFER(eth0) 10.100.30.90 58:55:ca:12:fa:79

Interestingly, the dracut log shows no mention of 2.6.40.6.  It's dated September 13th.  attached for reference.

[root@sc440 log]# ls -l dracut.log
-rw-r--r--. 1 root root 541867 Sep 13 10:35 dracut.log
Comment 3 Mike Pittaro 2011-11-18 11:15:18 EST
Created attachment 534416 [details]
dracut.log for review
Comment 4 Mike Pittaro 2011-11-18 11:25:49 EST
FWIW, there's a vague reference to a similar problem on the forums:
http://www.fedoraforum.org/forum/showthread.php?t=271875
Comment 5 Dave Jones 2011-11-22 12:37:05 EST
if you run dracut by hand for the 2.6.40.6 kernel, is there anything interesting output, or in the log afterwards ?
Comment 6 Mike Pittaro 2011-11-29 11:42:19 EST
I ran dracut by hand, it completed successfully with no apparent errors. 
the ramdisk is created, the only thing I notice is it's significantly smaller than the older ones, not sure if that's relevant.


[root@sc440 boot]# ls -l
total 50235
-rw-r--r--  1 root root   117475 Aug 16 00:16 config-2.6.40.3-0.fc15.x86_64
-rw-r--r--  1 root root   117475 Aug 30 10:44 config-2.6.40.4-5.fc15.x86_64
-rw-r--r--  1 root root   117475 Oct  3 20:45 config-2.6.40.6-0.fc15.x86_64
drwxr-xr-x. 3 root root     1024 Nov 19  2008 efi
-rw-r--r--  1 root root   178436 Jun  2 05:10 elf-memtest86+-4.20
drwxr-xr-x  2 root root     1024 Jun  2 16:32 extlinux
drwxr-xr-x. 2 root root     1024 Nov 17 20:46 grub
-rw-r--r--  1 root root 15394432 Sep  6 10:29 initramfs-2.6.40.3-0.fc15.x86_64.img
-rw-r--r--  1 root root 15528965 Sep 13 10:35 initramfs-2.6.40.4-5.fc15.x86_64.img
-rw-r--r--. 1 root root   574949 Jun  2 13:59 initrd-plymouth.img
drwx------. 2 root root    12288 Dec 26  2008 lost+found
-rw-r--r--  1 root root   176760 Jun  2 05:10 memtest86+-4.20
-rw-------  1 root root  2356599 Aug 16 00:16 System.map-2.6.40.3-0.fc15.x86_64
-rw-------  1 root root  2358422 Aug 30 10:44 System.map-2.6.40.4-5.fc15.x86_64
-rw-------  1 root root  2358812 Oct  3 20:45 System.map-2.6.40.6-0.fc15.x86_64
-rwxr-xr-x  1 root root  3970320 Aug 16 00:16 vmlinuz-2.6.40.3-0.fc15.x86_64
-rwxr-xr-x  1 root root  3972400 Aug 30 10:44 vmlinuz-2.6.40.4-5.fc15.x86_64
-rwxr-xr-x  1 root root  3974320 Oct  3 20:45 vmlinuz-2.6.40.6-0.fc15.x86_64

[root@sc440 boot]# /sbin/dracut -f /boot/initramfs-2.6.40.6-0.fc15.x86_64.img 2.6.40.6-0.fc15.x86_64

[root@sc440 boot]# ls -l
total 58359
-rw-r--r--  1 root root   117475 Aug 16 00:16 config-2.6.40.3-0.fc15.x86_64
-rw-r--r--  1 root root   117475 Aug 30 10:44 config-2.6.40.4-5.fc15.x86_64
-rw-r--r--  1 root root   117475 Oct  3 20:45 config-2.6.40.6-0.fc15.x86_64
drwxr-xr-x. 3 root root     1024 Nov 19  2008 efi
-rw-r--r--  1 root root   178436 Jun  2 05:10 elf-memtest86+-4.20
drwxr-xr-x  2 root root     1024 Jun  2 16:32 extlinux
drwxr-xr-x. 2 root root     1024 Nov 17 20:46 grub
-rw-r--r--  1 root root 15394432 Sep  6 10:29 initramfs-2.6.40.3-0.fc15.x86_64.img
-rw-r--r--  1 root root 15528965 Sep 13 10:35 initramfs-2.6.40.4-5.fc15.x86_64.img
-rw-r--r--  1 root root  8284858 Nov 29 11:26 initramfs-2.6.40.6-0.fc15.x86_64.img
-rw-r--r--. 1 root root   574949 Jun  2 13:59 initrd-plymouth.img
drwx------. 2 root root    12288 Dec 26  2008 lost+found
-rw-r--r--  1 root root   176760 Jun  2 05:10 memtest86+-4.20
-rw-------  1 root root  2356599 Aug 16 00:16 System.map-2.6.40.3-0.fc15.x86_64
-rw-------  1 root root  2358422 Aug 30 10:44 System.map-2.6.40.4-5.fc15.x86_64
-rw-------  1 root root  2358812 Oct  3 20:45 System.map-2.6.40.6-0.fc15.x86_64
-rwxr-xr-x  1 root root  3970320 Aug 16 00:16 vmlinuz-2.6.40.3-0.fc15.x86_64
-rwxr-xr-x  1 root root  3972400 Aug 30 10:44 vmlinuz-2.6.40.4-5.fc15.x86_64
-rwxr-xr-x  1 root root  3974320 Oct  3 20:45 vmlinuz-2.6.40.6-0.fc15.x86_64
Comment 7 Mike Pittaro 2011-11-29 11:45:39 EST
Created attachment 538114 [details]
dracut log for 2.6.40.6-0 run
Comment 8 Josh Boyer 2011-12-07 11:15:15 EST
Odd.  That log doesn't appear to have included any kernel modules...  let's move this over to dracut for now.
Comment 9 Harald Hoyer 2011-12-07 13:49:18 EST
(In reply to comment #6)
> [root@sc440 boot]# /sbin/dracut -f /boot/initramfs-2.6.40.6-0.fc15.x86_64.img
> 2.6.40.6-0.fc15.x86_64

can you attach the logfile dracut.log from:

# /sbin/dracut --debug -f /boot/initramfs-2.6.40.6-0.fc15.x86_64.img \
                          2.6.40.6-0.fc15.x86_64 \
  | tee dracut.log
Comment 10 Harald Hoyer 2011-12-07 14:06:33 EST
(In reply to comment #9)
> (In reply to comment #6)
> > [root@sc440 boot]# /sbin/dracut -f /boot/initramfs-2.6.40.6-0.fc15.x86_64.img
> > 2.6.40.6-0.fc15.x86_64
> 

correction

 can you attach the logfile dracut.log from:
 
# /sbin/dracut --debug -f /boot/initramfs-2.6.40.6-0.fc15.x86_64.img \
                          2.6.40.6-0.fc15.x86_64 \
  2>&1 | tee dracut.log
Comment 11 Mike Pittaro 2011-12-07 14:43:23 EST
Created attachment 542159 [details]
debug dracu log (compressed)

debug log from dracut attached.
Comment 12 Harald Hoyer 2011-12-08 08:36:46 EST
What is the output of:
# modprobe --set-version 2.6.40.6-0.fc15.x86_64 --ignore-install \
  --show-depends ansi_cprng

Did you forget to run "depmod" for this kernel?

What is the output of:

$ ls /lib/modules/2.6.40.6-0.fc15.x86_64/modules.dep*
Comment 13 Josh Boyer 2011-12-08 08:48:02 EST
(In reply to comment #12)
> What is the output of:
> # modprobe --set-version 2.6.40.6-0.fc15.x86_64 --ignore-install \
>   --show-depends ansi_cprng
> 
> Did you forget to run "depmod" for this kernel?

Just a note here, users shouldn't need to run depmod by hand.  It's run as part of the new-kernel-pkg (provided by grubby) invocation that is run in the RPM posttrans script for the kernel.  It's run immediately before the mkinitrd function, which calls dracut.

That's not to say that something didn't go wrong, but the average user shouldn't be worried about depmod at all so there's really nothing for them to forget :).  Mike, the information Harald asks for will help figure out if depmod was run as it should have been.
Comment 14 Mike Pittaro 2011-12-08 10:31:50 EST
I have not run depmod by hand.  So far, I've tried to leave the
system in as close a state as it was after the bad upgrade, to help
troubleshooting. The only change was to revert to the previous
kernel.

progress, I guess - modprobe fails with a missing modules.dep:

[root@sc440 ~]# ls -ld /lib/modules/2.6.40.6-0.fc15.x86_64/
drwxr-xr-x 6 root root 4096 Oct 26 14:43 /lib/modules/2.6.40.6-0.fc15.x86_64/
[root@sc440 ~]# ls -ld /lib/modules/2.6.40.6-0.fc15.x86_64/modules.d*
-rw-r--r-- 1 root root 246 Oct  3 20:45 /lib/modules/2.6.40.6-0.fc15.x86_64/modules.devname
-rw-r--r-- 1 root root  94 Oct  3 20:46 /lib/modules/2.6.40.6-0.fc15.x86_64/modules.drm
[root@sc440 ~]#  
[root@sc440 ~]# modprobe --set-version 2.6.40.6-0.fc15.x86_64 --ignore-install \
>   --show-depends ansi_cprng
WARNING: Deprecated config file /etc/modprobe.conf, all config files belong into /etc/modprobe.d/.
FATAL: Could not load /lib/modules/2.6.40.6-0.fc15.x86_64/modules.dep: No such file or directory
[root@sc440 ~]# 

Let me know if you want me to run depmod by hand, I don't want to
clobber anything. For reference, here is the list of currently
loaded modules from the previous kernel:

[root@sc440 ~]# uname -a
Linux sc440.oxford.lahondaresearch.org 2.6.40.4-5.fc15.x86_64 #1 SMP Tue Aug 30 14:38:32 UTC 2011 x86_64 x86_64 x86_64 GNU/Linux
[root@sc440 ~]# lsmod
Module                  Size  Used by
ppdev                   7508  0 
parport_pc             21184  0 
lp                      9677  0 
parport                32342  3 ppdev,parport_pc,lp
sunrpc                200079  1 
cpufreq_ondemand        5934  2 
acpi_cpufreq            9536  0 
mperf                   1449  1 acpi_cpufreq
snd_hda_codec_hdmi     22499  4 
i3000_edac              3319  0 
edac_core              40186  3 i3000_edac
snd_hda_intel          23896  0 
tg3                   115617  0 
snd_hda_codec          82508  2 snd_hda_codec_hdmi,snd_hda_intel
snd_hwdep               6328  1 snd_hda_codec
snd_seq                52322  0 
snd_seq_device          5941  1 snd_seq
snd_pcm                78424  3 snd_hda_codec_hdmi,snd_hda_intel,snd_hda_codec
snd_timer              19372  2 snd_seq,snd_pcm
snd                    63380  8 snd_hda_codec_hdmi,snd_hda_intel,snd_hda_codec,snd_hwdep,snd_seq,snd_seq_device,snd_pcm,snd_timer
iTCO_wdt               13000  0 
iTCO_vendor_support     2578  1 iTCO_wdt
i2c_i801                9237  0 
soundcore               6267  1 snd
snd_page_alloc          7343  2 snd_hda_intel,snd_pcm
serio_raw               4414  0 
dcdbas                  7716  0 
microcode              18587  0 
ipv6                  284570  66 
raid1                  19025  2 
radeon                690599  0 
nouveau               545642  1 
ttm                    55273  2 nouveau,radeon
drm_kms_helper         26474  2 nouveau,radeon
drm                   193923  4 nouveau,radeon,ttm,drm_kms_helper
mxm_wmi                 1727  1 nouveau
wmi                     9033  1 mxm_wmi
video                  12340  1 nouveau
i2c_algo_bit            4974  2 nouveau,radeon
i2c_core               25712  6 i2c_i801,nouveau,radeon,drm_kms_helper,drm,i2c_algo_bit
Comment 15 Harald Hoyer 2011-12-09 06:14:38 EST
(In reply to comment #14)
> 
> Let me know if you want me to run depmod by hand, I don't want to
> clobber anything. For reference, here is the list of currently
> loaded modules from the previous kernel:
> 

yes, please run as root

# depmod -a

and retry comment 12 and the dracut initramfs generation
Comment 16 Mike Pittaro 2011-12-09 13:51:55 EST
depmod -a  2.6.40.6-0.fc15.x86_64 2>&1 > depmod.2.6.40.6-0.fc15.x86_64.txt

# no output from depmod

[root@sc440 Documents]# ls -l /lib/modules/2.6.40.6-0.fc15.x86_64/modules.dep*
-rw-r--r-- 1 root root 243215 Dec  9 13:21 /lib/modules/2.6.40.6-0.fc15.x86_64/modules.dep
-rw-r--r-- 1 root root 347976 Dec  9 13:21 /lib/modules/2.6.40.6-0.fc15.x86_64/modules.dep.bin

# those timestamps are current 

[root@sc440 Documents]#  modprobe --set-version 2.6.40.6-0.fc15.x86_64 --ignore-install --show-depends ansi_cprng 
WARNING: Deprecated config file /etc/modprobe.conf, all config files belong into /etc/modprobe.d/.
insmod /lib/modules/2.6.40.6-0.fc15.x86_64/kernel/crypto/ansi_cprng.ko 


[root@sc440 Documents] /sbin/dracut --debug -f /boot/initramfs-2.6.40.6-0.fc15.x86_64.img  2.6.40.6-0.fc15.x86_64  2>&1 | tee dracut.20121209.log

# will attach the log
Comment 17 Mike Pittaro 2011-12-09 13:53:17 EST
Created attachment 544653 [details]
updated dracut log
Comment 18 Harald Hoyer 2011-12-13 06:19:48 EST
so, it looks like dracut did it's job well with a correctly "depmod"

reassigning to grubby for not doing "depmod -a"
Comment 19 Brian Lane 2012-05-21 15:51:19 EDT
*** Bug 823631 has been marked as a duplicate of this bug. ***
Comment 20 Fedora End Of Life 2012-08-07 16:02:02 EDT
This message is a notice that Fedora 15 is now at end of life. Fedora
has stopped maintaining and issuing updates for Fedora 15. It is
Fedora's policy to close all bug reports from releases that are no
longer maintained. At this time, all open bugs with a Fedora 'version'
of '15' have been closed as WONTFIX.

(Please note: Our normal process is to give advanced warning of this
occurring, but we forgot to do that. A thousand apologies.)

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, feel free to reopen
this bug and simply change the 'version' to a later Fedora version.

Bug Reporter: Thank you for reporting this issue and we are sorry that
we were unable to fix it before Fedora 15 reached end of life. If you
would still like to see this bug fixed and are able to reproduce it
against a later version of Fedora, you are encouraged to click on
"Clone This Bug" (top right of this page) and open it against that
version of Fedora.

Although we aim to fix as many bugs as possible during every release's
lifetime, sometimes those efforts are overtaken by events. Often a
more recent Fedora release includes newer upstream software that fixes
bugs or makes them obsolete.

The process we are following is described here:
http://fedoraproject.org/wiki/BugZappers/HouseKeeping

Note You need to log in before you can comment on or make changes to this bug.