Bug 835340

That sounds like your initramfs is messed up.  Do:

lsinitrd /boot/initramfs-3.4.2-1.fc16.x86_64.img | grep libgcc

and see if it outputs any files.

Also, you say that 3.3.8-1 is the version impacted, but your reproduce steps only mention 3.4.2-1.

Josh,

Here's what I get:

joker bash-4.2 ~# lsinitrd /boot/initramfs-3.4.2-1.fc16.x86_64.img | grep libgcc
-rwxr-xr-x   1 root     root        87016 Mar  6 09:54 run/initramfs/lib64/libgcc_s-4.6.3-20120306.so.1
lrwxrwxrwx   1 root     root           28 Jun 26 12:57 run/initramfs/lib64/libgcc_s.so.1 -> libgcc_s-4.6.3-20120306.so.1

Notice that the "run" above does NOT have "/var/" preceding it.

I re-created the initramfs-3.4.2-1.fc16.x86_64.img file in /boot but get the same results sort of. Now it's a panic loop which I'm unable to capture. Argh! The message "udevd... killing /sbin/blkid -o udev -u /dev/sda6 appears frequently then there's a trace dump then it repeats... looping I'm guessing. I let it run for about 20 mins...

My partitions are old style (see below). Perhaps someone has made an assumption that everyone has all these partitions in one big partition?


/dev/sda2        2093160   1638816    433376  80% /
/dev/sda5        3138272    829016   2277800  27% /var
/dev/sda1        2093160    241204   1830988  12% /boot
/dev/sda7       30334620  19285024  11049596  64% /usr
/dev/sda8      668965300 534091292 128177460  81% /export/home
/dev/sda6       21953708  21379060    354448  99% /opt

Which version of dracut is this?

What is the contents of /etc/dracut.conf and /etc/dracut.conf.d/*

Harald,

At the time of the failure, the version of dracut was dracut-013-22.fc16.noarch. I got a newer version of dracut from the git repository for the purposes of getting system dumps. The newer version is dracut-018-55.git20120606.fc16.noarch.

George...

(In reply to comment #3)
> What is the contents of /etc/dracut.conf and /etc/dracut.conf.d/*

you did not answer my question

Does it work with dracut-018-55.git20120606.fc16 from the updates?

Note, you have recreate the initramfs.

# dracut -f /boot/initramfs-<kernel-version>.img <kernel-version>
or a shortcut:
# dracut -f '' <kernel-version>

Created attachment 596675 [details]
gzip'd tar of dracut config fiies

Harald,

It looks like my config file tar ball didn't make it into this bug.

I have just rebuilt the initramfs and will test in a few minutes and post the results here.

Thanks,

George...

there is a fips.conf in /etc/dracut.conf.d . Are you sure, you want a FIPS system? If not, uninstall dracut-fips or remove the config file.

Harald,

I have not made any config changes with dracut. As far as I know, the FIPS stuff came along with dracut updates and/or the install of this system.

Also, I re-recreated the initramfs with dracut. The resulting boot was highly unstable. Without a way to record the messages during boot (new systemd?) I have no way to show just what happened. There are some messages in /var/log/messages though. I see there's a new kernel in town, I'll try booting this and then do the dracut if it appears necessary.

THANKS!

George...

(In reply to comment #8)
> Harald,
> 
> I have not made any config changes with dracut. As far as I know, the FIPS
> stuff came along with dracut updates and/or the install of this system.
> 
> Also, I re-recreated the initramfs with dracut. The resulting boot was
> highly unstable. Without a way to record the messages during boot (new
> systemd?) I have no way to show just what happened. There are some messages
> in /var/log/messages though. I see there's a new kernel in town, I'll try
> booting this and then do the dracut if it appears necessary.
> 
> THANKS!
> 
> George...

# yum remove dracut-fips

with systemd and udev > 020-57 debugging will be a lot easier... just try rawhide tomorrow

# journalctl -a

will display _all_ messages

Harald,

I removed the dracut-fps.

What is "systemd and udev > 020-57"? A config file option? Which config file? Anything else that I need to do?

journalctl -a gives me a "command not found" msg. What package?

Thanks,

George...

(In reply to comment #10)
> Harald,
> 
> I removed the dracut-fps.
> 
> What is "systemd and udev > 020-57"? A config file option? Which config
> file? Anything else that I need to do?
> 
> journalctl -a gives me a "command not found" msg. What package?
> 
> Thanks,
> 
> George...

Oh ... sorry, you are on F16. Forget that comment.

(In reply to comment #8)
> Harald,
> 
> I have not made any config changes with dracut. As far as I know, the FIPS
> stuff came along with dracut updates and/or the install of this system.
> 
> Also, I re-recreated the initramfs with dracut. The resulting boot was
> highly unstable. Without a way to record the messages during boot (new
> systemd?) I have no way to show just what happened. There are some messages
> in /var/log/messages though. I see there's a new kernel in town, I'll try
> booting this and then do the dracut if it appears necessary.
> 
> THANKS!
> 
> George...

With F16 the debugging procedure is as following:

- add "rd.debug rd.shell" to the kernel command line
- if dracut does not boot, you are dropped to a shell
- "/run/initramfs/init.log" and the output of dmesg contain the debug information
- you can mount /boot or an USB stick by hand and copy over the /run/initramfs/init.log

Created attachment 599870 [details]
flat file containing current dmesg buffer

Harald,

I still have a kernel that doesn't want to boot and/or execute properly. Here's the current dmesg buffer from my system. It shows a TON of msgs from dracut.

I'm attempting to capture console messages from the failing kernel but that's not working very well. I'm trying the netconsole procedure now.

I have only one operational kernel now since I have removed the kernel-3.4.4-4.fc16.x86_64 which is still having trouble from my system.

Your thoughts please?

Regards,

George...

[   49.071088] bootchart-collector started as pid 1769 with 2 args: '--dump' '/tmp/bootchart.JB5pGt7Qfn' 

seems like bootchart is running... maybe you should disable that first.

you could try to regenerate the initramfs.

# dracut -f /boot/initramfs-3.3.8-1.fc16.x86_64.img 3.3.8-1.fc16.x86_64

And maybe try and see, if it is dracut that is failing by verifying that you cannot get to a shell with "init=/bin/sh" on the kernel command line.

Harald,

I ran "yum update" yesterday and got a new kernel (3.4.6-1.fc16.x86_64) and this problem seems to have gone away.

We can wait a few days and then close this case if you want to.

Regards and THANKS for all your help.

George...