Bug 835340 - panic during boot: unable to open libgcc_s.so.1
panic during boot: unable to open libgcc_s.so.1
Status: CLOSED CANTFIX
Product: Fedora
Classification: Fedora
Component: dracut (Show other bugs)
16
x86_64 Linux
unspecified Severity high
: ---
: ---
Assigned To: dracut-maint
Fedora Extras Quality Assurance
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2012-06-25 23:25 EDT by George R. Goffe
Modified: 2012-09-27 05:30 EDT (History)
7 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2012-09-27 05:30:26 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
gzip'd tar of dracut config fiies (1.07 KB, application/x-gzip)
2012-07-06 13:54 EDT, George R. Goffe
no flags Details
flat file containing current dmesg buffer (245.10 KB, application/octet-stream)
2012-07-23 18:22 EDT, George R. Goffe
no flags Details

  None (edit)
Description George R. Goffe 2012-06-25 23:25:37 EDT
Description of problem:

Panic during boot with msg, "unable to open shared libraries libgcc_s.so.1. No such file or directory.

Version-Release number of selected component (if applicable):

3.3.8-1.fc16.x86_64

How reproducible:

Always

Steps to Reproduce:
1.Boot system
2.select kernel from grub2 menu: 3.4.2-1.fc16.x86_64 or 3.4.2-1.fc16.x86_64.debug
3.observe panic
  
Actual results:

Panic

Expected results:

No panic

Additional info:
Comment 1 Josh Boyer 2012-06-26 08:54:44 EDT
That sounds like your initramfs is messed up.  Do:

lsinitrd /boot/initramfs-3.4.2-1.fc16.x86_64.img | grep libgcc

and see if it outputs any files.

Also, you say that 3.3.8-1 is the version impacted, but your reproduce steps only mention 3.4.2-1.
Comment 2 George R. Goffe 2012-06-26 17:42:07 EDT
Josh,

Here's what I get:

joker bash-4.2 ~# lsinitrd /boot/initramfs-3.4.2-1.fc16.x86_64.img | grep libgcc
-rwxr-xr-x   1 root     root        87016 Mar  6 09:54 run/initramfs/lib64/libgcc_s-4.6.3-20120306.so.1
lrwxrwxrwx   1 root     root           28 Jun 26 12:57 run/initramfs/lib64/libgcc_s.so.1 -> libgcc_s-4.6.3-20120306.so.1

Notice that the "run" above does NOT have "/var/" preceding it.

I re-created the initramfs-3.4.2-1.fc16.x86_64.img file in /boot but get the same results sort of. Now it's a panic loop which I'm unable to capture. Argh! The message "udevd... killing /sbin/blkid -o udev -u /dev/sda6 appears frequently then there's a trace dump then it repeats... looping I'm guessing. I let it run for about 20 mins...

My partitions are old style (see below). Perhaps someone has made an assumption that everyone has all these partitions in one big partition?


/dev/sda2        2093160   1638816    433376  80% /
/dev/sda5        3138272    829016   2277800  27% /var
/dev/sda1        2093160    241204   1830988  12% /boot
/dev/sda7       30334620  19285024  11049596  64% /usr
/dev/sda8      668965300 534091292 128177460  81% /export/home
/dev/sda6       21953708  21379060    354448  99% /opt
Comment 3 Harald Hoyer 2012-07-03 06:20:59 EDT
Which version of dracut is this?

What is the contents of /etc/dracut.conf and /etc/dracut.conf.d/*
Comment 4 George R. Goffe 2012-07-05 19:39:22 EDT
Harald,

At the time of the failure, the version of dracut was dracut-013-22.fc16.noarch. I got a newer version of dracut from the git repository for the purposes of getting system dumps. The newer version is dracut-018-55.git20120606.fc16.noarch.

George...
Comment 5 Harald Hoyer 2012-07-06 04:11:52 EDT
(In reply to comment #3)
> What is the contents of /etc/dracut.conf and /etc/dracut.conf.d/*

you did not answer my question

Does it work with dracut-018-55.git20120606.fc16 from the updates?

Note, you have recreate the initramfs.

# dracut -f /boot/initramfs-<kernel-version>.img <kernel-version>
or a shortcut:
# dracut -f '' <kernel-version>
Comment 6 George R. Goffe 2012-07-06 13:54:37 EDT
Created attachment 596675 [details]
gzip'd tar of dracut config fiies

Harald,

It looks like my config file tar ball didn't make it into this bug.

I have just rebuilt the initramfs and will test in a few minutes and post the results here.

Thanks,

George...
Comment 7 Harald Hoyer 2012-07-09 02:35:36 EDT
there is a fips.conf in /etc/dracut.conf.d . Are you sure, you want a FIPS system? If not, uninstall dracut-fips or remove the config file.
Comment 8 George R. Goffe 2012-07-09 13:57:43 EDT
Harald,

I have not made any config changes with dracut. As far as I know, the FIPS stuff came along with dracut updates and/or the install of this system.

Also, I re-recreated the initramfs with dracut. The resulting boot was highly unstable. Without a way to record the messages during boot (new systemd?) I have no way to show just what happened. There are some messages in /var/log/messages though. I see there's a new kernel in town, I'll try booting this and then do the dracut if it appears necessary.

THANKS!

George...
Comment 9 Harald Hoyer 2012-07-10 04:48:20 EDT
(In reply to comment #8)
> Harald,
> 
> I have not made any config changes with dracut. As far as I know, the FIPS
> stuff came along with dracut updates and/or the install of this system.
> 
> Also, I re-recreated the initramfs with dracut. The resulting boot was
> highly unstable. Without a way to record the messages during boot (new
> systemd?) I have no way to show just what happened. There are some messages
> in /var/log/messages though. I see there's a new kernel in town, I'll try
> booting this and then do the dracut if it appears necessary.
> 
> THANKS!
> 
> George...

# yum remove dracut-fips

with systemd and udev > 020-57 debugging will be a lot easier... just try rawhide tomorrow

# journalctl -a

will display _all_ messages
Comment 10 George R. Goffe 2012-07-11 18:21:58 EDT
Harald,

I removed the dracut-fps.

What is "systemd and udev > 020-57"? A config file option? Which config file? Anything else that I need to do?

journalctl -a gives me a "command not found" msg. What package?

Thanks,

George...
Comment 11 Harald Hoyer 2012-07-12 04:22:45 EDT
(In reply to comment #10)
> Harald,
> 
> I removed the dracut-fps.
> 
> What is "systemd and udev > 020-57"? A config file option? Which config
> file? Anything else that I need to do?
> 
> journalctl -a gives me a "command not found" msg. What package?
> 
> Thanks,
> 
> George...

Oh ... sorry, you are on F16. Forget that comment.
Comment 12 Harald Hoyer 2012-07-12 04:24:35 EDT
(In reply to comment #8)
> Harald,
> 
> I have not made any config changes with dracut. As far as I know, the FIPS
> stuff came along with dracut updates and/or the install of this system.
> 
> Also, I re-recreated the initramfs with dracut. The resulting boot was
> highly unstable. Without a way to record the messages during boot (new
> systemd?) I have no way to show just what happened. There are some messages
> in /var/log/messages though. I see there's a new kernel in town, I'll try
> booting this and then do the dracut if it appears necessary.
> 
> THANKS!
> 
> George...

With F16 the debugging procedure is as following:

- add "rd.debug rd.shell" to the kernel command line
- if dracut does not boot, you are dropped to a shell
- "/run/initramfs/init.log" and the output of dmesg contain the debug information
- you can mount /boot or an USB stick by hand and copy over the /run/initramfs/init.log
Comment 13 George R. Goffe 2012-07-23 18:22:07 EDT
Created attachment 599870 [details]
flat file containing current dmesg buffer

Harald,

I still have a kernel that doesn't want to boot and/or execute properly. Here's the current dmesg buffer from my system. It shows a TON of msgs from dracut.

I'm attempting to capture console messages from the failing kernel but that's not working very well. I'm trying the netconsole procedure now.

I have only one operational kernel now since I have removed the kernel-3.4.4-4.fc16.x86_64 which is still having trouble from my system.

Your thoughts please?

Regards,

George...
Comment 14 Harald Hoyer 2012-07-24 06:13:36 EDT
[   49.071088] bootchart-collector started as pid 1769 with 2 args: '--dump' '/tmp/bootchart.JB5pGt7Qfn' 

seems like bootchart is running... maybe you should disable that first.

you could try to regenerate the initramfs.

# dracut -f /boot/initramfs-3.3.8-1.fc16.x86_64.img 3.3.8-1.fc16.x86_64

And maybe try and see, if it is dracut that is failing by verifying that you cannot get to a shell with "init=/bin/sh" on the kernel command line.
Comment 15 George R. Goffe 2012-07-29 15:36:18 EDT
Harald,

I ran "yum update" yesterday and got a new kernel (3.4.6-1.fc16.x86_64) and this problem seems to have gone away.

We can wait a few days and then close this case if you want to.

Regards and THANKS for all your help.

George...

Note You need to log in before you can comment on or make changes to this bug.