This service will be undergoing maintenance at 00:00 UTC, 2016-08-01. It is expected to last about 1 hours
Bug 638091 - Kernel panic at boot after glibc update
Kernel panic at boot after glibc update
Status: CLOSED ERRATA
Product: Fedora
Classification: Fedora
Component: glibc (Show other bugs)
14
All Linux
high Severity urgent
: ---
: ---
Assigned To: Andreas Schwab
Fedora Extras Quality Assurance
:
: 638114 638202 638203 638208 638210 (view as bug list)
Depends On:
Blocks: F14Blocker/F14FinalBlocker
  Show dependency treegraph
 
Reported: 2010-09-28 03:45 EDT by Artem
Modified: 2010-09-30 02:16 EDT (History)
21 users (show)

See Also:
Fixed In Version: glibc-2.12.90-14
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2010-09-30 02:16:00 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:


Attachments (Terms of Use)
screenshot with kernel panic 2.6.35.4-28.fc14.i686.PAE #1 (297.03 KB, image/jpeg)
2010-09-28 09:55 EDT, Steve Tyler
no flags Details

  None (edit)
Description Artem 2010-09-28 03:45:26 EDT
Description of problem:
After update to glibc I'm not able to boot system anymore


Version-Release number of selected component (if applicable):
Sep 28 09:00:36 Updated: glibc-common-2.12.90-13.x86_64                                                                                                                             
Sep 28 09:00:45 Updated: glibc-2.12.90-13.x86_64                                                                                                                                    
Sep 28 09:00:47 Updated: glibc-headers-2.12.90-13.x86_64  

How reproducible:
always

Steps to Reproduce:
1.
2.
3.
  
Actual results:


Expected results:


Additional info:
After update (before system restart) such records appeared in messages:

........
Sep 28 10:09:46 kivagoncharov1 kernel: Aborting core                                                                                                                                
Sep 28 10:09:46 kivagoncharov1 init: tty (/dev/tty2) main process (1734) killed by SEGV signal                                                                                      
Sep 28 10:09:46 kivagoncharov1 init: tty (/dev/tty2) main process ended, respawning                                                                                                 
Sep 28 10:09:46 kivagoncharov1 kernel: sh[16714]: segfault at 3d03c1fbf8 ip 0000003d03a033c7 sp 00007fff9fc96ef0 error 7 in ld-2.12.90.so[3d03a00000+1f000]                         
Sep 28 10:09:46 kivagoncharov1 kernel: abrt-hook-ccpp[16715]: segfault at 3d03c1fbf8 ip 0000003d03a033c7 sp 00007fff0aa62620 error 7 in ld-2.12.90.so[3d03a00000+1f000]             
Sep 28 10:09:46 kivagoncharov1 kernel: Process 16715(abrt-hook-ccpp) has RLIMIT_CORE set to 1                                                                                       
Sep 28 10:09:46 kivagoncharov1 kernel: Aborting core                                                                                                                                
Sep 28 10:09:46 kivagoncharov1 init: tty (/dev/tty2) main process (16714) killed by SEGV signal                                                                                     
Sep 28 10:09:46 kivagoncharov1 init: tty (/dev/tty2) main process ended, respawning                                                                                                 
Sep 28 10:09:46 kivagoncharov1 kernel: sh[16716]: segfault at 3d03c1fbf8 ip 0000003d03a033c7 sp 00007fff3bedd920 error 7 in ld-2.12.90.so[3d03a00000+1f000]                         
Sep 28 10:09:46 kivagoncharov1 kernel: abrt-hook-ccpp[16717]: segfault at 3d03c1fbf8 ip 0000003d03a033c7 sp 00007fffd4b4c2c0 error 7 in ld-2.12.90.so[3d03a00000+1f000]             
Sep 28 10:09:46 kivagoncharov1 kernel: Process 16717(abrt-hook-ccpp) has RLIMIT_CORE set to 1 

and pretty much more for all running processes. Also after screen was locked I had no possibility to unlock it due to some process crashed.
Comment 1 Artem 2010-09-28 03:57:47 EDT
I can see problem only trying to boot with nomodeset (listing through the paper, therefore might be something wrong):

dracut:switching root
init[1]: segfault at 3d03c1fbf8 ip 0000003d03a033c7 sp 00007fff4ad80140 error 7 in ld-2.12.90.so[3d03a00000+1f000] 
init used greatest stack depth: 3069 bytes left
Kernel panic - ....
Pid:1, comm: init not tained kernel 2.6.35.4-28.fc14.x86_64



I tried to boot with older kernel, but it is the same.
Comment 2 Edouard Bourguignon 2010-09-28 04:37:18 EDT
same problem here, glibc-2.12.90-13 update broke everything :(
Comment 3 Andreas Schwab 2010-09-28 06:30:24 EDT
*** Bug 638114 has been marked as a duplicate of this bug. ***
Comment 4 Andreas Schwab 2010-09-28 08:28:46 EDT
I'm unable to reproduce.  Please provide a backtrace.
Comment 5 Artem 2010-09-28 08:37:49 EDT
Well, it's hard, since I had already downgraded to have working system, but while I had it I tried to write down at least function names in backtrace, so here it is (without addresses):

panic
do_exit
do_group_exit
get_signal_to_deliver
do_signal
printk
bad_area_access_error+0x47/0x4e
lockdep_sys_exit
do_notify_resume
retinit_signal

It is basically all (except of addresses). Is it sufficient, or do you need addresses? If it is not enough info please tell me how can I get complete backtrace (maybe to add some option in grub)?
Comment 6 Jeff Layton 2010-09-28 09:41:04 EDT
I've had a similar problem on two machines that I've upgraded with this glibc version. It seems to work for a while after the upgrade and then everything starts segfaulting. I booted to a rescue cd and did a rpm -V glibc. Several libraries, including ld-*.so were corrupt with bad sizes and MD5 sums.

I reinstalled the same glibc package and it corrected the problem, but now I'm wondering -- what corrupted the files in the first place? prelink problems maybe?
Comment 7 Jeff Layton 2010-09-28 09:42:49 EDT
As a quick test, I just ran /etc/cron.daily/prelink after fixing one of these machines and sure enough, everything started segfaulting again almost immediately. That seems to be triggering whatever the problem is.
Comment 8 Jeffrey C. Ollie 2010-09-28 09:53:17 EDT
For the record, I hit this problem as well.  I fixed my system by booting the F14 RC3 live CD and using "yum --installroot=... downgrade glibc\*" to downgrade glibc to 2.12.90-11.

Paul Frields posted a good transcription the boot panic here:

http://lists.fedoraproject.org/pipermail/test/2010-September/094213.html
Comment 9 Steve Tyler 2010-09-28 09:55:51 EDT
Created attachment 450204 [details]
screenshot with kernel panic 2.6.35.4-28.fc14.i686.PAE #1

screenshot showing kernel panic after dracut, init.

This is on my laptop after the last update.
Could not reproduce in a VM.
Comment 10 Nikolay Nikolaev 2010-09-28 09:59:03 EDT
I'm manually copying it from the problematic console, so there may be some typing errors:

dracut: Switching root
init[1]: segfault at 381421fbf8 ip 00000038140033c7 sp 00007fffa38ddd30 error 7 in ld-2.12.90.so[3814000000+1f000]
Kernel panic - not syncing: Attempted to kill init!
Pid: 1, comm: init Tainted: G        W   2.6.35.4-28.fc14.x86_64 #1
Call Trace:
 [<ffffffff8149b08b>] panic+0x8b/0x110
 [<ffffffff81054a89>] do_exit+0x7b/0x7d0
 [<ffffffff81055474>] do_group_exit+0x88/0xb6
 [<ffffffff810628f4>] get_signal_to_deliver+0x3d6/0x3f5
 [<ffffffff8107e44f>] ? lock_release+0x19a/0x1a6
 [<ffffffff81008f91>] do_signal+0x72/0x690
 [<ffffffff81042b7c>] ? mmdrop+0x1a/0x2a
 [<ffffffff8149b178>] ? printk+0x68/0x70
 [<ffffffff8103f052>] ? need_resched+0x23/0x2d
 [<ffffffff8149b85f>] ? schedule+0x5dd/0x5f7
 [<ffffffff8107f15e>] ? lockdep_sys_exit+0x20/0x76
 [<ffffffff810095f0>] do_notify_resume+0x28/0x86
 [<ffffffff8149e39b>] retint_signal+0x4d/0x92
Comment 11 Steve Tyler 2010-09-28 10:19:30 EDT
(In reply to comment #7)
> As a quick test, I just ran /etc/cron.daily/prelink after fixing one of these
> machines and sure enough, everything started segfaulting again almost
> immediately. That seems to be triggering whatever the problem is.

Excellent suggestion.
Reproduced in a VM by running "anacron -fn" after updating.
Now reboot, ls, ps segfault. pwd, true, echo do not.
Comment 12 Fedora Update System 2010-09-28 10:24:46 EDT
glibc-2.12.90-14 has been submitted as an update for Fedora 14.
https://admin.fedoraproject.org/updates/glibc-2.12.90-14
Comment 13 Andreas Schwab 2010-09-28 10:25:37 EDT
*** Bug 638210 has been marked as a duplicate of this bug. ***
Comment 14 Kamil Dudka 2010-09-28 10:28:42 EDT
*** Bug 638202 has been marked as a duplicate of this bug. ***
Comment 15 Kamil Dudka 2010-09-28 10:28:44 EDT
*** Bug 638203 has been marked as a duplicate of this bug. ***
Comment 16 Joachim Backes 2010-09-28 10:41:47 EDT
(In reply to comment #6)
> I've had a similar problem on two machines that I've upgraded with this glibc
> version. It seems to work for a while after the upgrade and then everything
> starts segfaulting.

Exactly what I reported in #638114 (system worked for a while, then all started with segfaulting). System was no more rebootable.

> I booted to a rescue cd and did a rpm -V glibc. Several
> libraries, including ld-*.so were corrupt with bad sizes and MD5 sums.
> I reinstalled the same glibc package and it corrected the problem, but now I'm
> wondering -- what corrupted the files in the first place? prelink problems
> maybe?
Comment 17 Edouard Bourguignon 2010-09-28 10:48:27 EDT
seems to be ok with glibc-2.12.90-14
Comment 18 Joachim Backes 2010-09-28 10:58:00 EDT
(In reply to comment #17)
> seems to be ok with glibc-2.12.90-14

I can confirm this. No more crash messages in /var/log messages.
Comment 19 Andreas Schwab 2010-09-28 12:14:13 EDT
*** Bug 638208 has been marked as a duplicate of this bug. ***
Comment 20 Fedora Update System 2010-09-28 13:32:21 EDT
glibc-2.12.90-14 has been pushed to the Fedora 14 testing repository.  If problems still persist, please make note of it in this bug report.
 If you want to test the update, you can install it with 
 su -c 'yum --enablerepo=updates-testing update glibc'.  You can provide feedback for this update here: https://admin.fedoraproject.org/updates/glibc-2.12.90-14
Comment 21 Ronald L Humble 2010-09-28 15:10:48 EDT
Command below from Live! CD shows only 90-13. I assume I am in error?

yum --enablerepo=updates-testing --installroot=... list glibc

Installed Packages
glibc.x86_64                                        2.12.90-13                                        @updates-testing
Available Packages
glibc.i686                                          2.12.90-13                                        updates-testing 

(In reply to comment #20)
> glibc-2.12.90-14 has been pushed to the Fedora 14 testing repository.  If
> problems still persist, please make note of it in this bug report.
>  If you want to test the update, you can install it with 
>  su -c 'yum --enablerepo=updates-testing update glibc'.  You can provide
> feedback for this update here:
> https://admin.fedoraproject.org/updates/glibc-2.12.90-14
Comment 22 Steve Tyler 2010-09-28 15:41:54 EDT
(In reply to comment #21)
> Command below from Live! CD shows only 90-13. I assume I am in error?
...
glibc-2.12.90-14 hasn't propagated to the mirrors yet:
$ sudo repoquery --releasever=14 --enablerepo=fedora --enablerepo='updates*' --disablerepo='rpm*' glibc
glibc-0:2.12.90-13.i686
glibc-0:2.12.90-13.x86_64

It's here if you don't want to wait:
http://koji.fedoraproject.org/koji/buildinfo?buildID=197187
Comment 23 Martin Naď 2010-09-28 16:34:10 EDT
glibc-0:2.12.90-14.x86_64 work ok, but restart work only with root privileges with user moustly hangs .Maybe is bug in systemd but with glibc-0:2.12.90-10.x86_64 systemd work ok ,I not sure on 100%
Comment 24 Artem 2010-09-29 06:33:27 EDT
glibc-2.12.90-14 works for me as well, thanks for the fast fix
Comment 25 Steve Tyler 2010-09-29 08:27:55 EDT
Also confirming that glibc-2.12.90-14 is working on my laptop and in a VM.

Here is how I recovered after glibc-2.12.90-13 was installed:

yum isn't on the F14-Beta-RC3 DVDs (nor is it on the F11, F12, or F13 DVDs),
so the rescue shell tries to run /mnt/sysimage/usr/bin/yum
and that fails with
ImportError: No module named yummain

However, rpm is on the discs,
so I used rpm to downgrade to glibc* on the DVD:

1. Boot the F14 installer DVD in rescue mode
   (no networking, target system mounted).
2. mkdir /mnt/dvd
3. mount /dev/sr0 /mnt/dvd
4. cd /mnt/dvd/Packages
5. rpm --root /mnt/sysimage --oldpackage -Uv glibc-2*.rpm glibc-common-2*.rpm
6. chroot /mnt/sysimage
7. ls # to verify that the old packages fix the segfault problem
8. exit
9. reboot
10. Run /etc/cron.daily/prelink to verify that the problem does not recur.
    (anacron will run prelink, but after a random delay,
    which can be disabled in /etc/anacrontab.
    That's why the segfaults began appearing a random time after booting.)
Comment 26 Jon Masters 2010-09-29 14:51:33 EDT
Andreas: Any chance of a summary in this bug describing what happened?
Comment 27 Fedora Update System 2010-09-30 02:15:35 EDT
glibc-2.12.90-14 has been pushed to the Fedora 14 stable repository.  If problems still persist, please make note of it in this bug report.

Note You need to log in before you can comment on or make changes to this bug.