Bug 336161

Nice. When during the boot process does this happen?

I can't say exactly when because the screen is flooded by printk messages about
the init segfault. What I can say is that it happens quickly after the kernel
initialization and before the /etc/rc scripts.  

The problem is the same with the 'single' kernel option on.

Does:

a) booting with init=/bin/bash work?
b) booting with selinux=0 work?

I have tested by starting from a new installation of fc6 (default packages from
the fc6 release, no yum update) then upgraded to f8test3 and the problem is the
same.  

The last line before the init segfault is:
"Freeing unused kernel memory ..."
"Write protecting the kernel read only data: 1036k"

a) booting with init=/bin/bash work? No
b) booting with selinux=0 work? No

One more problem that might be related is that the fc6 kernel is the default
choice in grub (which does not boot of course).  That grub entry should be
removed...

Created attachment 232491 [details]
upgrade.log of fc6 to f8test3

Created attachment 232501 [details]
upgrade.log.syslog of fc6 to f8test3

How does booting with init=/bin/bash fail - what sort of errors?

Exactly the same error as shown in my first comment.  In fact, init=/anything
shows the same error.

To be able to access upgrade.log and upgrade.log.syslog, I have booted from a
livecd.  If needed, I can provide more log files.

If you boot from that livecd and chroot into the system, do bash, ls, etc. run?

I have what might be the same problem; at least it's apparently related.

With kernel 2.6.23.1-23 on an i386 (nForce-based Asus A7N266-VM with an Athlon
XP 1800+, 1 GB ram) the boot process very quickly prints (copied by hand):

...
[Linux -initrd @ 0x37c72000, 0x37d681 bytes]

Booting: ABCDEUncompressing Linux... Ok, booting the kernel.
init[1]: segfault at 00000000 eip 00000000 esp bfbf891c error 4
init[1]: segfault at 00000000 eip 00000000 esp bfbf891c error 4
init[1]: segfault at 00000000 eip 00000000 esp bfbf891c error 4
...

The last line is printed 10 times very quickly, and then once every about 3
seconds.

The same thing happens on every boot. It's definitely the kernel and not init
that has a problem, as the originally installed kernel
(2.6.23-0.214.rc8.git2.fc8) works fine.

(In reply to comment #9)
> If you boot from that livecd and chroot into the system, do bash, ls, etc. run?

Chroot works well from the livecd:

[root@localhost ~]# mount /dev/sda3 /mnt
[root@localhost ~]# ls /mnt
bin   dev  halt  lib    lost+found  mnt  proc  sbin     srv  tmp  var
boot  etc  home  lib64  media       opt  root  selinux  sys  usr
[root@localhost ~]# chroot /mnt /bin/bash
[root@localhost /]# ls
bin   dev  halt  lib    lost+found  mnt  proc  sbin     srv  tmp  var
boot  etc  home  lib64  media       opt  root  selinux  sys  usr
[root@localhost /]# ls /mnt
[root@localhost /]# /sbin/init
Usage: init 0123456SsQqAaBbCcUu
[root@localhost /]#

OK, so:

- the fact that it happens with any init= (/sbin/init, /bin/bash) implies either
an issue with the base library set or possibly the kernel
- the fact that the same library set runs under chroot seems to implicate the kernel

Pushing there for now.

Does specifying a statically-linked program as init work, like "init=/sbin/nash"?

And, boot the rescue disk and remove "quiet" and "rhgb" from the kernel line in
/etc/grub.conf. Then see what it prints just before the segfaults start.

(In reply to comment #13)
> Does specifying a statically-linked program as init work, like "init=/sbin/nash"?

Prints the same messages as in comment #0 and #4.  I cannot get more lines
before the segfault because it gets quickly flooded. 

> And, boot the rescue disk and remove "quiet" and "rhgb" from the kernel line in
> /etc/grub.conf. Then see what it prints just before the segfaults start.
> 

I removed the "quiet" and "rhgb" from the kernel line in the grub boot menu with
the 'e' key and nothing more is printed.

Does booting with 'vdso=0' help?

(In reply to comment #14)
> (In reply to comment #13)
> > Does specifying a statically-linked program as init work, like
"init=/sbin/nash"?
> 
> Prints the same messages as in comment #0 and #4.  I cannot get more lines
> before the segfault because it gets quickly flooded. 
> 

Use boot_delay=500 to slow down the boot messages. Adjust the 500 as necessary
to get a reasonable slowdown.

Also, if you can tell, is this /sbin/init that is crashing, or the init from the
initrd?

OK, reproduced here - it's the initrd that's crashing. 

Jean-Philippe, Mikkel, I would suspect if you remake your initrd/initramfs, it
will boot fine.

So, what's happening on at least my test box is that the upgrade, done via
anaconda, installs new packages before removing the old packages. mkinitrd's ELF
dependency finder is pulling in the old copy of libc, instead of the new one. 
old libc + new ld-linux.so.2 == segfault.

Reassigning to mkinitrd.

Erm, reverse that. New libc, old ld-linux.so.2.

Created attachment 233291 [details]
patch for this issue

In the presence of multiple ld.so, we need to iterate over all of them,
preferring the latest, not the first. In doing so, we also need to ignore
symlinks.

Tested, this seems to DTRT.

Created attachment 233301 [details]
updated patch

Here's a fixed patch. Oops.

Built as 6.0.19-3.fc8.

*** Bug 343951 has been marked as a duplicate of this bug. ***

*** Bug 333521 has been marked as a duplicate of this bug. ***