Bug 182731

Summary: kernel locks up after write-protecting kernel rodata
Product: [Fedora] Fedora Reporter: Paul Black <paul.0000.black>
Component: kernelAssignee: Dave Jones <davej>
Status: CLOSED RAWHIDE QA Contact: Brian Brock <bbrock>
Severity: high Docs Contact:
Priority: medium    
Version: 5CC: adri_, ask, avi, bkyoung, briang, david.r.bentley, dkelson, fcdocs, fenlason, gnomeuser, hdegoede, jjniven, jr.express, k.georgiou, kheine7, lsof, michal, mishu, ndbecker2, pfrields, redhat-bugzilla, urorzm+bugzilla.redhat.com, wtogami
Target Milestone: ---   
Target Release: ---   
Hardware: i686   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2006-03-11 22:27:19 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
new-kernel-pkg aborted none

Description Paul Black 2006-02-24 11:56:44 UTC
Description of problem:
Kernel stop booting, hasn't locked up because CTRL-ALT-DEL still works.

Version-Release number of selected component (if applicable):
1977

How reproducible:
Every time

Steps to Reproduce:
Boot machine  


Actual results:
kernel: Using IPI No-Shortcut mode
kernel: ACPI wakeup devices: 
kernel: VBTN PCI0 USB0 USB2 USB3 PCI1  KBD 
kernel: ACPI: (supports S0 S1 S3 S4 S5)
kernel: Freeing unused kernel memory: 208k freed
kernel: Write protecting the kernel read-only data: 368k
kernel: input: ImPS/2 Generic Wheel Mouse as /class/input/input1

Nothing more is printed, no disk activity. Keyboard works fine.


Expected results:
The 1975 kernel produces the following:
kernel: Using IPI No-Shortcut mode
kernel: ACPI wakeup devices: 
kernel: VBTN PCI0 USB0 USB2 USB3 PCI1  KBD 
kernel: ACPI: (supports S0 S1 S3 S4 S5)
kernel: Freeing unused kernel memory: 208k freed
kernel: Write protecting the kernel read-only data: 368k
kernel: device-mapper: 4.5.0-ioctl (2005-10-04) initialised: dm-devel
kernel: input: ImPS/2 Generic Wheel Mouse as /class/input/input1
kernel: kjournald starting.  Commit interval 5 seconds
kernel: EXT3-fs: mounted filesystem with ordered data mode.

Note the extra device mapper line.

Additional info:

Comment 1 Dax Kelson 2006-02-24 18:47:08 UTC
I Dell Dimesion 4700 and I'm seeing this too.

The last line printed on my screen is:

Write protecting the kernel read-only data: 368k

Comment 2 Brian Gaynor 2006-02-24 22:06:59 UTC
I am seeing the same thing on a Dell Inspiron 5160 running the SMP kernel. Last
line printed is for the Synaptics touchpad, and CTRL-ALT-DEL still works. From
the fan noise I would guess the CPU is in a tight loop.

Comment 3 Avi Kivity 2006-02-26 11:26:53 UTC
same here, opteron (but running i686 kernel): 
 
... 
powernow-k8: MP systems not supported by PSB BIOS structure 
powernow-k8: MP systems not supported by PSB BIOS structure 
Using IPI No-Shortcut mode 
ACPI wakeup devices: 
HUB0 XVR0 XVR1 XVR2 XVR3 USB0 USB2 MMAC MMCI UAR1 
ACPI: (supports S0 S1 S3 S4 S5) 
Freeing unused kernel memory: 208k freed 
Write protecting the kernel read-only data: 368k 
 

Comment 4 Maarten Bremer 2006-02-26 19:25:03 UTC
I am seeing the same with a Asus RS 120 E3 webserver. The server stops loading
with the 1977 kernel after "Write protecting the kernel read-only data" while
the 1975 build works without any problems. I suspect it is a mouse or keyboard
issue, but the problem occurs without a mouse or keyboard connected too.

Comment 5 Dave Jones 2006-02-26 23:18:29 UTC
*** Bug 182978 has been marked as a duplicate of this bug. ***

Comment 6 Dave Jones 2006-02-26 23:19:54 UTC
*** Bug 183037 has been marked as a duplicate of this bug. ***

Comment 7 Dave Jones 2006-02-27 07:06:53 UTC
*** Bug 183024 has been marked as a duplicate of this bug. ***

Comment 8 Dave Jones 2006-02-27 07:08:07 UTC
*** Bug 183026 has been marked as a duplicate of this bug. ***

Comment 9 Dave Jones 2006-02-27 07:14:08 UTC
Should be fixed in the latest builds (Grab them early from
http://people.redhat.com/davej/kernels/Fedora/devel)

Comment 10 Paul Black 2006-02-27 08:57:06 UTC
1988 works for me.

Cheers


Comment 11 David Nielsen 2006-02-27 12:55:46 UTC
WORKSFORME, however the seperate mkinitrd issue is still alive

Comment 12 J. Rutten 2006-02-27 16:25:07 UTC
I have the same problem on my Abit VP6 Motherboard.

Comment 13 Need Real Name 2006-02-27 18:12:32 UTC
Created attachment 125334 [details]
new-kernel-pkg aborted

Tried 1990, but grubby stack traced with a double free when I tried to install.

stack attached in case it's yours, otherwise I'll open a new bug.

Comment 14 bkyoung 2006-02-27 18:28:11 UTC
1990 WORKSFORME, but had to workaround #13 by using --noscripts, and adjust
new-kernel-pkg some.

kernel-smp-2.6.15-1.1990_FC5
selinux-policy-targeted-2.2.21-7
gcc-4.1.0-0.29
glibc-2.3.90-38

[root@flood k1990]# /sbin/new-kernel-pkg -v --package kernel-smp --mkinitrd
--depmod --install 2.6.15-1.1990_FC5smp
initrdfile is /boot/initrd-2.6.15-1.1990_FC5smp.img
making it the default based on config
running depmod for 2.6.15-1.1990_FC5smp
creating initrd /boot/initrd-2.6.15-1.1990_FC5smp.img using 2.6.15-1.1990_FC5smp
found /boot/initrd-2.6.15-1.1990_FC5smp.img and using it with grubby
adding 2.6.15-1.1990_FC5smp to /boot/grub/grub.conf
*** glibc detected *** /sbin/grubby: double free or corruption (out): 0xbfb49130 ***
======= Backtrace: =========
/lib/libc.so.6[0xcdcde8]
/lib/libc.so.6(__libc_free+0x79)[0xce02ed]
/sbin/grubby[0x804fd12]
/sbin/grubby[0x804fe73]
/sbin/grubby[0x8050c9e]
/lib/libc.so.6(__libc_start_main+0xdc)[0xc8e7a4]
/sbin/grubby[0x804a001]
======= Memory map: ========
003ec000-00405000 r-xp 00000000 08:03 1572866    /lib/ld-2.3.90.so
00405000-00406000 r-xp 00018000 08:03 1572866    /lib/ld-2.3.90.so
00406000-00407000 rwxp 00019000 08:03 1572866    /lib/ld-2.3.90.so
0069e000-0069f000 r-xp 0069e000 00:00 0          [vdso]
00c79000-00d9c000 r-xp 00000000 08:03 1572882    /lib/libc-2.3.90.so
00d9c000-00d9f000 r-xp 00122000 08:03 1572882    /lib/libc-2.3.90.so
00d9f000-00da0000 rwxp 00125000 08:03 1572882    /lib/libc-2.3.90.so
00da0000-00da3000 rwxp 00da0000 00:00 0 
05dcb000-05dd6000 r-xp 00000000 08:03 1572897    /lib/libgcc_s-4.1.0-20060219.so.1
05dd6000-05dd7000 rwxp 0000a000 08:03 1572897    /lib/libgcc_s-4.1.0-20060219.so.1
08048000-08089000 r-xp 00000000 08:03 2982102    /sbin/grubby
08089000-0808c000 rw-p 00041000 08:03 2982102    /sbin/grubby
0808c000-08094000 rw-p 0808c000 00:00 0 
09d4b000-09d6c000 rw-p 09d4b000 00:00 0          [heap]
b7e00000-b7e21000 rw-p b7e00000 00:00 0 
b7e21000-b7f00000 ---p b7e21000 00:00 0 
b7f38000-b7f3a000 rw-p b7f38000 00:00 0 
bfb35000-bfb4a000 rw-p bfb35000 00:00 0          [stack]
/sbin/new-kernel-pkg: line 89:  3589 Aborted                 /sbin/grubby
--add-kernel=$bootPrefix/$kernelName-$version $INITRD --copy-default
$makedefault --title "$title" ${mbkernel:+--add-multiboot="$mbkernel"}
${mbargs:+--mbargs="$mbargs"} --args="root=$rootdevice $kernargs"
--remove-kernel="TITLE=$title"
/etc/lilo.conf does not exist, not running grubby
[root@flood k1990]# /sbin/new-kernel-pkg -v --package kernel-smp --mkinitrd
--depmod --install 2.6.15-1.1990_FC5smp
initrdfile is /boot/initrd-2.6.15-1.1990_FC5smp.img
making it the default based on config
running depmod for 2.6.15-1.1990_FC5smp
creating initrd /boot/initrd-2.6.15-1.1990_FC5smp.img using 2.6.15-1.1990_FC5smp
found /boot/initrd-2.6.15-1.1990_FC5smp.img and using it with grubby
adding 2.6.15-1.1990_FC5smp to /boot/grub/grub.conf
/etc/lilo.conf does not exist, not running grubby
[root@flood k1990]# exit


Comment 15 bkyoung 2006-02-27 21:55:15 UTC
Some patches for a #13 "work around of last resort" (ie. if nothing else works)
are attached. Use at your own risk!

BEGIN GRUBBY.C PATCH
*** grubby/grubby.c.orig	2006-02-15 11:42:28.000000000 -0800
--- grubby/grubby.c	2006-02-27 13:39:45.000000000 -0800
***************
*** 969,975 ****
          return 0;
  
      i = stat(dev, &sb);
!     free(dev);
      if (i)
  	return 0;
  
--- 969,975 ----
          return 0;
  
      i = stat(dev, &sb);
!     /*    free(dev); */
      if (i)
  	return 0;
END GRUBBY.C PATCH
BEGIN MAKEFILE PATCH
*** grubby/Makefile.orig	2006-02-22 17:16:25.000000000 -0800
--- grubby/Makefile	2006-02-27 13:15:35.000000000 -0800
***************
*** 18,24 ****
  	mkdir -p $(BUILDROOT)/$(mandir)/man8
  	install -m 755 new-kernel-pkg $(BUILDROOT)/sbin
  	if [ -f grubby ]; then \
! 		install -m 755 -s grubby $(BUILDROOT)/sbin ; \
  		install -m 644 grubby.8 $(BUILDROOT)/$(mandir)/man8 ; \
  	fi
  
--- 18,24 ----
  	mkdir -p $(BUILDROOT)/$(mandir)/man8
  	install -m 755 new-kernel-pkg $(BUILDROOT)/sbin
  	if [ -f grubby ]; then \
! 		install -m 755 grubby $(BUILDROOT)/sbin ; \
  		install -m 644 grubby.8 $(BUILDROOT)/$(mandir)/man8 ; \
  	fi
END MAKEFILE PATCH
BEGIN MKINITRD.SPEC PATCH
*** mkinitrd.spec.orig	2006-02-24 15:10:52.000000000 -0800
--- mkinitrd.spec	2006-02-27 13:41:36.000000000 -0800
***************
*** 1,10 ****
  Summary: Creates an initial ramdisk image for preloading modules.
  Name: mkinitrd
  Version: 5.0.28
! Release: 1
  License: GPL
  Group: System Environment/Base
  Source: mkinitrd-%{version}.tar.bz2
  ExclusiveOs: Linux
  Prereq: dev
  Requires: /bin/sh, /sbin/insmod.static, /sbin/losetup
--- 1,12 ----
  Summary: Creates an initial ramdisk image for preloading modules.
  Name: mkinitrd
  Version: 5.0.28
! Release: 1bky2
  License: GPL
  Group: System Environment/Base
  Source: mkinitrd-%{version}.tar.bz2
+ Patch0: %{name}-%{version}-grubby_Makefile.patch
+ Patch1: %{name}-%{version}-grubby_grubby.c.patch
  ExclusiveOs: Linux
  Prereq: dev
  Requires: /bin/sh, /sbin/insmod.static, /sbin/losetup
***************
*** 13,18 ****
--- 15,21 ----
  Requires: e2fsprogs
  BuildRequires: popt e2fsprogs-devel parted-devel
  BuildRequires: device-mapper libselinux-devel libsepol-devel
+ BuildRequires: pump-devel
  %ifnarch s390 s390x
  Requires: dmraid
  %endif
***************
*** 38,45 ****
--- 41,54 ----
  
  %prep
  %setup -q
+ %patch0 -p0 -b .orig
+ %patch1 -p0 -b .orig
  
  %build
+ CFLAGS="${CFLAGS:-%optflags}"
+ CFLAGS="${CFLAGS/-O2/-O0}"
+ #CFLAGS="${CFLAGS/-mtune=generic/-mtune=pentium2}"
+ export CFLAGS
  make
  make test
  if [ $? != 0 ]; then exit 1; fi
END MKINITRD.SPEC PATCH


Comment 16 Darwin H. Webb 2006-02-28 08:13:39 UTC
SMP 1990 is working on the Pent IV and Pent III
however I am getting a lot of kobject  messages (46 screens of dmesg);

kobject vcsa7: registering. parent: vc, set: class_obj
kobject_uevent
fill_kobj_path: path = '/class/vc/vcsa7'
kobject_uevent
fill_kobj_path: path = '/class/vc/vcs7'
kobject vcs7: cleaning up
kobject_uevent
fill_kobj_path: path = '/class/vc/vcsa7'
kobject vcsa7: cleaning up
kobject vcs7: registering. parent: vc, set: class_obj
kobject_uevent
fill_kobj_path: path = '/class/vc/vcs7'
kobject vcsa7: registering. parent: vc, set: class_obj
kobject_uevent
fill_kobj_path: path = '/class/vc/vcsa7'


Comment 17 Frank Arnold 2006-02-28 09:42:05 UTC
*** Bug 183285 has been marked as a duplicate of this bug. ***

Comment 18 Darwin H. Webb 2006-02-28 10:26:45 UTC
SMP 1991 with updates for Feb 28th are running good on both Pent IV and Pent III
and the messages are gone.

Darwin

Comment 19 bkyoung 2006-02-28 17:13:49 UTC
WORKSFORME (PII)
(I'll assume 1990 will uninstall properly on next kernel update)

kernel-smp-2.6.15-1.1990_FC5
kernel-smp-2.6.15-1.1991_FC5
mkinitrd-5.0.29-1

gnome-desktop-2.13.92-1
gnome-desktop-devel-2.13.92-1
gnome-doc-utils-0.5.7-1
gnome-keyring-0.4.8-1
gnome-keyring-devel-0.4.8-1
gnome-media-2.13.93-1
gnome-panel-2.13.91-4
gnome-panel-devel-2.13.91-4
gnome-screensaver-2.13.92-1
gnome-system-monitor-2.13.92-1
gnome-vfs2-2.13.92-1
gnome-vfs2-devel-2.13.92-1
gnome-vfs2-smb-2.13.92-1
libgnome-2.13.90-1
libgnome-devel-2.13.90-1
libgnomeui-2.13.90-1
libgnomeui-devel-2.13.90-1


Be Like That (3 Doors Down)


Comment 20 Brian Gaynor 2006-02-28 17:41:41 UTC
1991 fixes this for me.

Comment 21 David Bentley 2006-03-01 00:06:14 UTC
Both 1990 and 1991 work OK also tried the new mkinitrd from warren's post to the
test-list and that works OK too.

Comment 22 Need Real Name 2006-03-01 07:40:39 UTC
1991 wfm

Comment 23 Paul Black 2006-03-03 13:42:07 UTC
Reappeared in 2008


Comment 24 David Bentley 2006-03-03 16:01:05 UTC
Yes fails in exactly the same manner now with 2008.

Can the summary be changed to reflect this. 
(for people searhing for bugs with 2008)

Comment 25 Dax Kelson 2006-03-03 16:56:52 UTC
2008 fails in the same spot and manner for "me too".

Comment 26 Brian Gaynor 2006-03-03 17:24:23 UTC
Reappeared for me as well in 2008, hangs just after initializing Synaptics
touchpad, fan noise would indicate processor is in a tight loop, CTRL-ALT-DEL
still works (Dell Inspiron 5160).

Comment 27 David Fletcher 2006-03-03 21:34:25 UTC
I first had the problem with the 1955 kernel. The 1991 kernel works for me but 
after updating today the 2008 kernel has the same problem again.

This system does not get updated every day.


Comment 28 Erik Sjölund 2006-03-04 09:53:49 UTC
The 2.6.15-1.2008_FC5smp kernel hangs after 
"Write protecting the kernel read-only data: 368k"

but the 2.6.15-1.2009.4.2_FC5smp kernel boots just fine on my Dell Dimension 5100.

Comment 29 Dave Jones 2006-03-04 21:48:11 UTC
*** Bug 183926 has been marked as a duplicate of this bug. ***

Comment 30 Need Real Name 2006-03-05 11:40:26 UTC
2009 WFM, didn't see kernel 2008 so couldn't test.

Comment 31 David Bentley 2006-03-05 14:09:37 UTC
The latest development kernel 2.6.15-1.2009.4.2_FC5smp works OK.

Comment 32 Dave Jones 2006-03-05 18:27:06 UTC
It's disturbing that this bug 'comes and goes' in different builds when the
changes between working and broken builds can't explain this behaviour, so I'll
leave this open until I understand more about why this is happening.


Comment 33 Kurt Heine 2006-03-07 01:10:08 UTC
The initial problem with the hanging after detecting the mouse seems to be fixed
with at least kernel version 2.6.15-1.2009.4.2_FC5smp.  I have not tried any
other previous version or the no SMP kernel (UP).

Comment 34 Michal Jaegermann 2006-03-07 23:12:52 UTC
I had my share of troubles with 1977 and 2008 on an SMP machine (never on UP)
but 2.6.15-1.2025_FC5smp seems to be, in a sense, even nastier.  It will
stop (re)booting sometimes and other times there is no problem.  So far
I got one incident of a stuck machine after "Starting udev" and one
after "Restarting system" while trying to reboot.  The first on hit on the
first attempt to reboot with a new kernel.  The other happened after some
six, or so, tries.  I did not see such incidents with 2.6.15-1.2009.4.2_FC5smp.

Comment 35 Brian Gaynor 2006-03-08 17:53:29 UTC
2009, 2025 and today's 2032 have both worked fine for me (1977 and 2008 didn't).

Comment 36 bkyoung 2006-03-08 23:49:06 UTC
2032 works for me.

I noticed an X86-smp-alternatives.patch merged into 2.6.16-rc3. Depending on
when, if ever, it is adopted into the FC5 kernel, it may or may not resolve the
issue, or, it may or may not expand the issue. Although, according to the
.src.rpm kernel ChangeLog CONFIG_SMP_ALTERNATIVES was removed, so this may be
irrelevant.

After reviewing (kernel-2.6.15-1.2032.src.rpm)
BUILD/kernel-2.6.15/linux-2.6.15.i386/Documentation/cpu-hotplug.txt and noticing
that smp-i686 .config sets CONFIG_HOTPLUG_CPU,  adjusting boot parameter maxcpus
may affect some issues.



Comment 37 Michal Jaegermann 2006-03-09 00:54:34 UTC
I just tried 2.6.15-1.2032smp.  On nine tries I got stuck once after
"Starting udev" and four times after "Restarting system" when shutting down.
One dot is printed on the next line and that is it.  None of the later
happened after a power switch, or hardware reset, were used.  I already
thought that this is consistent with a warm reboot but once it went through.
Still chances that reboot will fail look pretty high.

Comment 38 Dave Jones 2006-03-10 20:54:48 UTC
I just nailed this.  It's the same reason x86-64 doesn't boot in todays build.
The reasoning behind this bug is a real horror show.
In short: We ended up with no version string.


Comment 39 Dave Jones 2006-03-10 21:20:32 UTC
*** Bug 185076 has been marked as a duplicate of this bug. ***

Comment 40 Dave Jones 2006-03-10 23:18:40 UTC
*** Bug 185094 has been marked as a duplicate of this bug. ***

Comment 41 Need Real Name 2006-03-11 00:07:26 UTC
Come on! Give us the unabridged version!

Comment 42 Michal Jaegermann 2006-03-11 20:08:50 UTC
For what is worth, as this seems to be not necessarily the same issue,
with 2.6.15-1.2041_FC5smp I do not see troubles described in comment #34
and comment #37.

Comment 43 Michal Jaegermann 2006-03-11 20:18:40 UTC
Ugh! About comment #42 I recalled:

* Fri Mar 10 2006 Dave Jones <davej>
- Turn slab debug back on for one more build

Maybe this makes that issue disappear?  Things are probably slower.

Comment 44 Need Real Name 2006-03-11 20:38:24 UTC
(In reply to comment #41)
> Come on! Give us the unabridged version!

http://kernelslacker.livejournal.com/36756.html

Comment 45 Dave Jones 2006-03-11 22:34:09 UTC
*** Bug 185191 has been marked as a duplicate of this bug. ***