Bug 487618

Summary:	Fedora 10 s2c Crash Prevents Completed Installation
Product:	[Community] Spacewalk	Reporter:	Devan Goodwin <dgoodwin>
Component:	Server	Assignee:	Pradeep Kilambi <pkilambi>
Status:	CLOSED CURRENTRELEASE	QA Contact:	Red Hat Satellite QA List <satqe-list>
Severity:	medium	Docs Contact:
Priority:	low
Version:	0.5	CC:	jesusr
Target Milestone:	---
Target Release:	---
Hardware:	All
OS:	Linux
Whiteboard:
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2009-09-17 07:10:40 UTC	Type:	---
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	456553

Description Devan Goodwin 2009-02-27 02:27:49 UTC

Description of problem:

When installing Spacewalk on F10, final restart of services hangs, rhn-installation.log seems to indicate the problem is an s2c crash.


How reproducible:

Suspect 100%, replicated on i386 physical host and i386 KVM guest.

Steps to Reproduce:
1. Install Spacewalk on Fedora 10.
  
Actual results:

Installation hangs during "Restarting services". Stack trace printed in /var/log/rhn-installation.log:

Starting rhn-search...
Starting cobbler daemon: [  OK  ]

Starting RHN Taskomatic...
Done.
*** glibc detected *** /usr/bin/s2s: corrupted double-linked list: 0x091b01f8 ***
======= Backtrace: =========
/lib/libc.so.6[0x884da9]
/lib/libc.so.6[0x885f77]
/lib/libc.so.6(cfree+0x96)[0x886356]
/usr/bin/s2s[0x8064621]
/usr/bin/s2s[0x8058c82]
/usr/bin/s2s[0x804ee19]
/lib/libc.so.6(__libc_start_main+0xe5)[0x82b6e5]
/usr/bin/s2s[0x804a9f1]
======= Memory map: ========
00101000-0010c000 r-xp 00000000 08:02 3629       /usr/lib/libudns.so.0
0010c000-0010d000 rw-p 0000b000 08:02 3629       /usr/lib/libudns.so.0
0010f000-00123000 r-xp 00000000 08:02 3625       /usr/lib/libgsasl.so.7.6.0
00123000-00124000 rw-p 00014000 08:02 3625       /usr/lib/libgsasl.so.7.6.0
00126000-0012d000 r-xp 00000000 08:02 3415       /usr/lib/libntlm.so.0.0.15
0012d000-0012e000 rw-p 00006000 08:02 3415       /usr/lib/libntlm.so.0.0.15
00130000-00150000 r-xp 00000000 08:02 3676       /usr/lib/libgc.so.1.0.3
00150000-00152000 rw-p 0001f000 08:02 3676       /usr/lib/libgc.so.1.0.3
00152000-001be000 rw-p 00152000 00:00 0 
001f1000-001fc000 r-xp 00000000 08:02 13201      /lib/libpam.so.0.81.12
001fc000-001fd000 rw-p 0000a000 08:02 13201      /lib/libpam.so.0.81.12
00557000-00558000 r-xp 00557000 00:00 0          [vdso]
007f0000-00810000 r-xp 00000000 08:02 319556     /lib/ld-2.9.so
00811000-00812000 r--p 00020000 08:02 319556     /lib/ld-2.9.so
00812000-00813000 rw-p 00021000 08:02 319556     /lib/ld-2.9.so
00815000-00983000 r-xp 00000000 08:02 319557     /lib/libc-2.9.so
00983000-00985000 r--p 0016e000 08:02 319557     /lib/libc-2.9.so
00985000-00986000 rw-p 00170000 08:02 319557     /lib/libc-2.9.so
00986000-00989000 rw-p 00986000 00:00 0 
0098b000-0098e000 r-xp 00000000 08:02 319564     /lib/libdl-2.9.so
0098e000-0098f000 r--p 00002000 08:02 319564     /lib/libdl-2.9.so
0098f000-00990000 rw-p 00003000 08:02 319564     /lib/libdl-2.9.so
00992000-009a8000 r-xp 00000000 08:02 319560     /lib/libpthread-2.9.so
009a8000-009a9000 r--p 00015000 08:02 319560     /lib/libpthread-2.9.so
009a9000-009aa000 rw-p 00016000 08:02 319560     /lib/libpthread-2.9.so
009aa000-009ac000 rw-p 009aa000 00:00 0 
009d9000-009f3000 r-xp 00000000 08:02 319566     /lib/libselinux.so.1
009f3000-009f4000 r--p 00019000 08:02 319566     /lib/libselinux.so.1
009f4000-009f5000 rw-p 0001a000 08:02 319566     /lib/libselinux.so.1
00adc000-00aef000 r-xp 00000000 08:02 319568     /lib/libz.so.1.2.3
00aef000-00af0000 rw-p 00012000 08:02 319568     /lib/libz.so.1.2.3
00c0f000-00c11000 r-xp 00000000 08:02 321611     /lib/libcom_err.so.2.1
00c11000-00c12000 rw-p 00001000 08:02 321611     /lib/libcom_err.so.2.1
00cf5000-00d1a000 r-xp 00000000 08:02 319569     /lib/libexpat.so.1.5.2
00d1a000-00d1c000 rw-p 00024000 08:02 319569     /lib/libexpat.so.1.5.2
026ef000-02702000 r-xp 00000000 08:02 319577     /lib/libresolv-2.9.so
02702000-02703000 r--p 00012000 08:02 319577     /lib/libresolv-2.9.so
02703000-02704000 rw-p 00013000 08:02 319577     /lib/libresolv-2.9.so
02704000-02706000 rw-p 02704000 00:00 0 
02708000-02739000 r-xp 00000000 08:02 14400      /lib/libidn.so.11.5.28
02739000-0273a000 rw-p 00030000 08:02 14400      /lib/libidn.so.11.5.28
02866000-02868000 r-xp 00000000 08:02 319576     /lib/libkeyutils-1.2.so
02868000-02869000 rw-p 00001000 08:02 319576     /lib/libkeyutils-1.2.so
0286b000-02898000 r-xp 00000000 08:02 1933       /usr/lib/libgssapi_krb5.so.2.2
02898000-0289a000 rw-p 0002d000 08:02 1933       /usr/lib/libgssapi_krb5.so.2.2
0289c000-028c0000 r-xp 00000000 08:02 1931       /usr/lib/libk5crypto.so.3.1
028c0000-028c1000 rw-p 00024000 08:02 1931       /usr/lib/libk5crypto.so.3.1
028c3000-0295f000 r-xp 00000000 08:02 1932       /usr/lib/libkrb5.so.3.3
0295f000-02962000 rw-p 0009b000 08:02 1932       /usr/lib/libkrb5.so.3.3
02964000-029ab000 r-xp 00000000 08:02 319580     /lib/libssl.so.0.9.8g
029ab000-029af000 rw-p 00046000 08:02 319580     /lib/libssl.so.0.9.8g
029b1000-02ae8000 r-xp 00000000 08:02 319579     /lib/libcrypto.so.0.9.8g
02ae8000-02afc000 rw-p 00136000 08:02 319579     /lib/libcrypto.so.0.9.8g
02afc000-02aff000 rw-p 02afc000 00:00 0 
02b5d000-02b66000 r-xp 00000000 08:02 1930       /usr/lib/libkrb5support.so.0.1
02b66000-02b67000 rw-p 00008000 08:02 1930       /usr/lib/libkrb5support.so.0.1
02bd8000-02be5000 r-xp 00000000 08:02 319584     /lib/libgcc_s-4.3.2-20081105.so.1
02be5000-02be6000 rw-p 0000c000 08:02 319584     /lib/libgcc_s-4.3.2-20081105.so.1
02c35000-02c4d000 r-xp 00000000 08:02 319585     /lib/libaudit.so.0.0.0
02c4d000-02c4e000 r--p 00018000 08:02 319585     /lib/libaudit.so.0.0.0
02c4e000-02c4f000 rw-p 00019000 08:02 319585     /lib/libaudit.so.0.0.0
08048000-0806f000 r-xp 00000000 08:02 3684       /usr/bin/s2s
0806f000-08071000 rw-p 00027000 08:02 3684       /usr/bin/s2s
0918c000-091cc000 rw-p 0918c000 00:00 0          [heap]
b7e00000-b7e21000 rw-p b7e00000 00:00 0 
b7e21000-b7f00000 ---p b7e21000 00:00 0 
b7f30000-b7f37000 rw-p b7f30000 00:00 0 
bfa33000-bfa48000 rw-p bffeb000 00:00 0          [stack]

Expected results:

No stack trace.

Additional info:

Problem is not just during installation, it will happen even if you Ctrl+C and 
"rhn-satellite restart". Suspect it isn't causing any critical problems just breaks osad.

Considering the possibility it's a bug in the actual jabberd package in F10 but not sure if this indicates that's not the case:

(root@elaine /var/log/rhn) $ service jabberd restart
Terminating jabberd processes ...
Stopping router:                                           [  OK  ]
Stopping sm:                                               [FAILED]
Stopping c2s:                                              [FAILED]
Stopping s2s:                                              [FAILED]
Initializing jabberd processes ...
Starting router:                                           [  OK  ]
Starting sm:                                               [  OK  ]
Starting c2s:                                              [  OK  ]
Starting s2s:                                              [  OK  ]
(root@elaine /var/log/rhn) $ service jabberd restart
Terminating jabberd processes ...
Stopping router:                                           [  OK  ]
Stopping sm:                                               [  OK  ]
Stopping c2s:                                              [  OK  ]
Stopping s2s:                                              [  OK  ]
Initializing jabberd processes ...
Starting router:                                           [  OK  ]
Starting sm:                                               [  OK  ]
Starting c2s:                                              [  OK  ]
Starting s2s:                                              [  OK  ]

Need someone who knows more about jabberd and how we use it to determine if in fact this isn't even our bug and we should file against F10 jabberd pkg.

Comment 2 Devan Goodwin 2009-03-10 23:16:21 UTC

No idea what to make of this but I noticed that jabberd was writing an authentication error to /var/log/messages before it crashed. I did some comparisons on /etc/jabberd in F10 vs CentOS 5 and noticed that the router-users.xml file has a hashed password in F10, but was always just "secret" in other versions. I tried changing this on a whim and the crash ceases to happen.

Not sure why/how the authentication works but this is a workaround.

Authentication failure leading to crash sounds like a jabberd bug in and of itself.

Comment 3 Devan Goodwin 2009-03-11 01:15:17 UTC

Installing jabberd on a non-Spacewalk system and modifying the default pw in router-users.xml causes a segfault as well. Filed a separate bug against jabberd in: https://bugzilla.redhat.com/show_bug.cgi?id=489632

I guess we just need to identify how we get our authentication credentials straight for Spacewalk?

Comment 4 Devan Goodwin 2009-03-13 18:41:53 UTC

*** Bug 489998 has been marked as a duplicate of this bug. ***

Comment 6 Jesus M. Rodriguez 2009-04-14 14:13:16 UTC

Spacewalk 0.5 released.

Comment 7 Miroslav Suchý 2009-09-17 07:10:40 UTC

Spacewalk 0.5 has been released for long time ago.