Bug 433583 - Saslauthd crashes randomly
Saslauthd crashes randomly
Status: CLOSED ERRATA
Product: Red Hat Enterprise Linux 5
Classification: Red Hat
Component: cyrus-sasl (Show other bugs)
5.1
x86_64 Linux
medium Severity high
: rc
: ---
Assigned To: Tomas Mraz
Brian Brock
: Security
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2008-02-20 00:14 EST by Devrim GUNDUZ
Modified: 2014-05-26 21:28 EDT (History)
6 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2009-09-02 06:12:35 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Devrim GUNDUZ 2008-02-20 00:14:06 EST
Description of problem:
saslauthd is crashing randomly on a RHEL 5.1 box.

Version-Release number of selected component (if applicable):
2.1.22-4

How reproducible:
Rarely

Additional info:

This is from last week's logs:

Feb 12 16:52:07 namsb kernel: saslauthd[10937]: segfault at ffffffffe4033160 rip
0000003fe1276170 rsp 00007fff228e2b28 error 4
Feb 12 16:52:07 namsb kernel: saslauthd[10939]: segfault at ffffffffe4033160 rip
0000003fe1276170 rsp 00007fff228e2b28 error 4
Feb 12 16:52:21 namsb kernel: saslauthd[10940]: segfault at ffffffffe4033160 rip
0000003fe1276170 rsp 00007fff228e2b28 error 4
Feb 12 16:52:21 namsb kernel: saslauthd[10938]: segfault at ffffffffe4033160 rip
0000003fe1276170 rsp 00007fff228e2b28 error 4
Feb 14 09:39:31 namsb kernel: saslauthd[10936]: segfault at ffffffffe4033160 rip
0000003fe1276170 rsp 00007fff228e2b28 error 4

This is a e-mail server running Postfix.
Comment 2 Devrim GUNDUZ 2008-06-26 17:10:26 EDT
Now I got a core file. This is RHEL 5.2 :

# gdb /usr/sbin/saslauthd core.25441
GNU gdb Red Hat Linux (6.5-37.el5_2.2rh)
Copyright (C) 2006 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu"...Using host libthread_db
library "/lib64/libthread_db.so.1".


warning: .dynamic section for "/usr/lib64/libgssapi_krb5.so.2" is not at the
expected address

warning: difference appears to be caused by prelink, adjusting expectations

warning: .dynamic section for "/usr/lib64/libkrb5.so.3" is not at the expected
address

warning: .dynamic section for "/usr/lib64/libk5crypto.so.3" is not at the
expected address

warning: .dynamic section for "/lib64/libcom_err.so.2" is not at the expected
address

warning: difference appears to be caused by prelink, adjusting expectations

warning: .dynamic section for "/usr/lib64/libkrb5support.so.0" is not at the
expected address

warning: .dynamic section for "/lib64/libresolv.so.2" is not at the expected address

warning: difference appears to be caused by prelink, adjusting expectations

warning: .dynamic section for "/lib64/libcrypt.so.1" is not at the expected address

warning: .dynamic section for "/lib64/libpam.so.0" is not at the expected address

warning: .dynamic section for "/usr/lib64/libldap-2.3.so.0" is not at the
expected address

warning: difference appears to be caused by prelink, adjusting expectations

warning: .dynamic section for "/usr/lib64/liblber-2.3.so.0" is not at the
expected address

warning: difference appears to be caused by prelink, adjusting expectations

warning: .dynamic section for "/lib64/libcrypto.so.6" is not at the expected address

warning: difference appears to be caused by prelink, adjusting expectations

warning: .dynamic section for "/lib64/libc.so.6" is not at the expected address

warning: .dynamic section for "/lib64/libdl.so.2" is not at the expected address

warning: difference appears to be caused by prelink, adjusting expectations

warning: .dynamic section for "/lib64/libkeyutils.so.1" is not at the expected
address

warning: difference appears to be caused by prelink, adjusting expectations

warning: .dynamic section for "/lib64/libselinux.so.1" is not at the expected
address

warning: difference appears to be caused by prelink, adjusting expectations

warning: .dynamic section for "/lib64/libaudit.so.0" is not at the expected address

warning: .dynamic section for "/usr/lib64/libsasl2.so.2" is not at the expected
address

warning: difference appears to be caused by prelink, adjusting expectations

warning: .dynamic section for "/lib64/libssl.so.6" is not at the expected address

warning: difference appears to be caused by prelink, adjusting expectations

warning: .dynamic section for "/usr/lib64/libz.so.1" is not at the expected address

warning: difference appears to be caused by prelink, adjusting expectations

warning: .dynamic section for "/lib64/ld-linux-x86-64.so.2" is not at the
expected address

warning: .dynamic section for "/lib64/libsepol.so.1" is not at the expected address

warning: difference appears to be caused by prelink, adjusting expectations
Reading symbols from /usr/lib64/libgssapi_krb5.so.2...done.
Loaded symbols for /usr/lib64/libgssapi_krb5.so.2
Reading symbols from /usr/lib64/libkrb5.so.3...done.
Loaded symbols for /usr/lib64/libkrb5.so.3
Reading symbols from /usr/lib64/libk5crypto.so.3...done.
Loaded symbols for /usr/lib64/libk5crypto.so.3
Reading symbols from /lib64/libcom_err.so.2...done.
Loaded symbols for /lib64/libcom_err.so.2
Reading symbols from /usr/lib64/libkrb5support.so.0...done.
Loaded symbols for /usr/lib64/libkrb5support.so.0
Reading symbols from /lib64/libresolv.so.2...done.
Loaded symbols for /lib64/libresolv.so.2
Reading symbols from /lib64/libcrypt.so.1...done.
Loaded symbols for /lib64/libcrypt.so.1
Reading symbols from /lib64/libpam.so.0...done.
Loaded symbols for /lib64/libpam.so.0
Reading symbols from /usr/lib64/libldap-2.3.so.0...done.
Loaded symbols for /usr/lib64/libldap-2.3.so.0
Reading symbols from /usr/lib64/liblber-2.3.so.0...done.
Loaded symbols for /usr/lib64/liblber-2.3.so.0
Reading symbols from /lib64/libcrypto.so.6...done.
Loaded symbols for /lib64/libcrypto.so.6
Reading symbols from /lib64/libc.so.6...done.
Loaded symbols for /lib64/libc.so.6
Reading symbols from /lib64/libdl.so.2...done.
Loaded symbols for /lib64/libdl.so.2
Reading symbols from /lib64/libkeyutils.so.1...done.
Loaded symbols for /lib64/libkeyutils.so.1
Reading symbols from /lib64/libselinux.so.1...done.
Loaded symbols for /lib64/libselinux.so.1
Reading symbols from /lib64/libaudit.so.0...done.
Loaded symbols for /lib64/libaudit.so.0
Reading symbols from /usr/lib64/libsasl2.so.2...Reading symbols from
/usr/lib/debug/usr/lib64/libsasl2.so.2.0.22.debug...done.
done.
Loaded symbols for /usr/lib64/libsasl2.so.2
Reading symbols from /lib64/libssl.so.6...done.
Loaded symbols for /lib64/libssl.so.6
Reading symbols from /usr/lib64/libz.so.1...done.
Loaded symbols for /usr/lib64/libz.so.1
Reading symbols from /lib64/ld-linux-x86-64.so.2...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Reading symbols from /lib64/libsepol.so.1...done.
Loaded symbols for /lib64/libsepol.so.1
Reading symbols from /lib64/libnss_files.so.2...done.
Loaded symbols for /lib64/libnss_files.so.2
Core was generated by `/usr/sbin/saslauthd -m /var/run/saslauthd -a shadow'.
Program terminated with signal 11, Segmentation fault.
#0  0x0000003fe1276170 in ?? ()
Comment 3 Devrim GUNDUZ 2008-06-26 17:11:22 EDT
I'm also increasing severity -- problem occured twice in last 24 hours.
Comment 5 David Lawrence 2008-06-26 17:31:12 EDT
Comment 4 removed.
Comment 6 Devrim GUNDUZ 2008-06-26 17:55:14 EDT
(gdb) bt full
#0  0x0000003fe1276170 in ?? ()
No symbol table info available.
#1  0x0000003fe1275ea6 in ?? ()
No symbol table info available.
#2  0x00007fff06bea3e0 in ?? ()
No symbol table info available.
#3  0x0000003fe154cf60 in ?? ()
No symbol table info available.
#4  0x00007fff06bea3e0 in ?? ()
No symbol table info available.
#5  0x0000000000406635 in auth_shadow (login=0xffffffffe4033160 <Address
0xffffffffe4033160 out of bounds>, password=0x7fff06bea2d0 "PASSWORD-BLOCKED", 
    service=<value optimized out>, realm=<value optimized out>) at
saslauthd/auth_shadow.c:183
	today = <value optimized out>
	cpw = <value optimized out>
	pw = <value optimized out>
	sp = (struct spwd *) 0x3fe154cf60
#6  0x000000000040e1dd in do_auth (_login=<value optimized out>, password=0x6f
<Address 0x6f out of bounds>, service=0x7fff06bea1c0 "smtp", 
    realm=0x7fff06bea0b0 "HOSTNAME-BLOCKED") at saslauthd/saslauthd-main.c:399
	lkup_result = {bucket = {creds = '\0' <repeats 59 times>, user_offt = 0,
realm_offt = 0, service_offt = 0, pwd_digest = '\0' <repeats 15 times>, 
    created = 0}, read_bucket = 0x0, hash_offset = 0, status = 0}
	response = <value optimized out>
	cached = 0
	login_buff = '\0' <repeats 512 times>
	login = 0x7fff06bea3e0 "news"
	__func__ = "do_auth"
#7  0x000000000040c3fe in do_request (conn_fd=7) at saslauthd/ipc_unix.c:426
	count = 18
	ncount = <value optimized out>
	response = <value optimized out>
	login =
"news\000ting\000tor\000\000\000��T�?\000\000\000#W)�?\000\000\000��T�?\000\000\000`�T�?\000\000\000\005\000\000\000\000\000\000\000`Aa\000\000\000\000\000�>a\000\000\000\000\000`�T�?\000\000\000\030\000\000\000\000\000\000\000
��\006�\177\000\000P��\006�\177", '\0' <repeats 19 times>, "\235t\000\000 L\202
��\006�\177\000\000`Aa\000\000\000\000\000���\006�\177\000\000�H:\035", '\0'
<repeats 12 times>, "\001", '\0' <repeats 15 times>,
"\023�&�\000\000\000\000���\006�\177\000\000\000\000\000\000\000\000\000\00025437\n\000\000"...
	password = "PASSWORD-BLOCKED", '\0' <repeats 13 times>,
"`�T�?\000\000\000����\000\000\000\000\220u��?\000\000\000.\000\000\000\000\000\000\000`\224T�?",
'\0' <repeats 11 times>,
"\034+'�?\000\000\000.\000\000\000\000\000\000\000\220��\006�\177\000\000\002\037��?\000\000\000M\r'�?\000\000\000/\000\000\000\000\000\000\000/\000\000\000\000\000\000\000�$:\035\000\000\000\0001p&�?\000\000\000\000\200��x\000\000\000���\006�\177\000\000\005\000\000\000\000\000\000\000`Aa\000\000\000\000\000�>a",
'\0' <repeats 22 times>, "\235t\000\000 L\2020\000\000\0000\000\000\000"...
	service =
"smtp\000*\000\000\000\000\000\000\000\000\000\000\001\000\000\000\000\000\000\0000��\006�\177\000\000\000\000\000\000\000\000\000\000\023��\006�\177\000\000���\006�\177\000\000���\006�\177\000\000\001\000\000\000\000\000\000\000���\006�\177\000\000q\027A\000\000\000\000\000c\000\000\000\000\000\000\0000��\006�\177\000\000(\030.�?\000\000\000\001\200��\000\000\000\000���\006�\177\000\000���\006�\177\000\000���\006�\177\000\000���\006�\177\000\000���\006�\177\000\000\023��\006�\177\000\000���\006�\177\000\000\023��\006�\177",
'\0' <repeats 42 times>, "/\000\000\000\004\000\000\000"...
	realm = "HOSTNAME-BLOCKED\000\006�\177\000\0006���?", '\0' <repeats 11 times>,
"\001\000\000\000\000\000\000\000С�25437\001\000\000\000\000\000\000\000H\034`�?\000\000\000
�\203�?", '\0' <repeats 11 times>,
"�/`�?\000\000\000(\000\000\0000\000\000\000\200��\006�\177\000\000���\006�\177\000\000\000����*\000\000�\235\001�\026\000\000\000�",
'\0' <repeats 15 times>,
"�\215�\006�\177\000\000\002\000\000\000\000\000\000\000�N���*\000\000\000\000\000\000\000\000\000\000\001\000\000\000\000\000\000\000���\006�\177\000\000�\023��?",
'\0' <repeats 11 times>...
	__func__ = "do_request"
#8  0x000000000040c71e in ipc_loop () at saslauthd/ipc_unix.c:277
	rc = 0
	conn_fd = 7
	__func__ = "ipc_loop"
#9  0x000000000040de3a in main (argc=<value optimized out>, argv=<value
optimized out>) at saslauthd/saslauthd-main.c:358
	option = <value optimized out>
	rc = <value optimized out>
	x = 4
	lockinfo = {l_type = 1, l_whence = 0, l_start = 0, l_len = 0, l_pid = -519988288}
---Type <return> to continue, or q <return> to quit---
	auth_mech_name = 0x1d3a23b0 "shadow"
	pid_file_size = 4
	__func__ = "main"
#10 0x0000003fe121d8a4 in ?? ()
No symbol table info available.
#11 0x00007fff06bea698 in ?? ()
No symbol table info available.
#12 0x00007fff06bea698 in ?? ()
No symbol table info available.
#13 0x00000005aaaab000 in ?? ()
No symbol table info available.
#14 0x000000000040da80 in show_usage () at saslauthd/saslauthd-main.c:994
	pid_fd = 4
	pid_file_lock = 0x1d3a24d0 "/var/run/saslauthd/saslauthd.pid"
	master_pid = 25437
	startup_pipe = {4, 5}
	pid_file_lock_fd = 3
	pid_file = 0x1d3a24d0 "/var/run/saslauthd/saslauthd.pid"
	num_procs = 5
	run_path = 0x1d3a2460 "/var/run/saslauthd"
	auth_mech = (authmech_t *) 0x613de0
	g_argv = (char **) 0x7fff06bea698
	g_argc = 5
	flags = 356
	mech_option = 0x0
#15 0x0000003fe1019bc0 in ?? ()
No symbol table info available.
#16 0x826a7d0c73ece9aa in ?? ()
No symbol table info available.
#17 0x0000000000000000 in ?? ()
No symbol table info available.
Comment 7 Alvaro Herrera 2008-06-26 20:46:36 EDT
This seems to point to this code in saslauthd/auth_shadow.c:

    /*
     * Note: no check for SHADOW_PW_LOCKED. Returning a "locked" notification
     * would allow login-id namespace probes, and violates our policy of
     * not returning any information about a login until we have validated
     * the password.
     */
    cpw = strdup((const char *)crypt(password, sp->sp_pwdp));
    if (strcmp(sp->sp_pwdp, cpw)) {
        if (flags & VERBOSE) {
            syslog(LOG_DEBUG, "DEBUG: auth_shadow: pw mismatch: '%s' != '%s'",
                   sp->sp_pwdp, cpw);
        }
        free(cpw);
        RETURN("NO");
    }
    free(cpw);


Note that the crypt() function can return a NULL on errors, but this is not
checked.  strdup on a NULL pointer would cause a segmentation fault which could
be what we're seeing here.

In any case, I'm not seeing the point of strdup() here at all.  Perhaps the call
can just be removed.  (In the case the strdup() is there because of a possible
problem with multithreading creating a race condition to the access of the
static pointer returned by crypt(), then it would seem that the solution
involves locking a mutex before calling crypt() and only releasing it after
strdup()).
Comment 8 Tomas Mraz 2008-07-01 16:37:05 EDT
I don't think crypt can fail in normal circumstances. It is just computing the
hash and it is not allocating anything. Even the man page says the only errno
for crypt is "ENOSYS The  crypt()  function  was not implemented, probably
because of U.S.A. export restrictions." Also the saslauthd is not multithreaded
so a race should not happen here either. Can you please install also
glibc-debuginfo package and show the backtrace with it?
Comment 9 Devrim GUNDUZ 2008-07-14 13:01:21 EDT
Ok, installed glibc-debuginfo (sorry for the delay, I was on a holiday).
Comment 10 Alvaro Herrera 2008-09-09 17:54:30 EDT
Okay, so here's the new backtrace:

(gdb) bt
#0  0x000000326c0782d0 in strlen () from /lib64/libc.so.6
#1  0x000000326c078006 in *__GI___strdup (s=0x6e2371a0 <Address 0x6e2371a0 out of bounds>)
    at strdup.c:42
#2  0x0000000000406635 in auth_shadow (login=0x7fff524e1000 "mail", 
    password=0x7fff524e0ef0 "123456", service=<value optimized out>, 
    realm=<value optimized out>) at saslauthd/auth_shadow.c:183
#3  0x000000000040e1dd in do_auth (_login=<value optimized out>, 
    password=0x78 <Address 0x78 out of bounds>, service=0x7fff524e0de0 "smtp", 
    realm=0x7fff524e0cd0 "REMOVED") at saslauthd/saslauthd-main.c:399
#4  0x000000000040c3fe in do_request (conn_fd=7) at saslauthd/ipc_unix.c:426
#5  0x000000000040c71e in ipc_loop () at saslauthd/ipc_unix.c:277
#6  0x000000000040de3a in main (argc=<value optimized out>, argv=<value optimized out>)
    at saslauthd/saslauthd-main.c:358
#7  0x000000326c01d8b4 in __libc_start_main (main=0x40da80 <main>, argc=5, 
    ubp_av=0x7fff524e12b8, init=<value optimized out>, fini=<value optimized out>, 
    rtld_fini=<value optimized out>, stack_end=0x7fff524e12a8) at libc-start.c:231
#8  0x00000000004035f9 in _start ()
Comment 11 Alvaro Herrera 2008-09-09 17:56:46 EDT
bt full:

#0  0x000000326c0782d0 in strlen () from /lib64/libc.so.6
        mallstream = (FILE *) 0x0
        tr_old_memalign_hook = (void *(*)(size_t, size_t, const void *)) 0
        tr_old_malloc_hook = (void *(*)(size_t, const void *)) 0
        tr_old_realloc_hook = (void *(*)(void *, size_t, const void *)) 0
        lock = 0
        mallenv = "MALLOC_TRACE"
        malloc_trace_buffer = 0x0
        tr_old_free_hook = (void (*)(void *, const void *)) 0
        mallwatch = (void *) 0x0
#1  0x000000326c078006 in *__GI___strdup (s=0x6e2371a0 <Address 0x6e2371a0 out of bounds>)
    at strdup.c:42
        len = <value optimized out>
        new = <value optimized out>
#2  0x0000000000406635 in auth_shadow (login=0x7fff524e1000 "mail", 
    password=0x7fff524e0ef0 "123456", service=<value optimized out>, 
    realm=<value optimized out>) at saslauthd/auth_shadow.c:183
        today = <value optimized out>
        cpw = <value optimized out>
        pw = <value optimized out>
        sp = (struct spwd *) 0x326c34ffc0
#3  0x000000000040e1dd in do_auth (_login=<value optimized out>, 
    password=0x78 <Address 0x78 out of bounds>, service=0x7fff524e0de0 "smtp", 
    realm=0x7fff524e0cd0 "REMOVED") at saslauthd/saslauthd-main.c:399
        lkup_result = {bucket = {creds = '\0' <repeats 59 times>, user_offt = 0, 
    realm_offt = 0, service_offt = 0, pwd_digest = '\0' <repeats 15 times>, created = 0}, 
  read_bucket = 0x0, hash_offset = 0, status = 0}
        response = <value optimized out>
        cached = 0
        login_buff = '\0' <repeats 512 times>
        login = 0x7fff524e1000 "mail"
        __func__ = "do_auth"
Comment 12 Tomas Mraz 2008-09-10 03:28:53 EDT
This is caused by missing prototype for crypt() in auth_shadow.c.

As this is DoS although not triggerable on-demand I am adding Security keyword.
Comment 14 marcf 2009-04-13 12:11:48 EDT
This issue appears to have been fixed upstream:

http://archives.free.net.ph/message/20090220.215950.01b05224.en.html
https://bugzilla.andrew.cmu.edu/cgi-bin/cvsweb.cgi/src/sasl/saslauthd/auth_shadow.c.diff?r1=1.9&r2=1.10

Any chance of getting it included in a future security update?
Comment 15 Tomas Mraz 2009-04-14 03:19:39 EDT
Yes, this is scheduled for RHEL 5.4 cyrus-sasl errata.
Comment 16 Devrim GUNDUZ 2009-04-14 04:53:31 EDT
I'm glad that you did not defer it to RHEL 6. (!)

Current version is crashing servers, and some people treat this as a security issue. Why don't you push it to RHEL 5.3 immediately?

Sincerely, Devrim
Comment 18 Chris Ward 2009-07-21 09:27:01 EDT
Good afternoon,

I would like to ask you to please grab the updated version of cyrus-sasl for testing. Your feedback on this issue would be greatly appreciated. The following package builds should resolve all the issues reported in this bug.

http://people.redhat.com/~cward/5.4.0/cyrus-sasl/

Thanks!
Comment 22 Darrin Mison 2009-08-20 00:54:33 EDT
Can I have confirmation that this bug *only* affects 64-bit ?
Comment 23 Tomas Mraz 2009-08-20 04:03:52 EDT
Yes, on 32bit the pointers have the same length as int.
Comment 24 Alvaro Herrera 2009-08-20 11:03:32 EDT
I no longer had access to the customer machine that was showing the problem, back when the bug was fixed, so I cannot confirm that it solves the problem nor deny it.  Sorry.
Comment 25 errata-xmlrpc 2009-09-02 06:12:35 EDT
An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on therefore solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHBA-2009-1330.html

Note You need to log in before you can comment on or make changes to this bug.