Description of problem: saslauthd is crashing randomly on a RHEL 5.1 box. Version-Release number of selected component (if applicable): 2.1.22-4 How reproducible: Rarely Additional info: This is from last week's logs: Feb 12 16:52:07 namsb kernel: saslauthd[10937]: segfault at ffffffffe4033160 rip 0000003fe1276170 rsp 00007fff228e2b28 error 4 Feb 12 16:52:07 namsb kernel: saslauthd[10939]: segfault at ffffffffe4033160 rip 0000003fe1276170 rsp 00007fff228e2b28 error 4 Feb 12 16:52:21 namsb kernel: saslauthd[10940]: segfault at ffffffffe4033160 rip 0000003fe1276170 rsp 00007fff228e2b28 error 4 Feb 12 16:52:21 namsb kernel: saslauthd[10938]: segfault at ffffffffe4033160 rip 0000003fe1276170 rsp 00007fff228e2b28 error 4 Feb 14 09:39:31 namsb kernel: saslauthd[10936]: segfault at ffffffffe4033160 rip 0000003fe1276170 rsp 00007fff228e2b28 error 4 This is a e-mail server running Postfix.
Now I got a core file. This is RHEL 5.2 : # gdb /usr/sbin/saslauthd core.25441 GNU gdb Red Hat Linux (6.5-37.el5_2.2rh) Copyright (C) 2006 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "x86_64-redhat-linux-gnu"...Using host libthread_db library "/lib64/libthread_db.so.1". warning: .dynamic section for "/usr/lib64/libgssapi_krb5.so.2" is not at the expected address warning: difference appears to be caused by prelink, adjusting expectations warning: .dynamic section for "/usr/lib64/libkrb5.so.3" is not at the expected address warning: .dynamic section for "/usr/lib64/libk5crypto.so.3" is not at the expected address warning: .dynamic section for "/lib64/libcom_err.so.2" is not at the expected address warning: difference appears to be caused by prelink, adjusting expectations warning: .dynamic section for "/usr/lib64/libkrb5support.so.0" is not at the expected address warning: .dynamic section for "/lib64/libresolv.so.2" is not at the expected address warning: difference appears to be caused by prelink, adjusting expectations warning: .dynamic section for "/lib64/libcrypt.so.1" is not at the expected address warning: .dynamic section for "/lib64/libpam.so.0" is not at the expected address warning: .dynamic section for "/usr/lib64/libldap-2.3.so.0" is not at the expected address warning: difference appears to be caused by prelink, adjusting expectations warning: .dynamic section for "/usr/lib64/liblber-2.3.so.0" is not at the expected address warning: difference appears to be caused by prelink, adjusting expectations warning: .dynamic section for "/lib64/libcrypto.so.6" is not at the expected address warning: difference appears to be caused by prelink, adjusting expectations warning: .dynamic section for "/lib64/libc.so.6" is not at the expected address warning: .dynamic section for "/lib64/libdl.so.2" is not at the expected address warning: difference appears to be caused by prelink, adjusting expectations warning: .dynamic section for "/lib64/libkeyutils.so.1" is not at the expected address warning: difference appears to be caused by prelink, adjusting expectations warning: .dynamic section for "/lib64/libselinux.so.1" is not at the expected address warning: difference appears to be caused by prelink, adjusting expectations warning: .dynamic section for "/lib64/libaudit.so.0" is not at the expected address warning: .dynamic section for "/usr/lib64/libsasl2.so.2" is not at the expected address warning: difference appears to be caused by prelink, adjusting expectations warning: .dynamic section for "/lib64/libssl.so.6" is not at the expected address warning: difference appears to be caused by prelink, adjusting expectations warning: .dynamic section for "/usr/lib64/libz.so.1" is not at the expected address warning: difference appears to be caused by prelink, adjusting expectations warning: .dynamic section for "/lib64/ld-linux-x86-64.so.2" is not at the expected address warning: .dynamic section for "/lib64/libsepol.so.1" is not at the expected address warning: difference appears to be caused by prelink, adjusting expectations Reading symbols from /usr/lib64/libgssapi_krb5.so.2...done. Loaded symbols for /usr/lib64/libgssapi_krb5.so.2 Reading symbols from /usr/lib64/libkrb5.so.3...done. Loaded symbols for /usr/lib64/libkrb5.so.3 Reading symbols from /usr/lib64/libk5crypto.so.3...done. Loaded symbols for /usr/lib64/libk5crypto.so.3 Reading symbols from /lib64/libcom_err.so.2...done. Loaded symbols for /lib64/libcom_err.so.2 Reading symbols from /usr/lib64/libkrb5support.so.0...done. Loaded symbols for /usr/lib64/libkrb5support.so.0 Reading symbols from /lib64/libresolv.so.2...done. Loaded symbols for /lib64/libresolv.so.2 Reading symbols from /lib64/libcrypt.so.1...done. Loaded symbols for /lib64/libcrypt.so.1 Reading symbols from /lib64/libpam.so.0...done. Loaded symbols for /lib64/libpam.so.0 Reading symbols from /usr/lib64/libldap-2.3.so.0...done. Loaded symbols for /usr/lib64/libldap-2.3.so.0 Reading symbols from /usr/lib64/liblber-2.3.so.0...done. Loaded symbols for /usr/lib64/liblber-2.3.so.0 Reading symbols from /lib64/libcrypto.so.6...done. Loaded symbols for /lib64/libcrypto.so.6 Reading symbols from /lib64/libc.so.6...done. Loaded symbols for /lib64/libc.so.6 Reading symbols from /lib64/libdl.so.2...done. Loaded symbols for /lib64/libdl.so.2 Reading symbols from /lib64/libkeyutils.so.1...done. Loaded symbols for /lib64/libkeyutils.so.1 Reading symbols from /lib64/libselinux.so.1...done. Loaded symbols for /lib64/libselinux.so.1 Reading symbols from /lib64/libaudit.so.0...done. Loaded symbols for /lib64/libaudit.so.0 Reading symbols from /usr/lib64/libsasl2.so.2...Reading symbols from /usr/lib/debug/usr/lib64/libsasl2.so.2.0.22.debug...done. done. Loaded symbols for /usr/lib64/libsasl2.so.2 Reading symbols from /lib64/libssl.so.6...done. Loaded symbols for /lib64/libssl.so.6 Reading symbols from /usr/lib64/libz.so.1...done. Loaded symbols for /usr/lib64/libz.so.1 Reading symbols from /lib64/ld-linux-x86-64.so.2...done. Loaded symbols for /lib64/ld-linux-x86-64.so.2 Reading symbols from /lib64/libsepol.so.1...done. Loaded symbols for /lib64/libsepol.so.1 Reading symbols from /lib64/libnss_files.so.2...done. Loaded symbols for /lib64/libnss_files.so.2 Core was generated by `/usr/sbin/saslauthd -m /var/run/saslauthd -a shadow'. Program terminated with signal 11, Segmentation fault. #0 0x0000003fe1276170 in ?? ()
I'm also increasing severity -- problem occured twice in last 24 hours.
Comment 4 removed.
(gdb) bt full #0 0x0000003fe1276170 in ?? () No symbol table info available. #1 0x0000003fe1275ea6 in ?? () No symbol table info available. #2 0x00007fff06bea3e0 in ?? () No symbol table info available. #3 0x0000003fe154cf60 in ?? () No symbol table info available. #4 0x00007fff06bea3e0 in ?? () No symbol table info available. #5 0x0000000000406635 in auth_shadow (login=0xffffffffe4033160 <Address 0xffffffffe4033160 out of bounds>, password=0x7fff06bea2d0 "PASSWORD-BLOCKED", service=<value optimized out>, realm=<value optimized out>) at saslauthd/auth_shadow.c:183 today = <value optimized out> cpw = <value optimized out> pw = <value optimized out> sp = (struct spwd *) 0x3fe154cf60 #6 0x000000000040e1dd in do_auth (_login=<value optimized out>, password=0x6f <Address 0x6f out of bounds>, service=0x7fff06bea1c0 "smtp", realm=0x7fff06bea0b0 "HOSTNAME-BLOCKED") at saslauthd/saslauthd-main.c:399 lkup_result = {bucket = {creds = '\0' <repeats 59 times>, user_offt = 0, realm_offt = 0, service_offt = 0, pwd_digest = '\0' <repeats 15 times>, created = 0}, read_bucket = 0x0, hash_offset = 0, status = 0} response = <value optimized out> cached = 0 login_buff = '\0' <repeats 512 times> login = 0x7fff06bea3e0 "news" __func__ = "do_auth" #7 0x000000000040c3fe in do_request (conn_fd=7) at saslauthd/ipc_unix.c:426 count = 18 ncount = <value optimized out> response = <value optimized out> login = "news\000ting\000tor\000\000\000��T�?\000\000\000#W)�?\000\000\000��T�?\000\000\000`�T�?\000\000\000\005\000\000\000\000\000\000\000`Aa\000\000\000\000\000�>a\000\000\000\000\000`�T�?\000\000\000\030\000\000\000\000\000\000\000 ��\006�\177\000\000P��\006�\177", '\0' <repeats 19 times>, "\235t\000\000 L\202 ��\006�\177\000\000`Aa\000\000\000\000\000���\006�\177\000\000�H:\035", '\0' <repeats 12 times>, "\001", '\0' <repeats 15 times>, "\023�&�\000\000\000\000���\006�\177\000\000\000\000\000\000\000\000\000\00025437\n\000\000"... password = "PASSWORD-BLOCKED", '\0' <repeats 13 times>, "`�T�?\000\000\000����\000\000\000\000\220u��?\000\000\000.\000\000\000\000\000\000\000`\224T�?", '\0' <repeats 11 times>, "\034+'�?\000\000\000.\000\000\000\000\000\000\000\220��\006�\177\000\000\002\037��?\000\000\000M\r'�?\000\000\000/\000\000\000\000\000\000\000/\000\000\000\000\000\000\000�$:\035\000\000\000\0001p&�?\000\000\000\000\200��x\000\000\000���\006�\177\000\000\005\000\000\000\000\000\000\000`Aa\000\000\000\000\000�>a", '\0' <repeats 22 times>, "\235t\000\000 L\2020\000\000\0000\000\000\000"... service = "smtp\000*\000\000\000\000\000\000\000\000\000\000\001\000\000\000\000\000\000\0000��\006�\177\000\000\000\000\000\000\000\000\000\000\023��\006�\177\000\000���\006�\177\000\000���\006�\177\000\000\001\000\000\000\000\000\000\000���\006�\177\000\000q\027A\000\000\000\000\000c\000\000\000\000\000\000\0000��\006�\177\000\000(\030.�?\000\000\000\001\200��\000\000\000\000���\006�\177\000\000���\006�\177\000\000���\006�\177\000\000���\006�\177\000\000���\006�\177\000\000\023��\006�\177\000\000���\006�\177\000\000\023��\006�\177", '\0' <repeats 42 times>, "/\000\000\000\004\000\000\000"... realm = "HOSTNAME-BLOCKED\000\006�\177\000\0006���?", '\0' <repeats 11 times>, "\001\000\000\000\000\000\000\000�25437\001\000\000\000\000\000\000\000H\034`�?\000\000\000 �\203�?", '\0' <repeats 11 times>, "�/`�?\000\000\000(\000\000\0000\000\000\000\200��\006�\177\000\000���\006�\177\000\000\000����*\000\000�\235\001�\026\000\000\000�", '\0' <repeats 15 times>, "�\215�\006�\177\000\000\002\000\000\000\000\000\000\000�N���*\000\000\000\000\000\000\000\000\000\000\001\000\000\000\000\000\000\000���\006�\177\000\000�\023��?", '\0' <repeats 11 times>... __func__ = "do_request" #8 0x000000000040c71e in ipc_loop () at saslauthd/ipc_unix.c:277 rc = 0 conn_fd = 7 __func__ = "ipc_loop" #9 0x000000000040de3a in main (argc=<value optimized out>, argv=<value optimized out>) at saslauthd/saslauthd-main.c:358 option = <value optimized out> rc = <value optimized out> x = 4 lockinfo = {l_type = 1, l_whence = 0, l_start = 0, l_len = 0, l_pid = -519988288} ---Type <return> to continue, or q <return> to quit--- auth_mech_name = 0x1d3a23b0 "shadow" pid_file_size = 4 __func__ = "main" #10 0x0000003fe121d8a4 in ?? () No symbol table info available. #11 0x00007fff06bea698 in ?? () No symbol table info available. #12 0x00007fff06bea698 in ?? () No symbol table info available. #13 0x00000005aaaab000 in ?? () No symbol table info available. #14 0x000000000040da80 in show_usage () at saslauthd/saslauthd-main.c:994 pid_fd = 4 pid_file_lock = 0x1d3a24d0 "/var/run/saslauthd/saslauthd.pid" master_pid = 25437 startup_pipe = {4, 5} pid_file_lock_fd = 3 pid_file = 0x1d3a24d0 "/var/run/saslauthd/saslauthd.pid" num_procs = 5 run_path = 0x1d3a2460 "/var/run/saslauthd" auth_mech = (authmech_t *) 0x613de0 g_argv = (char **) 0x7fff06bea698 g_argc = 5 flags = 356 mech_option = 0x0 #15 0x0000003fe1019bc0 in ?? () No symbol table info available. #16 0x826a7d0c73ece9aa in ?? () No symbol table info available. #17 0x0000000000000000 in ?? () No symbol table info available.
This seems to point to this code in saslauthd/auth_shadow.c: /* * Note: no check for SHADOW_PW_LOCKED. Returning a "locked" notification * would allow login-id namespace probes, and violates our policy of * not returning any information about a login until we have validated * the password. */ cpw = strdup((const char *)crypt(password, sp->sp_pwdp)); if (strcmp(sp->sp_pwdp, cpw)) { if (flags & VERBOSE) { syslog(LOG_DEBUG, "DEBUG: auth_shadow: pw mismatch: '%s' != '%s'", sp->sp_pwdp, cpw); } free(cpw); RETURN("NO"); } free(cpw); Note that the crypt() function can return a NULL on errors, but this is not checked. strdup on a NULL pointer would cause a segmentation fault which could be what we're seeing here. In any case, I'm not seeing the point of strdup() here at all. Perhaps the call can just be removed. (In the case the strdup() is there because of a possible problem with multithreading creating a race condition to the access of the static pointer returned by crypt(), then it would seem that the solution involves locking a mutex before calling crypt() and only releasing it after strdup()).
I don't think crypt can fail in normal circumstances. It is just computing the hash and it is not allocating anything. Even the man page says the only errno for crypt is "ENOSYS The crypt() function was not implemented, probably because of U.S.A. export restrictions." Also the saslauthd is not multithreaded so a race should not happen here either. Can you please install also glibc-debuginfo package and show the backtrace with it?
Ok, installed glibc-debuginfo (sorry for the delay, I was on a holiday).
Okay, so here's the new backtrace: (gdb) bt #0 0x000000326c0782d0 in strlen () from /lib64/libc.so.6 #1 0x000000326c078006 in *__GI___strdup (s=0x6e2371a0 <Address 0x6e2371a0 out of bounds>) at strdup.c:42 #2 0x0000000000406635 in auth_shadow (login=0x7fff524e1000 "mail", password=0x7fff524e0ef0 "123456", service=<value optimized out>, realm=<value optimized out>) at saslauthd/auth_shadow.c:183 #3 0x000000000040e1dd in do_auth (_login=<value optimized out>, password=0x78 <Address 0x78 out of bounds>, service=0x7fff524e0de0 "smtp", realm=0x7fff524e0cd0 "REMOVED") at saslauthd/saslauthd-main.c:399 #4 0x000000000040c3fe in do_request (conn_fd=7) at saslauthd/ipc_unix.c:426 #5 0x000000000040c71e in ipc_loop () at saslauthd/ipc_unix.c:277 #6 0x000000000040de3a in main (argc=<value optimized out>, argv=<value optimized out>) at saslauthd/saslauthd-main.c:358 #7 0x000000326c01d8b4 in __libc_start_main (main=0x40da80 <main>, argc=5, ubp_av=0x7fff524e12b8, init=<value optimized out>, fini=<value optimized out>, rtld_fini=<value optimized out>, stack_end=0x7fff524e12a8) at libc-start.c:231 #8 0x00000000004035f9 in _start ()
bt full: #0 0x000000326c0782d0 in strlen () from /lib64/libc.so.6 mallstream = (FILE *) 0x0 tr_old_memalign_hook = (void *(*)(size_t, size_t, const void *)) 0 tr_old_malloc_hook = (void *(*)(size_t, const void *)) 0 tr_old_realloc_hook = (void *(*)(void *, size_t, const void *)) 0 lock = 0 mallenv = "MALLOC_TRACE" malloc_trace_buffer = 0x0 tr_old_free_hook = (void (*)(void *, const void *)) 0 mallwatch = (void *) 0x0 #1 0x000000326c078006 in *__GI___strdup (s=0x6e2371a0 <Address 0x6e2371a0 out of bounds>) at strdup.c:42 len = <value optimized out> new = <value optimized out> #2 0x0000000000406635 in auth_shadow (login=0x7fff524e1000 "mail", password=0x7fff524e0ef0 "123456", service=<value optimized out>, realm=<value optimized out>) at saslauthd/auth_shadow.c:183 today = <value optimized out> cpw = <value optimized out> pw = <value optimized out> sp = (struct spwd *) 0x326c34ffc0 #3 0x000000000040e1dd in do_auth (_login=<value optimized out>, password=0x78 <Address 0x78 out of bounds>, service=0x7fff524e0de0 "smtp", realm=0x7fff524e0cd0 "REMOVED") at saslauthd/saslauthd-main.c:399 lkup_result = {bucket = {creds = '\0' <repeats 59 times>, user_offt = 0, realm_offt = 0, service_offt = 0, pwd_digest = '\0' <repeats 15 times>, created = 0}, read_bucket = 0x0, hash_offset = 0, status = 0} response = <value optimized out> cached = 0 login_buff = '\0' <repeats 512 times> login = 0x7fff524e1000 "mail" __func__ = "do_auth"
This is caused by missing prototype for crypt() in auth_shadow.c. As this is DoS although not triggerable on-demand I am adding Security keyword.
This issue appears to have been fixed upstream: http://archives.free.net.ph/message/20090220.215950.01b05224.en.html https://bugzilla.andrew.cmu.edu/cgi-bin/cvsweb.cgi/src/sasl/saslauthd/auth_shadow.c.diff?r1=1.9&r2=1.10 Any chance of getting it included in a future security update?
Yes, this is scheduled for RHEL 5.4 cyrus-sasl errata.
I'm glad that you did not defer it to RHEL 6. (!) Current version is crashing servers, and some people treat this as a security issue. Why don't you push it to RHEL 5.3 immediately? Sincerely, Devrim
Good afternoon, I would like to ask you to please grab the updated version of cyrus-sasl for testing. Your feedback on this issue would be greatly appreciated. The following package builds should resolve all the issues reported in this bug. http://people.redhat.com/~cward/5.4.0/cyrus-sasl/ Thanks!
Can I have confirmation that this bug *only* affects 64-bit ?
Yes, on 32bit the pointers have the same length as int.
I no longer had access to the customer machine that was showing the problem, back when the bug was fixed, so I cannot confirm that it solves the problem nor deny it. Sorry.
An advisory has been issued which should help the problem described in this bug report. This report is therefore being closed with a resolution of ERRATA. For more information on therefore solution and/or where to find the updated files, please follow the link below. You may reopen this bug report if the solution does not work for you. http://rhn.redhat.com/errata/RHBA-2009-1330.html