Bug 1030142

Summary: glibc module 'crypt', function 'crypt' leads to segfaults and coredumps
Product: [Fedora] Fedora Reporter: Heinz-Peter Heidinger <hph>
Component: glibcAssignee: Carlos O'Donell <codonell>
Status: CLOSED NOTABUG QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: high Docs Contact:
Priority: unspecified    
Version: 19CC: berni, codonell, fweimer, jakub, law, pfrankli, schwab, spoyarek
Target Milestone: ---   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2014-09-29 11:09:17 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Heinz-Peter Heidinger 2013-11-14 02:03:31 UTC
Description of problem:
After upgrading from Fedora-17 to Fedora-19 I found that the Tivoli-Storage-Manager-Client 'dsmc' segfaults and coredumps.

-------------------------------------------------------------------------------
[root@hphws ~]# dsmc
IBM Tivoli Storage Manager
Command Line Backup-Archive Client Interface
  Client Version 6, Release 4, Level 1.0  
  Client date/time: 11/14/2013 00:41:25
(c) Copyright by IBM Corporation and other(s) 1990, 2013. All Rights Reserved.

Node Name: HPHWS
Aborted (core dumped)
[root@hphws ~]# 
-------------------------------------------------------------------------------

I tried downgrading the TSM client software from 6.4.0.7 -> 6.3.1 -> 6.2.2 - but always the same result. All versions worked fine in Fedora-17. An update
to TSM client 6.4.1 (see above) lead to the same result.

So I ran the program thru GDB ...

-------------------------------------------------------------------------------
[root@hphws ~]# gdb dsmc
GNU gdb (GDB) Fedora 7.6.1-42.fc19
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /opt/tivoli/tsm/client/ba/bin/dsmc...done.
(gdb) run
Starting program: /bin/dsmc 
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
[New Thread 0x7fffefa44700 (LWP 3231)]
IBM Tivoli Storage Manager
Command Line Backup-Archive Client Interface
  Client Version 6, Release 4, Level 1.0  
  Client date/time: 11/14/2013 00:48:57
(c) Copyright by IBM Corporation and other(s) 1990, 2013. All Rights Reserved.

Node Name: HPHWS

Program received signal SIGSEGV, Segmentation fault.
0x0000000000685ad6 in psCreateCryptoKey(unsigned char*, char*) ()
Missing separate debuginfos, use: debuginfo-install TIVsm-BA-6.4.1-0.x86_64
(gdb)
-------------------------------------------------------------------------------

Well, the program traps in a module 'psCreateCryptoKey'.
I then downloaded debuginfos for glibc (2.17 as for F-19) and stepped
through the programm with 'ddd dsmc'. By that I found out that 'psCreateCryptoKey' does a call to 'crypt' from glibc module 'crypt'.

I then used GDB again directly with breakpoints @ 'main', 'psCreateCryptoKey'
and 'crypt'; here is the result:

-------------------------------------------------------------------------------
[root@hphws ~]# gdb dsmc
GNU gdb (GDB) Fedora 7.6.1-42.fc19
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /opt/tivoli/tsm/client/ba/bin/dsmc...done.
(gdb) break main
Breakpoint 1 at 0x4278b8
(gdb) break psCreateCryptoKey
Breakpoint 2 at 0x685a3c
(gdb) break crypt
Breakpoint 3 at 0x425f40
(gdb) run
Starting program: /bin/dsmc 
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".

Breakpoint 1, 0x00000000004278b8 in main ()
Missing separate debuginfos, use: debuginfo-install TIVsm-BA-6.4.1-0.x86_64
(gdb) next
Single stepping until exit from function main,
which has no line number information.
[New Thread 0x7fffefa44700 (LWP 5144)]
IBM Tivoli Storage Manager
Command Line Backup-Archive Client Interface
  Client Version 6, Release 4, Level 1.0  
  Client date/time: 11/14/2013 01:26:12
(c) Copyright by IBM Corporation and other(s) 1990, 2013. All Rights Reserved.

Node Name: HPHWS

Breakpoint 2, 0x0000000000685a3c in psCreateCryptoKey(unsigned char*, char*) ()
(gdb) next
Single stepping until exit from function _Z17psCreateCryptoKeyPhPc,
which has no line number information.

Breakpoint 3, crypt (key=0x7ffffffefbc0 " 2301152\032", salt=0x7ffffffefbc0 " 2301152\032") at crypt-entry.c:164
164	{
(gdb) finish
Run till exit from #0  crypt (key=0x7ffffffefbc0 " 2301152\032", salt=0x7ffffffefbc0 " 2301152\032") at crypt-entry.c:164
0x0000000000685ad6 in psCreateCryptoKey(unsigned char*, char*) ()
Value returned is $1 = 0x0 
(gdb) step
Single stepping until exit from function _Z17psCreateCryptoKeyPhPc,
which has no line number information.

Program received signal SIGSEGV, Segmentation fault.
0x0000000000685ad6 in psCreateCryptoKey(unsigned char*, char*) ()
(gdb) 
-------------------------------------------------------------------------------

I then  extracted 'libcrypt' from previous releases of glibc-2.16
supplied with Fedora-18; starting stepping back to glibc-2.16-34.
I symlinked that in /usr/lib64 as follows:

------------------------------------------------------------------------------

[root@hphws ~]# cd /usr/lib64
[root@hphws lib64]# ls -l libcrypt*
-rwxr-xr-x  1 root root   43808 Oct 28 23:42 libcrypt-2.17.so
lrwxrwxrwx  1 root root      19 Oct 31 16:52 libcrypto.so.10 -> libcrypto.so.1.0.1e
-rwxr-xr-x  1 root root 1960416 Oct 29 17:16 libcrypto.so.1.0.1e
lrwxrwxrwx. 1 root root      22 Oct 24 11:24 libcryptsetup.so.4 -> libcryptsetup.so.4.5.0
-rwxr-xr-x  1 root root  155992 Aug  4 20:39 libcryptsetup.so.4.5.0
lrwxrwxrwx  1 root root      25 Nov  7 17:15 libcrypt.so -> ../../lib64/libcrypt.so.1
lrwxrwxrwx  1 root root      16 Nov  9 09:32 libcrypt.so.1 -> libcrypt-2.17.so
[root@hphws lib64]# rm libcrypt.so.1
rm: remove symbolic link 'libcrypt.so.1'? y

[root@hphws lib64]# ln -s /usr/local/glibc-2.16-34/lib64/libcrypt-2.16.so libcrypt.so.1
[root@hphws lib64]# ls -l libcrypt*
-rwxr-xr-x  1 root root   43808 Oct 28 23:42 libcrypt-2.17.so
lrwxrwxrwx  1 root root      19 Oct 31 16:52 libcrypto.so.10 -> libcrypto.so.1.0.1e
-rwxr-xr-x  1 root root 1960416 Oct 29 17:16 libcrypto.so.1.0.1e
lrwxrwxrwx. 1 root root      22 Oct 24 11:24 libcryptsetup.so.4 -> libcryptsetup.so.4.5.0
-rwxr-xr-x  1 root root  155992 Aug  4 20:39 libcryptsetup.so.4.5.0
lrwxrwxrwx  1 root root      25 Nov  7 17:15 libcrypt.so -> ../../lib64/libcrypt.so.1
lrwxrwxrwx  1 root root      47 Nov 14 01:45 libcrypt.so.1 -> /usr/local/glibc-2.16-34/lib64/libcrypt-2.16.so
[root@hphws lib64]# 
------------------------------------------------------------------------------

Same result. The program coredumps already with glibc-2.16-34 from Fedora-18.
Since everything worked fine with F-17 before the upgrade I stepped back
through to glibc-2.16-24 (PL24!!) by unpacking the RPM package to somewhere.
Then I bent the link in /usr/lib64/libcrypt.so.1 to that libcrypt-2.16 in PL24.

------------------------------------------------------------------------------
[root@hphws lib64]# ls -l libcrypt*
-rwxr-xr-x  1 root root   43808 Oct 28 23:42 libcrypt-2.17.so
lrwxrwxrwx  1 root root      19 Oct 31 16:52 libcrypto.so.10 -> libcrypto.so.1.0.1e
-rwxr-xr-x  1 root root 1960416 Oct 29 17:16 libcrypto.so.1.0.1e
lrwxrwxrwx. 1 root root      22 Oct 24 11:24 libcryptsetup.so.4 -> libcryptsetup.so.4.5.0
-rwxr-xr-x  1 root root  155992 Aug  4 20:39 libcryptsetup.so.4.5.0
lrwxrwxrwx  1 root root      25 Nov  7 17:15 libcrypt.so -> ../../lib64/libcrypt.so.1
lrwxrwxrwx  1 root root      47 Nov 14 01:45 libcrypt.so.1 -> /usr/local/glibc-2.16-24/lib64/libcrypt-2.16.so
[root@hphws lib64]# 
------------------------------------------------------------------------------


And here we are ... the TSM client program 'dsmc' works again.
I get a 'tsm>' prompt and communication with the server works, since
authentication to the server with connect credentials works with
glibc-2.16-24 as expected:
------------------------------------------------------------------------------
[root@hphws lib64]# dsmc
IBM Tivoli Storage Manager
Command Line Backup-Archive Client Interface
  Client Version 6, Release 4, Level 1.0  
  Client date/time: 11/14/2013 02:01:14
(c) Copyright by IBM Corporation and other(s) 1990, 2013. All Rights Reserved.

Node Name: HPHWS
Session established with server TSM6LNX: Linux/x86_64
  Server Version 6, Release 3, Level 4.0
  Server date/time: 11/14/2013 02:01:14  Last access: 11/14/2013 02:00:10

tsm> query node
Please enter your user id <HPHWS>: 

Please enter password for user id "HPHWS": 

Session established with server TSM6LNX: Linux/x86_64
  Server Version 6, Release 3, Level 4.0
  Server date/time: 11/14/2013 02:01:42  Last access: 11/14/2013 02:01:14

    Node Name   Platform   Policy Domain   Days Since   Days Since   Locked?
                           Name            Last Access  Password Set        
    _________   _________  _____________   __________   ____________ _______
        HPHWS  Linux x86-64    LF_VTAPE           0         678           0
tsm> 
------------------------------------------------------------------------------




Version-Release number of selected component (if applicable):
glibc-2.17(F-19)        ... carries the bug
glibc-2.16-PL34(F-18)   ... introduced the bug
glibg-2.16-PL24(F-18)   ... everything works fine if steping back to this
                            release  

How reproducible:
see above


Steps to Reproduce:
1. see above
2.
3.

Actual results:
The TSM client 'dsmc' traps due to a NULL pointer returned by
internal function '__crypt_r()' in function 'crypt'. 
(see source file crypt-entry.c, function 'crypt'
{
   ...  
   return __crypt_r (key, salt, &_ufc_foobar);
}
... returns the NULL pointer)

------------------------------------------------------------------------------
comserve@hphws crypt]$ pwd
/usr/src/debug/glibc-2.17-c758a686/crypt
[comserve@hphws crypt]$ grep __crypt_r *
crypt-entry.c:__crypt_r (key, salt, data)
crypt-entry.c:weak_alias (__crypt_r, crypt_r)
crypt-entry.c:      /* Let __crypt_r deal with the error code if FIPS is enabled.  */
crypt-entry.c:  return __crypt_r (key, salt, &_ufc_foobar);
crypt-private.h:extern char *__crypt_r (const char *__key, const char *__salt,
[comserve@hphws crypt]$ 
------------------------------------------------------------------------------



Expected results:
Working version or function 'crypt'.


Additional info:
glibc-2.16 PL34 introduced a lot of patches to the 'crypt' modules:
------------------------------------------------------------------------------
[root@hphws glibc-2.16-34-SRC]# grep crypt *.patch | cut -f1 -d":" | sort | uniq
glibc-fedora.patch
glibc-rh811753-2.patch
glibc-rh811753.patch
glibc-rh984829.patch
------------------------------------------------------------------------------

I can assist with tests since I am equiped with a TSM test environment.
But the trouble is not with TSM but a 'crypt' function that returns
NULL pointers.

Comment 1 Carlos O'Donell 2013-11-14 02:23:24 UTC
*** Bug 1030118 has been marked as a duplicate of this bug. ***

Comment 2 Carlos O'Donell 2013-11-14 02:51:55 UTC
(In reply to Heinz-Peter Heidinger from comment #0)
> Run till exit from #0  crypt (key=0x7ffffffefbc0 " 2301152\032",
> salt=0x7ffffffefbc0 " 2301152\032") at crypt-entry.c:164
> 0x0000000000685ad6 in psCreateCryptoKey(unsigned char*, char*) ()
> Value returned is $1 = 0x0 

In glibc 2.17 the crypt function implementation was tightened up
to allow for future implementation improvements, standards 
conformance, and FIPS compliance. From the 2.17 NEWS:

* The `crypt' function now fails if passed salt bytes that violate the
  specification for those values.  On Linux, the `crypt' function will
  consult /proc/sys/crypto/fips_enabled to determine if "FIPS mode" is
  enabled, and fail on encrypted strings using the MD5 or DES algorithm
  when the mode is enabled.

In this particular case the salt is an invalid POSIX salt that 
contains an ASCII space " " in character 0 outside of the allowed
alphabet for salts which is `./0-9a-zA-Z' and therefore crypt will
reject that salt as invalid, return NULL, and set errno.

Previous to glibc 2.17 a call to crypt would accept invalid POSIX
salts.

You have a couple of options:

* Fix the caller to pass a valid POSIX salt and/or ensure that 
  you check the return of crypt for a NULL return.

* Preload a shared library with a custom implementation of crypt
  that supports non-POSIX salts.

* Preload a shared library with a wrapper for crypt which fixes
  up the salt inputs and then uses RTLD_NEXT to call the real crypt
  with the new arguments.

Does that answer your question?

Comment 3 Bernhard Schmidt 2013-11-25 12:53:07 UTC
Upstream TSM bug:

http://www-01.ibm.com/support/docview.wss?uid=swg1IC92662&myns=apar&mynp=DO                                                               

IC92662: TSM CLIENT CAN CRASH WITH CERTAIN NODENAMES ON LINUX DISTRIBUTIONS IF USING GLIBC 2.16 OR HIGHER IN THE FUTURE

APAR status

    Closed as program error.

Error description

    A code defect has been detected in the
    TSM Linux x86 client. While it does not manifest
    today in any currently-supported Linux distributions,
    here would be the symptom:
    When PASSWORDACCESS is set to GENERATE, TSM Linux client
    could crash when trying to read or write the password file.

    This will happen when installed glibc is version 2.16 or higher,
    and only with certain node names. It is not possible to
    identify the node name pattern triggering the issue. The
    crash can occur on short and long names, with and without
    non-alpha characters. However, when the failing combination
    of characters is used as the node name, the problem is
    consistently recreatable.

    One class of node names known to trigger the issue are the
    names of 3 symbols or less. For example, ABC, F35, R2 etc.

    An example of a longer node name is LINUX-123456

    Since currently-supported RedHet and SuSE distributions
    do not yet support this Glibc level, there is no current
    error. However, this APAR is being opened to address
    the potential future issue when that Glibc level is
    supported.

Local fix

    One of the following workarounds can be used:

    1. Downgrade glibc to version 2.15 or lower, if possible.

    2. Change the node name. In most cases, adding, deleting or
       modifying a single character is sufficient.

    3. Set PASSWORDACCESS to PROMPT and add PASSWORD option to
       your dsm.sys options file. Make sure to restrict file
       system access to dsm.sys so non-authorized users can't
       see the node password.

Problem summary

    ****************************************************************
    * USERS AFFECTED: All clients versions 6.3 and 6.4 running     *
    *                 on Linux platforms with glibc version 2.16   *
    *                 or higher.                                   *
    ****************************************************************
    * PROBLEM DESCRIPTION: See ERROR DESCRIPTION.                  *
    ****************************************************************
    * RECOMMENDATION: Apply fixing level when available. This      *
    *                 problem is currently projected to be fixed   *
    *                 in levels 6.3.2 and 6.4.2. Note that until   *
    *                 these levels are available, this             *
    *                 information is subject to change at the      *
    *                 discretion of IBM.                           *
    ****************************************************************
    *

Problem conclusion

    The problem has been fixed so that it no longer occurs.

Comment 4 Heinz-Peter Heidinger 2014-09-15 18:36:56 UTC
As Bernhard Schmidt stated above: This bug vanished with TSM-Client version
6.4.2.0 (maintenance release) for linux X64.

I tried this a some weeks ago (Aug/2014) with the recent kernel in Fedora at that time (3.15.xxx) and everything was working fine - as before ...

Comment 5 Siddhesh Poyarekar 2014-09-29 11:09:17 UTC
Fixed in third-part application, not a bug in glibc.