Description of problem: After upgrading from Fedora-17 to Fedora-19 I found that the Tivoli-Storage-Manager-Client 'dsmc' segfaults and coredumps. ------------------------------------------------------------------------------- [root@hphws ~]# dsmc IBM Tivoli Storage Manager Command Line Backup-Archive Client Interface Client Version 6, Release 4, Level 1.0 Client date/time: 11/14/2013 00:41:25 (c) Copyright by IBM Corporation and other(s) 1990, 2013. All Rights Reserved. Node Name: HPHWS Aborted (core dumped) [root@hphws ~]# ------------------------------------------------------------------------------- I tried downgrading the TSM client software from 6.4.0.7 -> 6.3.1 -> 6.2.2 - but always the same result. All versions worked fine in Fedora-17. An update to TSM client 6.4.1 (see above) lead to the same result. So I ran the program thru GDB ... ------------------------------------------------------------------------------- [root@hphws ~]# gdb dsmc GNU gdb (GDB) Fedora 7.6.1-42.fc19 Copyright (C) 2013 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-redhat-linux-gnu". For bug reporting instructions, please see: <http://www.gnu.org/software/gdb/bugs/>... Reading symbols from /opt/tivoli/tsm/client/ba/bin/dsmc...done. (gdb) run Starting program: /bin/dsmc [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". [New Thread 0x7fffefa44700 (LWP 3231)] IBM Tivoli Storage Manager Command Line Backup-Archive Client Interface Client Version 6, Release 4, Level 1.0 Client date/time: 11/14/2013 00:48:57 (c) Copyright by IBM Corporation and other(s) 1990, 2013. All Rights Reserved. Node Name: HPHWS Program received signal SIGSEGV, Segmentation fault. 0x0000000000685ad6 in psCreateCryptoKey(unsigned char*, char*) () Missing separate debuginfos, use: debuginfo-install TIVsm-BA-6.4.1-0.x86_64 (gdb) ------------------------------------------------------------------------------- Well, the program traps in a module 'psCreateCryptoKey'. I then downloaded debuginfos for glibc (2.17 as for F-19) and stepped through the programm with 'ddd dsmc'. By that I found out that 'psCreateCryptoKey' does a call to 'crypt' from glibc module 'crypt'. I then used GDB again directly with breakpoints @ 'main', 'psCreateCryptoKey' and 'crypt'; here is the result: ------------------------------------------------------------------------------- [root@hphws ~]# gdb dsmc GNU gdb (GDB) Fedora 7.6.1-42.fc19 Copyright (C) 2013 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-redhat-linux-gnu". For bug reporting instructions, please see: <http://www.gnu.org/software/gdb/bugs/>... Reading symbols from /opt/tivoli/tsm/client/ba/bin/dsmc...done. (gdb) break main Breakpoint 1 at 0x4278b8 (gdb) break psCreateCryptoKey Breakpoint 2 at 0x685a3c (gdb) break crypt Breakpoint 3 at 0x425f40 (gdb) run Starting program: /bin/dsmc [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". Breakpoint 1, 0x00000000004278b8 in main () Missing separate debuginfos, use: debuginfo-install TIVsm-BA-6.4.1-0.x86_64 (gdb) next Single stepping until exit from function main, which has no line number information. [New Thread 0x7fffefa44700 (LWP 5144)] IBM Tivoli Storage Manager Command Line Backup-Archive Client Interface Client Version 6, Release 4, Level 1.0 Client date/time: 11/14/2013 01:26:12 (c) Copyright by IBM Corporation and other(s) 1990, 2013. All Rights Reserved. Node Name: HPHWS Breakpoint 2, 0x0000000000685a3c in psCreateCryptoKey(unsigned char*, char*) () (gdb) next Single stepping until exit from function _Z17psCreateCryptoKeyPhPc, which has no line number information. Breakpoint 3, crypt (key=0x7ffffffefbc0 " 2301152\032", salt=0x7ffffffefbc0 " 2301152\032") at crypt-entry.c:164 164 { (gdb) finish Run till exit from #0 crypt (key=0x7ffffffefbc0 " 2301152\032", salt=0x7ffffffefbc0 " 2301152\032") at crypt-entry.c:164 0x0000000000685ad6 in psCreateCryptoKey(unsigned char*, char*) () Value returned is $1 = 0x0 (gdb) step Single stepping until exit from function _Z17psCreateCryptoKeyPhPc, which has no line number information. Program received signal SIGSEGV, Segmentation fault. 0x0000000000685ad6 in psCreateCryptoKey(unsigned char*, char*) () (gdb) ------------------------------------------------------------------------------- I then extracted 'libcrypt' from previous releases of glibc-2.16 supplied with Fedora-18; starting stepping back to glibc-2.16-34. I symlinked that in /usr/lib64 as follows: ------------------------------------------------------------------------------ [root@hphws ~]# cd /usr/lib64 [root@hphws lib64]# ls -l libcrypt* -rwxr-xr-x 1 root root 43808 Oct 28 23:42 libcrypt-2.17.so lrwxrwxrwx 1 root root 19 Oct 31 16:52 libcrypto.so.10 -> libcrypto.so.1.0.1e -rwxr-xr-x 1 root root 1960416 Oct 29 17:16 libcrypto.so.1.0.1e lrwxrwxrwx. 1 root root 22 Oct 24 11:24 libcryptsetup.so.4 -> libcryptsetup.so.4.5.0 -rwxr-xr-x 1 root root 155992 Aug 4 20:39 libcryptsetup.so.4.5.0 lrwxrwxrwx 1 root root 25 Nov 7 17:15 libcrypt.so -> ../../lib64/libcrypt.so.1 lrwxrwxrwx 1 root root 16 Nov 9 09:32 libcrypt.so.1 -> libcrypt-2.17.so [root@hphws lib64]# rm libcrypt.so.1 rm: remove symbolic link 'libcrypt.so.1'? y [root@hphws lib64]# ln -s /usr/local/glibc-2.16-34/lib64/libcrypt-2.16.so libcrypt.so.1 [root@hphws lib64]# ls -l libcrypt* -rwxr-xr-x 1 root root 43808 Oct 28 23:42 libcrypt-2.17.so lrwxrwxrwx 1 root root 19 Oct 31 16:52 libcrypto.so.10 -> libcrypto.so.1.0.1e -rwxr-xr-x 1 root root 1960416 Oct 29 17:16 libcrypto.so.1.0.1e lrwxrwxrwx. 1 root root 22 Oct 24 11:24 libcryptsetup.so.4 -> libcryptsetup.so.4.5.0 -rwxr-xr-x 1 root root 155992 Aug 4 20:39 libcryptsetup.so.4.5.0 lrwxrwxrwx 1 root root 25 Nov 7 17:15 libcrypt.so -> ../../lib64/libcrypt.so.1 lrwxrwxrwx 1 root root 47 Nov 14 01:45 libcrypt.so.1 -> /usr/local/glibc-2.16-34/lib64/libcrypt-2.16.so [root@hphws lib64]# ------------------------------------------------------------------------------ Same result. The program coredumps already with glibc-2.16-34 from Fedora-18. Since everything worked fine with F-17 before the upgrade I stepped back through to glibc-2.16-24 (PL24!!) by unpacking the RPM package to somewhere. Then I bent the link in /usr/lib64/libcrypt.so.1 to that libcrypt-2.16 in PL24. ------------------------------------------------------------------------------ [root@hphws lib64]# ls -l libcrypt* -rwxr-xr-x 1 root root 43808 Oct 28 23:42 libcrypt-2.17.so lrwxrwxrwx 1 root root 19 Oct 31 16:52 libcrypto.so.10 -> libcrypto.so.1.0.1e -rwxr-xr-x 1 root root 1960416 Oct 29 17:16 libcrypto.so.1.0.1e lrwxrwxrwx. 1 root root 22 Oct 24 11:24 libcryptsetup.so.4 -> libcryptsetup.so.4.5.0 -rwxr-xr-x 1 root root 155992 Aug 4 20:39 libcryptsetup.so.4.5.0 lrwxrwxrwx 1 root root 25 Nov 7 17:15 libcrypt.so -> ../../lib64/libcrypt.so.1 lrwxrwxrwx 1 root root 47 Nov 14 01:45 libcrypt.so.1 -> /usr/local/glibc-2.16-24/lib64/libcrypt-2.16.so [root@hphws lib64]# ------------------------------------------------------------------------------ And here we are ... the TSM client program 'dsmc' works again. I get a 'tsm>' prompt and communication with the server works, since authentication to the server with connect credentials works with glibc-2.16-24 as expected: ------------------------------------------------------------------------------ [root@hphws lib64]# dsmc IBM Tivoli Storage Manager Command Line Backup-Archive Client Interface Client Version 6, Release 4, Level 1.0 Client date/time: 11/14/2013 02:01:14 (c) Copyright by IBM Corporation and other(s) 1990, 2013. All Rights Reserved. Node Name: HPHWS Session established with server TSM6LNX: Linux/x86_64 Server Version 6, Release 3, Level 4.0 Server date/time: 11/14/2013 02:01:14 Last access: 11/14/2013 02:00:10 tsm> query node Please enter your user id <HPHWS>: Please enter password for user id "HPHWS": Session established with server TSM6LNX: Linux/x86_64 Server Version 6, Release 3, Level 4.0 Server date/time: 11/14/2013 02:01:42 Last access: 11/14/2013 02:01:14 Node Name Platform Policy Domain Days Since Days Since Locked? Name Last Access Password Set _________ _________ _____________ __________ ____________ _______ HPHWS Linux x86-64 LF_VTAPE 0 678 0 tsm> ------------------------------------------------------------------------------ Version-Release number of selected component (if applicable): glibc-2.17(F-19) ... carries the bug glibc-2.16-PL34(F-18) ... introduced the bug glibg-2.16-PL24(F-18) ... everything works fine if steping back to this release How reproducible: see above Steps to Reproduce: 1. see above 2. 3. Actual results: The TSM client 'dsmc' traps due to a NULL pointer returned by internal function '__crypt_r()' in function 'crypt'. (see source file crypt-entry.c, function 'crypt' { ... return __crypt_r (key, salt, &_ufc_foobar); } ... returns the NULL pointer) ------------------------------------------------------------------------------ comserve@hphws crypt]$ pwd /usr/src/debug/glibc-2.17-c758a686/crypt [comserve@hphws crypt]$ grep __crypt_r * crypt-entry.c:__crypt_r (key, salt, data) crypt-entry.c:weak_alias (__crypt_r, crypt_r) crypt-entry.c: /* Let __crypt_r deal with the error code if FIPS is enabled. */ crypt-entry.c: return __crypt_r (key, salt, &_ufc_foobar); crypt-private.h:extern char *__crypt_r (const char *__key, const char *__salt, [comserve@hphws crypt]$ ------------------------------------------------------------------------------ Expected results: Working version or function 'crypt'. Additional info: glibc-2.16 PL34 introduced a lot of patches to the 'crypt' modules: ------------------------------------------------------------------------------ [root@hphws glibc-2.16-34-SRC]# grep crypt *.patch | cut -f1 -d":" | sort | uniq glibc-fedora.patch glibc-rh811753-2.patch glibc-rh811753.patch glibc-rh984829.patch ------------------------------------------------------------------------------ I can assist with tests since I am equiped with a TSM test environment. But the trouble is not with TSM but a 'crypt' function that returns NULL pointers.
*** Bug 1030118 has been marked as a duplicate of this bug. ***
(In reply to Heinz-Peter Heidinger from comment #0) > Run till exit from #0 crypt (key=0x7ffffffefbc0 " 2301152\032", > salt=0x7ffffffefbc0 " 2301152\032") at crypt-entry.c:164 > 0x0000000000685ad6 in psCreateCryptoKey(unsigned char*, char*) () > Value returned is $1 = 0x0 In glibc 2.17 the crypt function implementation was tightened up to allow for future implementation improvements, standards conformance, and FIPS compliance. From the 2.17 NEWS: * The `crypt' function now fails if passed salt bytes that violate the specification for those values. On Linux, the `crypt' function will consult /proc/sys/crypto/fips_enabled to determine if "FIPS mode" is enabled, and fail on encrypted strings using the MD5 or DES algorithm when the mode is enabled. In this particular case the salt is an invalid POSIX salt that contains an ASCII space " " in character 0 outside of the allowed alphabet for salts which is `./0-9a-zA-Z' and therefore crypt will reject that salt as invalid, return NULL, and set errno. Previous to glibc 2.17 a call to crypt would accept invalid POSIX salts. You have a couple of options: * Fix the caller to pass a valid POSIX salt and/or ensure that you check the return of crypt for a NULL return. * Preload a shared library with a custom implementation of crypt that supports non-POSIX salts. * Preload a shared library with a wrapper for crypt which fixes up the salt inputs and then uses RTLD_NEXT to call the real crypt with the new arguments. Does that answer your question?
Upstream TSM bug: http://www-01.ibm.com/support/docview.wss?uid=swg1IC92662&myns=apar&mynp=DO IC92662: TSM CLIENT CAN CRASH WITH CERTAIN NODENAMES ON LINUX DISTRIBUTIONS IF USING GLIBC 2.16 OR HIGHER IN THE FUTURE APAR status Closed as program error. Error description A code defect has been detected in the TSM Linux x86 client. While it does not manifest today in any currently-supported Linux distributions, here would be the symptom: When PASSWORDACCESS is set to GENERATE, TSM Linux client could crash when trying to read or write the password file. This will happen when installed glibc is version 2.16 or higher, and only with certain node names. It is not possible to identify the node name pattern triggering the issue. The crash can occur on short and long names, with and without non-alpha characters. However, when the failing combination of characters is used as the node name, the problem is consistently recreatable. One class of node names known to trigger the issue are the names of 3 symbols or less. For example, ABC, F35, R2 etc. An example of a longer node name is LINUX-123456 Since currently-supported RedHet and SuSE distributions do not yet support this Glibc level, there is no current error. However, this APAR is being opened to address the potential future issue when that Glibc level is supported. Local fix One of the following workarounds can be used: 1. Downgrade glibc to version 2.15 or lower, if possible. 2. Change the node name. In most cases, adding, deleting or modifying a single character is sufficient. 3. Set PASSWORDACCESS to PROMPT and add PASSWORD option to your dsm.sys options file. Make sure to restrict file system access to dsm.sys so non-authorized users can't see the node password. Problem summary **************************************************************** * USERS AFFECTED: All clients versions 6.3 and 6.4 running * * on Linux platforms with glibc version 2.16 * * or higher. * **************************************************************** * PROBLEM DESCRIPTION: See ERROR DESCRIPTION. * **************************************************************** * RECOMMENDATION: Apply fixing level when available. This * * problem is currently projected to be fixed * * in levels 6.3.2 and 6.4.2. Note that until * * these levels are available, this * * information is subject to change at the * * discretion of IBM. * **************************************************************** * Problem conclusion The problem has been fixed so that it no longer occurs.
As Bernhard Schmidt stated above: This bug vanished with TSM-Client version 6.4.2.0 (maintenance release) for linux X64. I tried this a some weeks ago (Aug/2014) with the recent kernel in Fedora at that time (3.15.xxx) and everything was working fine - as before ...
Fixed in third-part application, not a bug in glibc.