Description of problem:
When changing a password in fips mode, passwd segfaults

Version-Release number of selected component (if applicable):
pam-1.1.5-5.fc17.x86_64

How reproducible:
always

Steps to Reproduce:
1. install F17 test in a VM (or on hardware without AES-NI)
2. boot with boot=/dev/vda2 fips=1   (if /dev/vda2 is your /boot partition)
3. login as your user (root cannot login)
4. become superuser
5. passwd youruser
  
Actual results:
Program received signal SIGSEGV, Segmentation fault.
__strncmp_sse2 () at ../sysdeps/x86_64/strcmp.S:214
214             movlpd  (%rsi), %xmm2


Expected results:
a new password :)

Additional info:
(gdb) bt
#0  __strncmp_sse2 () at ../sysdeps/x86_64/strcmp.S:214
#1  0x00007fbc76cc7811 in create_password_hash (
    pamh=pamh@entry=0x7fbc81811df0, password=0x7fbc8181a0c0 "testme",
    ctrl=ctrl@entry=8395044, rounds=<optimized out>) at passverify.c:427
#2  0x00007fbc76cc4e87 in pam_sm_chauthtok (pamh=0x7fbc81811df0,
    flags=<optimized out>, argc=<optimized out>, argv=<optimized out>)
    at pam_unix_passwd.c:798
#3  0x00007fbc7eba00c4 in _pam_dispatch_aux (use_cached_chain=<optimized out>,
    resumed=<optimized out>, h=0x7fbc81817af0, flags=8192, pamh=0x7fbc81811df0)
    at pam_dispatch.c:110
#4  _pam_dispatch (pamh=pamh@entry=0x7fbc81811df0, flags=flags@entry=8192,
    choice=choice@entry=6) at pam_dispatch.c:407
#5  0x00007fbc7eba4312 in pam_chauthtok (pamh=0x7fbc81811df0, flags=0)
    at pam_password.c:46
#6  0x00007fbc7ffa3e5e in main (argc=<optimized out>, argv=<optimized out>)
    at passwd.c:532


passverify shows:
                crypt_make_salt(sp, 8);
                /* For now be conservative so the resulting hashes
                 * are not too long. 8 bytes of salt prevents dictionary
                 * attacks well enough. */
#ifdef HAVE_CRYPT_GENSALT_R
        }
#endif
        sp = crypt(password, salt);
        if (strncmp(algoid, sp, strlen(algoid)) != 0) {
                /* libxcrypt/libc doesn't know the algorithm, use MD5 */
                pam_syslog(pamh, LOG_ERR,
                           "Algo %s not supported by the crypto backend, "
                           "falling back to MD5\n",
                           on(UNIX_BLOWFISH_PASS, ctrl) ? "blowfish" :
                           on(UNIX_SHA256_PASS, ctrl) ? "sha256" :
                           on(UNIX_SHA512_PASS, ctrl) ? "sha512" : algoid);
                memset(sp, '\0', strlen(sp));
                return crypt_md5_wrapper(password);

According to gdb, sp is NULL and algoid = "$6$" (which seems to be UNIX_SHA512_PASS)

This is consistent with man 3 crypt which says NULL is returned on error, but this code blindly strncmp's the NULL.



So it seems the root cause is that in fips mode, libc does not know UNIX_SHA512_PASS.