Bug 4074

This bug was reported by Andrew V. Nikulin
<andrew>

I have the patch file saved in my home directory. The
tarball is called pam_patch.tgz --MJG

Andrew writes:

"I discovered a serious enough bug in pwdb and pam packages.

So, yesterday I've killed the entire day trying to
understand why sshd2 doesn't work correctly on my Ultra
SPARC box, but it works fine on Intel (both under RedHat
Linux 6.0). After a few sessions with ltrace, gdb and other
tools I've finally understood that a problem appears only
when MD5 hashes are enabled in pam_pwdb module.

Digging sources of pam_pwdb, pwdb and sshd2 I've got next
thing: pwdb and pam use their own routine crypt_md5() for
MD5 calculation, but sshd2 uses standart crypt() from GLIBC
for such purpose.

I wrote a short test program, which simply encrypts the
phrase "magic_word" with salt "$1$xx", calling GLIBC crypt()
and pam_pwdb crypt_md5(). After running this program on SUN
and Intel I've got the following results:

Intel:

pam_pwdb crypt_md5():   $1$xx$ipOG9IJsAo23Ba44.YOUd1
GLIBC crypt():          $1$xx$ipOG9IJsAo23Ba44.YOUd1

SPARC:

pam_pwdb crypt_md5():   $1$xx$tcVZsr28tzKG890dmISsL/
GLIBC crypt():          $1$xx$ipOG9IJsAo23Ba44.YOUd1

My God! - pam_pwdb crypt_md5() makes different hashes
depending on hardware platform.

Finally, I've got that the problem lies in an incorrect
detection of a byte order for the
architecture. Please, look at
pam-0.66/modules/pam_pwdb/md5.c:22, here is an
attempt to understand the byte order by checking the
presence of HIGHFIRST definition.  I had simply replaced
that string with

#if __BYTE_ORDER != __BIG_ENDIAN

and included an appropriate file endiand.h. Such changes I
had applied to pam-0.66/modules/pam_radius/md5.c and
pwdb-0.58/libpwdb/radius/md5.c too.  After rebuilding and
reinstalling packages everything works fine on both
Ultra SPARC and Intel."