There's a bug in pam_oddjob_mkhomedir.so: it's checking errno for ERANGE after getpwnam_r() instead of checking the result code directly.  There's also apparently a bug in libnss_files, and depending on how it's fixed (libc used to both set errno and return the error directly, so oddjob ended up doing the right thing despite it interpreting the result incorrectly), we may not end up needing to update this package.

We're updating the package to make other changes anyway, so fixing it here lets us avoid depending on #1099235's schedule.

Created attachment 970998 [details]
a test program for getpwnam_r()

Nalin, unfortunately, your patch isn't enough to fix this problem, because at least on RHEL7, getpwent_r() does not reliably return ERANGE when the buffer size is too small.

I cobbled together the attached test program from the getpwent_r() man page and pam_oddjob_mkhomedir.c. If I compile and run it on some of the new RHEL7 systems we're building, I get these results:

$ ./getpwnam_r 4 root
bufsize = 4, s = 0, pwd = 0x7fff4a5116d0, result = (nil)
Not found

$ ./getpwnam_r 14 root
bufsize = 14, s = 0, pwd = 0x7fff812ce3e0, result = 0x7fff812ce3e0
Name: ro; UID: 0

$ ./getpwnam_r 1024 root
bufsize = 1024, s = 0, pwd = 0x7fff4ad1ecc0, result = 0x7fff4ad1ecc0
Name: root; UID: 0

From these results, if the buffer size is below a certain minimum, getpwnam_r() returns the same results as it would if it were looking for a non-existent user. Increasing the buffer size more yields a successful result—but the results are truncated. Using a much larger buffer the (the sysconf(_SC_GETPW_R_SIZE_MAX) value, actually) yields successful results.

On my Fedora 20 system, getpwnam_r() appears to behave as its man page documents:

$ ./getpwnam_r 2 root
bufsize = 2, s = 34, pwd = 0x7fff4cdda750, result = (nil)
bufsize = 4, s = 34, pwd = 0x7fff4cdda750, result = (nil)
bufsize = 8, s = 34, pwd = 0x7fff4cdda750, result = (nil)
bufsize = 16, s = 34, pwd = 0x7fff4cdda750, result = (nil)
bufsize = 32, s = 34, pwd = 0x7fff4cdda750, result = (nil)
bufsize = 64, s = 0, pwd = 0x7fff4cdda750, result = 0x7fff4cdda750
Name: root; UID: 0

But the RHEL7 getpwnam_r() doesn't seem to actually function the way its man page describes.

I'm open to the possibility that 1) my test program is incorrect, or 2) something is messed up on our RHEL7 systems that is breaking getpwnam_r(). But if #1 is the case, I'm missing it, and I don't have the foggiest idea what #2 would be.

Created attachment 971010 [details]
fix pam_oddjob_mkhomedir.so getpwnam_r() handling

Here's a patch I created for pam_oddjob_mkhomedir.so that should squash this bug, regardless of whether getpwnam_r() returns ERANGE.  As per the example code in the getpwnam_r() man page, it selects the initial buffer size by calling sysconf(_SC_GETPW_R_SIZE_MAX); if that fails, it defaults to 1024 (which is the _SC_GETPW_R_SIZE_MAX value on Fedora 20). The size of the buffer is doubled every time ERANGE is returned, but this logic should virtually never be necessary.

Additionally, the patch adds support for passing a single option to pam_oddjob_mkhomedir.so, "debug", that enables sending debugging messages to syslog.

The problem with ERANGE is addressed in bug 1099235.

Is there any progress on getting a patched version of oddjob out there? Just deployed in a large environment, and one thing that did not appear during testing due to already existing homedirectories was this being broken.

Now we have to manually create home directories for all our roaming active directory users on new machines they start working on.

The update with the part of the fix that needs to be done in oddjob should already be present in the 7.1 beta, though we're also depending on bug #1099235 being fixed in glibc's nss_files module for cases where the user information's coming from /etc/passwd.

Nalin, I don't understand your thinking here.

As you observed, the updated oddjob in RHEL 7.1 beta won't fix this problem in the case where the user information is coming from /etc/passwd, due to bug 1099235.

But having user information in /etc/passwd isn't an esoteric corner case; it's actually a fairly common case. So for a significant number of customers, your patch won't fix the problem.

Even worse, there's been little movement on bug 1099235. It sat for almost 10 months before anyone even looked at it. How much longer is it going to sit before someone actually resolves it? Months? Years?

The patch I provided in comment 8 not only fixes the bug with oddjob, but works around bug 1099235 in glibc.

Yes, bug 1099235 in glibc should be fixed. But what is the reason for making Red Hat customers suffer with a broken oddjob (when you KNOW it is broken!) until that happens?

We (and I dare say, most other Red Hat) are care about software purity only obliquely. Primarily, we just want our sh*t not to break. And right now, your oddjob patch still breaks our sh*t.

I can confirm that oddjob-mkhomedir is still broken for redhat 7.1beta,redhat7.0 fedora21 . (4mar15) 

-/Don

--
: root@petrol1 var; rpm -q oddjob-mkhomedir oddjob
oddjob-mkhomedir-0.31.5-4.el7.x86_64
oddjob-0.31.5-4.el7.x86_64
: root@petrol1 var; lsb_release -a
LSB Version:	:core-4.1-amd64:core-4.1-noarch:cxx-4.1-amd64:cxx-4.1-noarch:desktop-4.1-amd64:desktop-4.1-noarch:languages-4.1-amd64:languages-4.1-noarch:printing-4.1-amd64:printing-4.1-noarch
Distributor ID:	RedHatEnterpriseServer
Description:	Red Hat Enterprise Linux Server release 7.1 Beta (Maipo)
Release:	7.1
Codename:	Maipo
: root@petrol1 var; ssh -X none@petrol1

none@petrol1's password: 
Last login: Wed Mar  4 13:07:19 2015 from petrol1.utdallas.edu
Welcome to petrol1.utdallas.edu redhat_7 x86_64
Could not chdir to home directory /var/none: No such file or directory
: none@petrol1 /; 
: none@petrol1 /; /usr/lib64/libc.so.6 
GNU C Library (GNU libc) stable release version 2.17, by Roland McGrath et al.
Copyright (C) 2012 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.
There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A
PARTICULAR PURPOSE.
Compiled by GNU CC version 4.8.3 20140911 (Red Hat 4.8.3-7).
Compiled on a Linux 3.10.0 system on 2014-11-12.
Available extensions:
	The C stubs add-on version 2.1.2.
	crypt add-on version 2.1 by Michael Glad and others
	GNU Libidn by Simon Josefsson
	Native POSIX Threads Library by Ulrich Drepper et al
	BIND-8.2.3-T5B
	RT using linux kernel aio
libc ABIs: UNIQUE IFUNC
For bug reporting instructions, please see:
<http://www.gnu.org/software/libc/bugs.html>.
: none@zul ~; grep -i mkhome /etc/sysconfig/authconfig
USEMKHOMEDIR=yes

---

sh -X none@zul

none@zul's password: 
Last login: Wed Mar  4 13:14:38 2015 from zul.utdallas.edu
Could not chdir to home directory /var/none: No such file or directory
/usr/bin/xauth:  error in locking authority file /var/none/.Xauthority
: none@zul /; lsb_release -a
LSB Version:	:core-4.1-amd64:core-4.1-noarch:cxx-4.1-amd64:cxx-4.1-noarch:desktop-4.1-amd64:desktop-4.1-noarch:languages-4.1-amd64:languages-4.1-noarch:printing-4.1-amd64:printing-4.1-noarch
Distributor ID:	Fedora
Description:	Fedora release 21 (Twenty One)
Release:	21
Codename:	TwentyOne
: none@zul /; rpm -q oddjob oddjob-mkhomedir 
oddjob-0.33-3.fc21.x86_64
oddjob-mkhomedir-0.33-3.fc21.x86_64
: none@zul /; /usr/lib64/libc.so.6 
GNU C Library (GNU libc) stable release version 2.20, by Roland McGrath et al.
Copyright (C) 2014 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.
There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A
PARTICULAR PURPOSE.
Compiled by GNU CC version 4.9.2 20141101 (Red Hat 4.9.2-1).
Available extensions:
	The C stubs add-on version 2.1.2.
	crypt add-on version 2.1 by Michael Glad and others
	GNU Libidn by Simon Josefsson
	Native POSIX Threads Library by Ulrich Drepper et al
	BIND-8.2.3-T5B
	RT using linux kernel aio
libc ABIs: UNIQUE IFUNC
For bug reporting instructions, please see:
<http://www.gnu.org/software/libc/bugs.html>.
: none@zul /; pwd
/
: none@petrol1 /; grep MKHOME /etc/sysconfig/authconfig
USEMKHOMEDIR=yes

===========
 su - none
Last login: Mon Oct 27 12:51:22 CDT 2014 from zul.utdallas.edu on pts/1
su: warning: cannot change directory to /var/none: No such file or directory
mkdir: cannot create directory '/var/none': Permission denied
cannot find home directory /var/none
: none@ion root; exit
logout
: root@ion ~#; lsb_release -a
LSB Version:	:core-4.1-amd64:core-4.1-noarch:cxx-4.1-amd64:cxx-4.1-noarch:desktop-4.1-amd64:desktop-4.1-noarch:languages-4.1-amd64:languages-4.1-noarch:printing-4.1-amd64:printing-4.1-noarch
Distributor ID:	RedHatEnterpriseServer
Description:	Red Hat Enterprise Linux Server release 7.0 (Maipo)
Release:	7.0
Codename:	Maipo

(In reply to James Ralston from comment #13)
> Nalin, I don't understand your thinking here.

At the time, I thought we'd be fixing the bits that needed to be fixed in libc as well, so guessing the right buffer size to start with wouldn't have ended up mattering, and the fix that was already made in the upstream source repository would have sufficed.  Without the libc change, we would only have been able to hope (to be fair, most of the time, correctly) that a newly-raised initial buffer size was large enough.  The rest of the time, we'd continue to see failures.

> But having user information in /etc/passwd isn't an esoteric corner case;
> it's actually a fairly common case. So for a significant number of
> customers, your patch won't fix the problem.

If I've given the impression that I consider nss_files to be a corner case, I apologize.  I don't consider it one.

Hello Don,
according to 
https://bugzilla.redhat.com/show_bug.cgi?id=1099235#c3
the glibc fix should be present in F21. Is there any chance you have tested it with an older version?

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2015-0446.html

Yes- I tested f21  at home, and this time there was a success!
Funny - I tested this scenario yesterday and got a fail. I
assume there was an update that did the trick?

-/don


: root@frs donmoore; lsb_release -a
LSB Version:	:core-4.1-amd64:core-4.1-noarch:cxx-4.1-amd64:cxx-4.1-noarch:desktop-4.1-amd64:desktop-4.1-noarch:languages-4.1-amd64:languages-4.1-noarch:printing-4.1-amd64:printing-4.1-noarch
Distributor ID:	Fedora
Description:	Fedora release 21 (Twenty One)
Release:	21
Codename:	TwentyOne
: root@frs donmoore; grep MKHOME /etc/sysconfig/authconfig
USEMKHOMEDIR=yes
: root@frs donmoore; grep none /etc/passwd
none:x:200:1225:none:/var/none:/bin/bash
: root@frs donmoore; ls -ld /var/none
ls: cannot access /var/none: No such file or directory
: root@frs donmoore; ssh -X none@frs
WARNING : Unauthorized access to this system is forbidden and will be
prosecuted by law. By accessing this system, you agree that your actions
may be monitored if unauthorized usage is suspected.
none@frs's password: 
Creating directory '/var/none'.
Last failed login: Thu Mar  5 08:55:36 CST 2015 from frs on ssh:notty
There were 3 failed login attempts since the last successful login.
Last login: Thu Mar  5 08:52:36 2015 from frs
/usr/bin/xauth:  file /var/none/.Xauthority does not exist
: none@frs ~; pwd
/var/none
: none@frs ~; ls -ld $HOME
drwx------ 1 none none 178 Mar  5 08:56 /var/none
: none@frs ~; date
Thu Mar  5 08:56:35 CST 2015
: none@frs ~; /usr/lib64/libc.so.6 
GNU C Library (GNU libc) stable release version 2.20, by Roland McGrath et al.
Copyright (C) 2014 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.
There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A
PARTICULAR PURPOSE.
Compiled by GNU CC version 4.9.2 20150212 (Red Hat 4.9.2-6).
Available extensions:
	The C stubs add-on version 2.1.2.
	crypt add-on version 2.1 by Michael Glad and others
	GNU Libidn by Simon Josefsson
	Native POSIX Threads Library by Ulrich Drepper et al
	BIND-8.2.3-T5B
	RT using linux kernel aio
libc ABIs: UNIQUE IFUNC
For bug reporting instructions, please see:
<http://www.gnu.org/software/libc/bugs.html>.
: none@frs ~;