Bug 86339

This also affects connections to Sendmail, and any other service that does a
reverse DNS lookup?

Have you rebooted the server in between, or at least restarted all services
which use NSS lookup after they have been started (e.g. service sshd restart)
after the upgrade?

Yes, restarting the SSH service appears to allow the connection, and clears this
particular bug.

However, I used RHN to upgrade this server.   It is my understanding that the
glibc upgrade would have been applied automatically, if I had not run up2date
manually?

If so, this would have prevented me from connecting to the server remotely via
SHH in order to restart the service and fix the problem.

Maybe glibc should be on the pkgSkipList of up2date?   Or should the services
mentioned been restarted by up2date?

I had a an identical problem, but it was solved by doing 'service sshd restart'

Same with me - I had to physically walk to over a 100 machines in my dept and
restart sshd..  would have been nice to have been warned!  

Tim makes a good point.  Besides the suggestion for not installing
automatically, the errata page
(https://rhn.redhat.com/errata/RHSA-2003-089.html) should mention which services
you need to restart or that you should reboot the system after installing this
update.  Perhaps a separate bug should be filed against the errata page.

Associated bug filed against the errata page:
https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=86395

It should be possible to restart services via RHN (not just schedule a reboot). This applies even if the errata says what services to restart, since it may be inconvenient to do that semimanually for a big number of servers.

(Or maybe a pair of running shoes should be included in the RHN subscription? :-))

This appears to be a wide-reaching problem with symbol resolution in the new
glibc package and will break anything that e.g. hasn't already go the
libnss_dns.so loaded and tries to load it - it has the old glibc .so image, the
new libnss* images, and they conflict:

[pid 20911] open("/lib/libnss_dns.so.2", O_RDONLY) = 3
[pid 20911] read(3, "<snip>"..., 1024) = 1024
[pid 20911] fstat64(3, {st_mode=S_IFREG|0755, st_size=16700, ...}) = 0
[pid 20911] old_mmap(NULL, 12704, PROT_READ|PROT_EXEC, MAP_PRIVATE, 3, 0) =
[pid 20911] mprotect(0x40359000, 416, PROT_NONE) = 0
[pid 20911] old_mmap(0x40359000, <snip>) = 0x40359000
[pid 20911] close(3)                    = 0
[pid 20911] munmap(0x40345000, 67073)   = 0
[pid 20911] writev(2, [{"/usr/sbin/sshd", 14}, {": ", 2}, {"relocation error",
16}, {": ", 2}, {"/lib/libnss_dns.so.2", 20}, {": ", 2}, {"symbol
__libc_res_nquery, versio"..., 107}, {"", 0}, {"", 0}, {"\n", 1}], 10) = 164
 * 14 bytes in buffer 0
 | 00000  2f 75 73 72 2f 73 62 69  6e 2f 73 73 68 64        /usr/sbi n/sshd   |
 * 2 bytes in buffer 1
 | 00000  3a 20                                             :                 |
 * 16 bytes in buffer 2
 | 00000  72 65 6c 6f 63 61 74 69  6f 6e 20 65 72 72 6f 72  relocati on error |
 * 2 bytes in buffer 3
 | 00000  3a 20                                             :                 |
 * 20 bytes in buffer 4
 | 00000  2f 6c 69 62 2f 6c 69 62  6e 73 73 5f 64 6e 73 2e  /lib/lib nss_dns. |
 | 00010  73 6f 2e 32                                       so.2              |
 * 2 bytes in buffer 5
 | 00000  3a 20                                             :                 |
 * 107 bytes in buffer 6
 | 00000  73 79 6d 62 6f 6c 20 5f  5f 6c 69 62 63 5f 72 65  symbol _ _libc_re |
 | 00010  73 5f 6e 71 75 65 72 79  2c 20 76 65 72 73 69 6f  s_nquery , versio |
 | 00020  6e 20 47 4c 49 42 43 5f  50 52 49 56 41 54 45 20  n GLIBC_ PRIVATE  |
 | 00030  6e 6f 74 20 64 65 66 69  6e 65 64 20 69 6e 20 66  not defi ned in f |
 | 00040  69 6c 65 20 6c 69 62 72  65 73 6f 6c 76 2e 73 6f  ile libr esolv.so |
 | 00050  2e 32 20 77 69 74 68 20  6c 69 6e 6b 20 74 69 6d  .2 with  link tim |
 | 00060  65 20 72 65 66 65 72 65  6e 63 65                 e refere nce      |


Guys - do you even *test* these packages before you ship them?

I had similar problems with a PAM module we wrote here.  Odd thing is, the disconnect behavior went away if we started nscd.  It looks to us like somehow the gethostby* execution path gets hardwired to thinking nscd is available.  I haven't tried mass reboots yet.  

And rebooting fixes the problem. 
For what it's worth, if you do the glibc upgrade without a reboot, the following 
pam code snippet will fail (if nscd is not running).  It would be nice if 
checking for this could make it into the test suite somehow. 
 
#include <sys/cdefs.h> 
#include <stddef.h> 
#include <syslog.h> 
#include <netdb.h> 
 
#define	PAM_SM_AUTH 
 
#include <security/pam_appl.h> 
#include <security/pam_modules.h> 
 
PAM_EXTERN int 
pam_sm_authenticate(pam_handle_t *pamh, int flags, 
    int argc, const char *argv[]) 
{ 
    struct hostent *hp; 
 
    if ((hp = gethostbyname("www.redhat.com")) == NULL)  
        return (PAM_AUTH_ERR); 
    return (PAM_SUCCESS); 
} 
 
PAM_EXTERN int 
pam_sm_setcred(pam_handle_t *pamh, int flags, 
    int argc, const char *argv[]) 
{ 
	return (PAM_SUCCESS); 
} 
 
#ifdef PAM_STATIC 
struct pam_module _pam_brokencrud_modstruct = { 
    "pam_brokencrud", 
    pam_sm_authenticate, 
    pam_sm_setcred, 
    NULL, 
    NULL, 
    NULL, 
    NULL, 
}; 
#endif 
 

I've added sshd condrestart to ftp://people.redhat.com/jakub/glibc/errata/8.0/
I think sshd is special in that when sshd is not restarted and you don't have
physical access, you cannot restart the rest of services or reboot.
Problem with enlarging the list of services to restart is that it will never be
complete anyway, the only 100% way of finishing glibc upgrade is either
reboot, or telinit 1; telinit 5 (but neither of this can be done from glibc %post).
Otherwise some apps still have old copies of glibc and some new copies;
especially with security erratas that's something that should be avoided.

An errata has been issued which should help the problem described in this bug report. 
This report is therefore being closed with a resolution of ERRATA. For more information
on the solution and/or where to find the updated files, please follow the link below. You may reopen 
this bug report if the solution does not work for you.

http://rhn.redhat.com/errata/RHBA-2003-136.html