Red Hat Bugzilla – Bug 83325
Updating from glibc-2.3.1-5 to glibc-2.3.1-38 broke qmail
Last modified: 2007-04-18 12:50:35 EDT
From Bugzilla Helper:User-Agent: Mozilla/4.78 (Linux 2.4.20-rc1-jam1 i686; U) Opera 6.11 [en] Description of problem:qmail no longer works after updating from glibc-2.3.1-5 to glibc-2.3.1-38. qmail has not been touched at all, and ldd /var/qmail/bin/qmail-smtpd says libc.so.6 => /lib/libc.so.6 (0x4002d000) /lib/ld-linux.so.2 => /lib/ld-linux.so.2 (0x40000000) so I would be pretty ceratin glibc upgrade broke it. Symptoms: > /var/qmail/bin/qmail-smtpd 421 unable to read controls (#4.3.0) it used to do this: > /var/qmail/bin/qmail-smtpd 220 v.iki.fi ESMTP<waits for input> Other qmail binaries are confused as well (give the same error). See the attached traces - qmail gives up after not find an optional control/ file. Version-Release number of selected component (if applicable): How reproducible: Always Steps to Reproduce: 1. execute any qmail binary 2. 3. Actual Results: > /var/qmail/bin/qmail-smtpd 421 unable to read controls (#4.3.0) Expected Results: > /var/qmail/bin/qmail-smtpd 220 v.iki.fi ESMTP <waits for input> Additional info: (working) strace with glibc-2.3.1-5 ----------------------------------------------------------------------------- execve("/var/qmail/bin/qmail-smtpd", ["/var/qmail/bin/qmail-smtpd"], [/* 48 vars */]) = 0 uname({sys="Linux", node="babbage", ...}) = 0 brk(0) = 0x804f45c old_mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x40015000 open("/etc/ld.so.preload", O_RDONLY) = -1 ENOENT (No such file or directory) open("/lib/libNoVersion.so.1", O_RDONLY) = -1 ENOENT (No such file or directory) open("/backup-versioned/2003-01-28T052001/lib/tls/i686/mmx/libc.so.6", O_RDONLY) = -1 ENOENT (No such file o r directory) stat64("/backup-versioned/2003-01-28T052001/lib/tls/i686/mmx", 0xbfffe610) = -1 ENOENT (No such file or dire ctory) open("/backup-versioned/2003-01-28T052001/lib/tls/i686/libc.so.6", O_RDONLY) = -1 ENOENT (No such file or di rectory) stat64("/backup-versioned/2003-01-28T052001/lib/tls/i686", 0xbfffe610) = -1 ENOENT (No such file or director y) open("/backup-versioned/2003-01-28T052001/lib/tls/mmx/libc.so.6", O_RDONLY) = -1 ENOENT (No such file or dir ectory) stat64("/backup-versioned/2003-01-28T052001/lib/tls/mmx", 0xbfffe610) = -1 ENOENT (No such file or directory ) open("/backup-versioned/2003-01-28T052001/lib/tls/libc.so.6", O_RDONLY) = -1 ENOENT (No such file or directo ry) stat64("/backup-versioned/2003-01-28T052001/lib/tls", 0xbfffe610) = -1 ENOENT (No such file or directory) open("/backup-versioned/2003-01-28T052001/lib/i686/mmx/libc.so.6", O_RDONLY) = -1 ENOENT (No such file or di rectory) stat64("/backup-versioned/2003-01-28T052001/lib/i686/mmx", 0xbfffe610) = -1 ENOENT (No such file or director y) open("/backup-versioned/2003-01-28T052001/lib/i686/libc.so.6", O_RDONLY) = 3 read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0\20[\1B"..., 1024) = 1024 close(3) = 0 stat64("/backup-versioned/2003-01-28T052001/lib/i686", {st_mode=S_IFDIR|0755, st_size=248, ...}) = 0 open("/backup-versioned/2003-01-28T052001/lib/mmx/libc.so.6", O_RDONLY) = -1 ENOENT (No such file or directo ry) stat64("/backup-versioned/2003-01-28T052001/lib/mmx", 0xbfffe610) = -1 ENOENT (No such file or directory) open("/backup-versioned/2003-01-28T052001/lib/libc.so.6", O_RDONLY) = 3 read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0P[\1\000"..., 1024) = 1024 fstat64(3, {st_mode=S_IFREG|0755, st_size=1210920, ...}) = 0 old_mmap(NULL, 1223588, PROT_READ|PROT_EXEC, MAP_PRIVATE, 3, 0) = 0x40016000 mprotect(0x40138000, 35748, PROT_NONE) = 0 old_mmap(0x40138000, 20480, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED, 3, 0x122000) = 0x40138000 old_mmap(0x4013d000, 15268, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x4013d000 close(3) = 0 rt_sigaction(SIGPIPE, {SIG_IGN}, NULL, 8) = 0 chdir("/var/qmail") = 0 open("control/me", O_RDONLY|O_NONBLOCK) = 3 read(3, "v.iki.fi\n", 64) = 9 close(3) = 0 open("control/smtpgreeting", O_RDONLY|O_NONBLOCK) = -1 ENOENT (No such file or directory) open("control/localiphost", O_RDONLY|O_NONBLOCK) = -1 ENOENT (No such file or directory) open("control/timeoutsmtpd", O_RDONLY|O_NONBLOCK) = -1 ENOENT (No such file or directory) open("control/rcpthosts", O_RDONLY|O_NONBLOCK) = 3 read(3, "v.iki.fi\nbabbage.netspan.fi\nbabb"..., 64) = 64 read(3, "\n\n", 64) = 2 read(3, "", 64) = 0 close(3) = 0 open("control/morercpthosts.cdb", O_RDONLY|O_NONBLOCK) = -1 ENOENT (No such file or directory) open("control/badmailfrom", O_RDONLY|O_NONBLOCK) = -1 ENOENT (No such file or directory) open("control/databytes", O_RDONLY|O_NONBLOCK) = -1 ENOENT (No such file or directory) socket(PF_INET, SOCK_STREAM, IPPROTO_IP) = 3 ioctl(3, 0x8912, 0xbffff188) = 0 ioctl(3, 0x8912, 0xbffff188) = 0 ioctl(3, 0x8913, 0x804e3b4) = 0 ioctl(3, 0x8915, 0x804e3b4) = 0 ioctl(3, 0x8913, 0x804e3d4) = 0 ioctl(3, 0x8915, 0x804e3d4) = 0 ioctl(3, 0x8913, 0x804e3f4) = 0 ioctl(3, 0x8915, 0x804e3f4) = 0 ioctl(3, 0x8913, 0x804e414) = 0 ioctl(3, 0x8913, 0x804e434) = 0 ioctl(3, 0x8915, 0x804e434) = 0 ioctl(3, 0x8913, 0x804e454) = 0 ioctl(3, 0x8915, 0x804e454) = 0 close(3) = 0 select(2, NULL, [1], NULL, {1200, 0}) = 1 (out [1], left {1200, 0}) write(1, "220 v.iki.fi ESMTP\r\n", 20) = 20 select(1, [0], NULL, NULL, {1200, 0} <unfinished ...> ----------------------------------------------------------------------------- (broken) strace with glibc-2.3.1-38 ----------------------------------------------------------------------------- execve("/var/qmail/bin/qmail-smtpd", ["/var/qmail/bin/qmail-smtpd"], [/* 48 vars */]) = 0 uname({sys="Linux", node="babbage", ...}) = 0 brk(0) = 0x804f45c old_mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x40015000 open("/etc/ld.so.preload", O_RDONLY) = -1 ENOENT (No such file or directory) open("/lib/libNoVersion.so.1", O_RDONLY) = -1 ENOENT (No such file or directory) open("/usr/local/lib/tls/i686/mmx/libc.so.6", O_RDONLY) = -1 ENOENT (No such file or directory) stat64("/usr/local/lib/tls/i686/mmx", 0xbfffe640) = -1 ENOENT (No such file or directory) open("/usr/local/lib/tls/i686/libc.so.6", O_RDONLY) = -1 ENOENT (No such file or directory) stat64("/usr/local/lib/tls/i686", 0xbfffe640) = -1 ENOENT (No such file or directory) open("/usr/local/lib/tls/mmx/libc.so.6", O_RDONLY) = -1 ENOENT (No such file or directory) stat64("/usr/local/lib/tls/mmx", 0xbfffe640) = -1 ENOENT (No such file or directory) open("/usr/local/lib/tls/libc.so.6", O_RDONLY) = -1 ENOENT (No such file or directory) stat64("/usr/local/lib/tls", 0xbfffe640) = -1 ENOENT (No such file or directory) open("/usr/local/lib/i686/mmx/libc.so.6", O_RDONLY) = -1 ENOENT (No such file or directory) stat64("/usr/local/lib/i686/mmx", 0xbfffe640) = -1 ENOENT (No such file or directory) open("/usr/local/lib/i686/libc.so.6", O_RDONLY) = -1 ENOENT (No such file or directory) stat64("/usr/local/lib/i686", 0xbfffe640) = -1 ENOENT (No such file or directory) open("/usr/local/lib/mmx/libc.so.6", O_RDONLY) = -1 ENOENT (No such file or directory) stat64("/usr/local/lib/mmx", 0xbfffe640) = -1 ENOENT (No such file or directory) open("/usr/local/lib/libc.so.6", O_RDONLY) = -1 ENOENT (No such file or directory) stat64("/usr/local/lib", {st_mode=S_IFDIR|0755, st_size=8192, ...}) = 0 open("/etc/ld.so.cache", O_RDONLY) = 3 fstat64(3, {st_mode=S_IFREG|0644, st_size=93157, ...}) = 0 old_mmap(NULL, 93157, PROT_READ, MAP_PRIVATE, 3, 0) = 0x40016000 close(3) = 0 open("/lib/libc.so.6", O_RDONLY) = 3 read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0 Y\1\000"..., 1024) = 1024 fstat64(3, {st_mode=S_IFREG|0755, st_size=1559788, ...}) = 0 old_mmap(NULL, 1231012, PROT_READ|PROT_EXEC, MAP_PRIVATE, 3, 0) = 0x4002d000 mprotect(0x40154000, 22692, PROT_NONE) = 0 old_mmap(0x40154000, 12288, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED, 3, 0x127000) = 0x40154000 old_mmap(0x40157000, 10404, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x40157000 close(3) = 0 munmap(0x40016000, 93157) = 0 rt_sigaction(SIGPIPE, {SIG_IGN}, NULL, 8) = 0 chdir("/var/qmail") = 0 open("control/me", O_RDONLY|O_NONBLOCK) = 3 read(3, "v.iki.fi\n", 64) = 9 close(3) = 0 open("control/smtpgreeting", O_RDONLY|O_NONBLOCK) = -1 ENOENT (No such file or directory) select(2, NULL, [1], NULL, {1200, 0}) = 1 (out [1], left {1200, 0}) write(1, "421 unable to read controls (#4."..., 38) = 38 semget(1, 134540800, 0x26|046) = -1 ENOSYS (Function not implemented) _exit(1) = ? -----------------------------------------------------------------------------
Created attachment 89781 [details] glibc-2.3.1-5 strace
Created attachment 89782 [details] glibc-2.3.1-38 strace
The straces don't reveal anything interesting, the only thing I wonder is why /lib/tls/libc.so.6 is not used. Do you have LD_ASSUME_KERNEL=2.2.5 in the environment, or do you run older kernel?
I didn't find anything conclusive in the straces either :(. LD_ASSUME_KERNEL is not set, and kernel is 2.4.20-rc1-jam1 (-jam patch set is based on -aa).
Is there anything I can/should try at this point? I could try to narrow down which glibc change broke qmail, but -5 to -38 is 33 releases... Do have any suspects? Is there an way to download all the releases in between from somewhere?
I'd start with booting a RH kernel if possible (so that you can see if the problem persists even with NPTL). And then I'd think best would be to actually look in the debugger where the code paths start to differ.
Created attachment 89889 [details] (Working) ltrace from glibc-2.3.1-5
Created attachment 89890 [details] (failing) ltrace from glibc-2.3.1-38
There are no differences in ltrace until the open("control/smtpgreeting", 2048, 027777772214) = -1 and then the program magically chooses to either call select (in the bad case), or open("control/localiphost", 2048, 027777772134) = -1 I'm afraid gdb or visual inspection of the code between opening smtpgreeting and localiphost is necessary.
Yes, I've been meaning to take a stab with gdb. (Sorry - I've been too busy for the two past days.) I was wondering about the 027777772214 vs 027777772134 difference, but that's not significant, I gather? I'll try to compile a debug version tonight, I'll let you know what it turns out. PS: glibc-2.3.1-38 -> 40 changes are probably not interesting, are they?
The third argument to open of course doesn't matter at all, the second argument is O_RDONLY|O_NONBLOCK, the third argument is only used with O_CREAT bit set. Given that there shouldn't be any difference between -5 and -38 either at least from the ltrace output, I think -38 -> -4{0,1} shouldn't be interesting at all.
Ok, it seems glibc-2.3.1-38 gives fd=-1,errno=0 from open() when the control/smtpgreeting does not exist - qmail expects fd=-1,errno=2(ENOENT), which is what glibc-2.3.1-5 gives. (Of course, qmail was and is compiled against much older glibc, but this doesn't seem right anyway.) Hold on, I'll provide more info shortly.
Created attachment 89893 [details] The relevant code path The (short) code path qmail-smtpd traverses before failure. I added some debug to it, and with glibc-2.3.1-38 it gives: root@babbage:/scratch>echo QUIT|./qmail-smtpd.DEBUG at control.c:64: control_readline(control/me): fd=3, errno=0 at control.c:66: expecting error_noent=2, ENOENT=2 at control.c:64: control_readline(control/smtpgreeting): fd=-1, errno=2 at control.c:66: expecting error_noent=2, ENOENT=2 at control.c:64: control_readline(control/localiphost): fd=-1, errno=2 at control.c:66: expecting error_noent=2, ENOENT=2 at control.c:64: control_readline(control/timeoutsmtpd): fd=-1, errno=2 at control.c:66: expecting error_noent=2, ENOENT=2 421 unable to read controls (#4.3.0) The same with glibc-2.3.1-5: root@babbage:/scratch>echo QUIT | LD_LIBRARY_PATH="/backup-versioned/2003-01-28 ./qmail-smtpd.DEBUG at control.c:64: control_readline(control/me): fd=3, errno=0 at control.c:66: expecting error_noent=2, ENOENT=2 at control.c:64: control_readline(control/smtpgreeting): fd=-1, errno=2 at control.c:66: expecting error_noent=2, ENOENT=2 at control.c:64: control_readline(control/localiphost): fd=-1, errno=2 at control.c:66: expecting error_noent=2, ENOENT=2 at control.c:64: control_readline(control/timeoutsmtpd): fd=-1, errno=2 at control.c:66: expecting error_noent=2, ENOENT=2 at control.c:64: control_readline(control/databytes): fd=-1, errno=2 at control.c:66: expecting error_noent=2, ENOENT=2 220 v.iki.fi ESMTP 221 v.iki.fi
Uhh, I was obviously wrong about the errno, sorry... Hold on.
Oh yes, I think I know what's going on. qmail is compiled against glibc 2.0.x and a binutils bug causes /lib/libc.so.6's errno to be updated, not the one in qmail's .dynbss.
Ok, the error I made with the attached debug output was that those fprintf's and compiling with -ggdb (instead of -O2) changed the way the program behaves. (And of course I was so eager about what I found that I forgot the double-check...) You explanation makes sense. But, I'm not quite sure what is the fix? ;) Do you need any further debug info?
Hmm, mkfs.vfat from dosfstools-2.8-1 (haven't upgraded yet) says this: Incorrectly built binary which accesses errno, h_errno or _res directly. Needs to be fixed. qmail doesn't... But I'm still a little lost as to what is the correct solution. Am I out of luck with my glibc-2.0 linked binaries (there still are quite a bunch of them) from now to eternity, or is there something that can be done?
Please try latest rawhide glibc (2.3.1-46 ATM).
With glibc-2.3.1-46, qmail seems to behave 100% right afaict from (really) short testing. If anything turns out in wider testing, I'll let you know. Thank you very much!
I came across this message from Phil Edwards: http://news.gmane.org/article.php?id=13960&group=gmane.mail.qmail.general which explains the breakage, and has a patch for qmail. Hopefully this is all moot now that glibc is fixed.