Bug 83325 - Updating from glibc-2.3.1-5 to glibc-2.3.1-38 broke qmail
Updating from glibc-2.3.1-5 to glibc-2.3.1-38 broke qmail
Status: CLOSED RAWHIDE
Product: Red Hat Raw Hide
Classification: Retired
Component: glibc (Show other bugs)
1.0
i686 Linux
medium Severity medium
: ---
: ---
Assigned To: Jakub Jelinek
Brian Brock
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2003-02-02 12:43 EST by Ville Herva
Modified: 2007-04-18 12:50 EDT (History)
0 users

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2003-02-18 05:42:30 EST
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:


Attachments (Terms of Use)
glibc-2.3.1-5 strace (4.41 KB, text/plain)
2003-02-02 12:44 EST, Ville Herva
no flags Details
glibc-2.3.1-38 strace (3.12 KB, text/plain)
2003-02-02 12:45 EST, Ville Herva
no flags Details
(Working) ltrace from glibc-2.3.1-5 (2.00 KB, text/plain)
2003-02-06 04:35 EST, Ville Herva
no flags Details
(failing) ltrace from glibc-2.3.1-38 (712 bytes, text/plain)
2003-02-06 04:35 EST, Ville Herva
no flags Details
The relevant code path (3.36 KB, text/plain)
2003-02-06 06:22 EST, Ville Herva
no flags Details

  None (edit)
Description Ville Herva 2003-02-02 12:43:55 EST
From Bugzilla Helper:User-Agent: Mozilla/4.78 (Linux 2.4.20-rc1-jam1 i686; U) Opera 6.11  [en] Description of problem:qmail no longer works after updating from glibc-2.3.1-5 to glibc-2.3.1-38.

qmail has not been touched at all, and
ldd /var/qmail/bin/qmail-smtpd says
        libc.so.6 => /lib/libc.so.6 (0x4002d000)
        /lib/ld-linux.so.2 => /lib/ld-linux.so.2 (0x40000000)
so I would be pretty ceratin glibc upgrade broke it.

Symptoms:
> /var/qmail/bin/qmail-smtpd
421 unable to read controls (#4.3.0)

it used to do this:
> /var/qmail/bin/qmail-smtpd
220 v.iki.fi ESMTP<waits for input>

Other qmail binaries are confused as well (give the same error).
See the attached traces - qmail gives up after not find an optional control/ file.

Version-Release number of selected component (if applicable):

How reproducible: Always
Steps to Reproduce:
1. execute any qmail binary
2.
3.   

Actual Results:  
> /var/qmail/bin/qmail-smtpd
421 unable to read controls (#4.3.0)

Expected Results:  
> /var/qmail/bin/qmail-smtpd
220 v.iki.fi ESMTP
<waits for input>

Additional info:
(working) strace with glibc-2.3.1-5
-----------------------------------------------------------------------------
execve("/var/qmail/bin/qmail-smtpd", ["/var/qmail/bin/qmail-smtpd"], [/* 48 vars */]) = 0
uname({sys="Linux", node="babbage", ...}) = 0
brk(0)                                  = 0x804f45c
old_mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x40015000
open("/etc/ld.so.preload", O_RDONLY)    = -1 ENOENT (No such file or directory)
open("/lib/libNoVersion.so.1", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/backup-versioned/2003-01-28T052001/lib/tls/i686/mmx/libc.so.6", O_RDONLY) = -1 ENOENT (No such file o
r directory)
stat64("/backup-versioned/2003-01-28T052001/lib/tls/i686/mmx", 0xbfffe610) = -1 ENOENT (No such file or dire
ctory)
open("/backup-versioned/2003-01-28T052001/lib/tls/i686/libc.so.6", O_RDONLY) = -1 ENOENT (No such file or di
rectory)
stat64("/backup-versioned/2003-01-28T052001/lib/tls/i686", 0xbfffe610) = -1 ENOENT (No such file or director
y)
open("/backup-versioned/2003-01-28T052001/lib/tls/mmx/libc.so.6", O_RDONLY) = -1 ENOENT (No such file or dir
ectory)
stat64("/backup-versioned/2003-01-28T052001/lib/tls/mmx", 0xbfffe610) = -1 ENOENT (No such file or directory
)
open("/backup-versioned/2003-01-28T052001/lib/tls/libc.so.6", O_RDONLY) = -1 ENOENT (No such file or directo
ry)
stat64("/backup-versioned/2003-01-28T052001/lib/tls", 0xbfffe610) = -1 ENOENT (No such file or directory)
open("/backup-versioned/2003-01-28T052001/lib/i686/mmx/libc.so.6", O_RDONLY) = -1 ENOENT (No such file or di
rectory)
stat64("/backup-versioned/2003-01-28T052001/lib/i686/mmx", 0xbfffe610) = -1 ENOENT (No such file or director
y)
open("/backup-versioned/2003-01-28T052001/lib/i686/libc.so.6", O_RDONLY) = 3
read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0\20[\1B"..., 1024) = 1024
close(3)                                = 0
stat64("/backup-versioned/2003-01-28T052001/lib/i686", {st_mode=S_IFDIR|0755, st_size=248, ...}) = 0
open("/backup-versioned/2003-01-28T052001/lib/mmx/libc.so.6", O_RDONLY) = -1 ENOENT (No such file or directo
ry)
stat64("/backup-versioned/2003-01-28T052001/lib/mmx", 0xbfffe610) = -1 ENOENT (No such file or directory)
open("/backup-versioned/2003-01-28T052001/lib/libc.so.6", O_RDONLY) = 3
read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0P[\1\000"..., 1024) = 1024
fstat64(3, {st_mode=S_IFREG|0755, st_size=1210920, ...}) = 0
old_mmap(NULL, 1223588, PROT_READ|PROT_EXEC, MAP_PRIVATE, 3, 0) = 0x40016000
mprotect(0x40138000, 35748, PROT_NONE)  = 0
old_mmap(0x40138000, 20480, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED, 3, 0x122000) = 0x40138000
old_mmap(0x4013d000, 15268, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x4013d000
close(3)                                = 0
rt_sigaction(SIGPIPE, {SIG_IGN}, NULL, 8) = 0
chdir("/var/qmail")                     = 0
open("control/me", O_RDONLY|O_NONBLOCK) = 3
read(3, "v.iki.fi\n", 64)               = 9
close(3)                                = 0
open("control/smtpgreeting", O_RDONLY|O_NONBLOCK) = -1 ENOENT (No such file or directory)
open("control/localiphost", O_RDONLY|O_NONBLOCK) = -1 ENOENT (No such file or directory)
open("control/timeoutsmtpd", O_RDONLY|O_NONBLOCK) = -1 ENOENT (No such file or directory)
open("control/rcpthosts", O_RDONLY|O_NONBLOCK) = 3
read(3, "v.iki.fi\nbabbage.netspan.fi\nbabb"..., 64) = 64
read(3, "\n\n", 64)                     = 2
read(3, "", 64)                         = 0
close(3)                                = 0
open("control/morercpthosts.cdb", O_RDONLY|O_NONBLOCK) = -1 ENOENT (No such file or directory)
open("control/badmailfrom", O_RDONLY|O_NONBLOCK) = -1 ENOENT (No such file or directory)
open("control/databytes", O_RDONLY|O_NONBLOCK) = -1 ENOENT (No such file or directory)
socket(PF_INET, SOCK_STREAM, IPPROTO_IP) = 3
ioctl(3, 0x8912, 0xbffff188)            = 0
ioctl(3, 0x8912, 0xbffff188)            = 0
ioctl(3, 0x8913, 0x804e3b4)             = 0
ioctl(3, 0x8915, 0x804e3b4)             = 0
ioctl(3, 0x8913, 0x804e3d4)             = 0
ioctl(3, 0x8915, 0x804e3d4)             = 0
ioctl(3, 0x8913, 0x804e3f4)             = 0
ioctl(3, 0x8915, 0x804e3f4)             = 0
ioctl(3, 0x8913, 0x804e414)             = 0
ioctl(3, 0x8913, 0x804e434)             = 0
ioctl(3, 0x8915, 0x804e434)             = 0
ioctl(3, 0x8913, 0x804e454)             = 0
ioctl(3, 0x8915, 0x804e454)             = 0
close(3)                                = 0
select(2, NULL, [1], NULL, {1200, 0})   = 1 (out [1], left {1200, 0})
write(1, "220 v.iki.fi ESMTP\r\n", 20)  = 20
select(1, [0], NULL, NULL, {1200, 0} <unfinished ...>

-----------------------------------------------------------------------------

(broken) strace with glibc-2.3.1-38
-----------------------------------------------------------------------------
execve("/var/qmail/bin/qmail-smtpd", ["/var/qmail/bin/qmail-smtpd"], [/* 48 vars */]) = 0
uname({sys="Linux", node="babbage", ...}) = 0
brk(0)                                  = 0x804f45c
old_mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x40015000
open("/etc/ld.so.preload", O_RDONLY)    = -1 ENOENT (No such file or directory)
open("/lib/libNoVersion.so.1", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/local/lib/tls/i686/mmx/libc.so.6", O_RDONLY) = -1 ENOENT (No such file or directory)
stat64("/usr/local/lib/tls/i686/mmx", 0xbfffe640) = -1 ENOENT (No such file or directory)
open("/usr/local/lib/tls/i686/libc.so.6", O_RDONLY) = -1 ENOENT (No such file or directory)
stat64("/usr/local/lib/tls/i686", 0xbfffe640) = -1 ENOENT (No such file or directory)
open("/usr/local/lib/tls/mmx/libc.so.6", O_RDONLY) = -1 ENOENT (No such file or directory)
stat64("/usr/local/lib/tls/mmx", 0xbfffe640) = -1 ENOENT (No such file or directory)
open("/usr/local/lib/tls/libc.so.6", O_RDONLY) = -1 ENOENT (No such file or directory)
stat64("/usr/local/lib/tls", 0xbfffe640) = -1 ENOENT (No such file or directory)
open("/usr/local/lib/i686/mmx/libc.so.6", O_RDONLY) = -1 ENOENT (No such file or directory)
stat64("/usr/local/lib/i686/mmx", 0xbfffe640) = -1 ENOENT (No such file or directory)
open("/usr/local/lib/i686/libc.so.6", O_RDONLY) = -1 ENOENT (No such file or directory)
stat64("/usr/local/lib/i686", 0xbfffe640) = -1 ENOENT (No such file or directory)
open("/usr/local/lib/mmx/libc.so.6", O_RDONLY) = -1 ENOENT (No such file or directory)
stat64("/usr/local/lib/mmx", 0xbfffe640) = -1 ENOENT (No such file or directory)
open("/usr/local/lib/libc.so.6", O_RDONLY) = -1 ENOENT (No such file or directory)
stat64("/usr/local/lib", {st_mode=S_IFDIR|0755, st_size=8192, ...}) = 0
open("/etc/ld.so.cache", O_RDONLY)      = 3
fstat64(3, {st_mode=S_IFREG|0644, st_size=93157, ...}) = 0
old_mmap(NULL, 93157, PROT_READ, MAP_PRIVATE, 3, 0) = 0x40016000
close(3)                                = 0
open("/lib/libc.so.6", O_RDONLY)        = 3
read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0 Y\1\000"..., 1024) = 1024
fstat64(3, {st_mode=S_IFREG|0755, st_size=1559788, ...}) = 0
old_mmap(NULL, 1231012, PROT_READ|PROT_EXEC, MAP_PRIVATE, 3, 0) = 0x4002d000
mprotect(0x40154000, 22692, PROT_NONE)  = 0
old_mmap(0x40154000, 12288, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED, 3, 0x127000) = 0x40154000
old_mmap(0x40157000, 10404, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x40157000
close(3)                                = 0
munmap(0x40016000, 93157)               = 0
rt_sigaction(SIGPIPE, {SIG_IGN}, NULL, 8) = 0
chdir("/var/qmail")                     = 0
open("control/me", O_RDONLY|O_NONBLOCK) = 3
read(3, "v.iki.fi\n", 64)               = 9
close(3)                                = 0
open("control/smtpgreeting", O_RDONLY|O_NONBLOCK) = -1 ENOENT (No such file or directory)
select(2, NULL, [1], NULL, {1200, 0})   = 1 (out [1], left {1200, 0})
write(1, "421 unable to read controls (#4."..., 38) = 38
semget(1, 134540800, 0x26|046)          = -1 ENOSYS (Function not implemented)
_exit(1)                                = ?

-----------------------------------------------------------------------------
Comment 1 Ville Herva 2003-02-02 12:44:48 EST
Created attachment 89781 [details]
glibc-2.3.1-5 strace
Comment 2 Ville Herva 2003-02-02 12:45:15 EST
Created attachment 89782 [details]
glibc-2.3.1-38 strace
Comment 3 Jakub Jelinek 2003-02-03 06:31:55 EST
The straces don't reveal anything interesting, the only thing I wonder
is why /lib/tls/libc.so.6 is not used. Do you have LD_ASSUME_KERNEL=2.2.5
in the environment, or do you run older kernel?
Comment 4 Ville Herva 2003-02-03 06:40:02 EST
I didn't find anything conclusive in the straces either :(.




LD_ASSUME_KERNEL is not set, and kernel is 2.4.20-rc1-jam1 (-jam patch set is 
based on -aa).


Comment 5 Ville Herva 2003-02-03 06:50:24 EST
Is there anything I can/should try at this point? 




I could try to narrow down which glibc change broke qmail, but -5 to -38 is 33 
releases... Do have any suspects? Is there an way to download all the releases 
in between from somewhere?
Comment 6 Jakub Jelinek 2003-02-03 10:03:34 EST
I'd start with booting a RH kernel if possible (so that you can see if
the problem persists even with NPTL).
And then I'd think best would be to actually look in the debugger where
the code paths start to differ.
Comment 7 Ville Herva 2003-02-06 04:35:02 EST
Created attachment 89889 [details]
(Working) ltrace from glibc-2.3.1-5
Comment 8 Ville Herva 2003-02-06 04:35:34 EST
Created attachment 89890 [details]
(failing) ltrace from glibc-2.3.1-38
Comment 9 Jakub Jelinek 2003-02-06 04:46:00 EST
There are no differences in ltrace until the
open("control/smtpgreeting", 2048, 027777772214) = -1
and then the program magically chooses to either call select (in the bad case),
or open("control/localiphost", 2048, 027777772134)  = -1
I'm afraid gdb or visual inspection of the code between opening smtpgreeting
and localiphost is necessary.
Comment 10 Ville Herva 2003-02-06 05:23:46 EST
Yes, I've been meaning to take a stab with gdb. (Sorry - I've been too busy for the two past days.) I was wondering about the 027777772214 vs 027777772134 difference, but that's not significant, I gather?

I'll try to compile a debug version tonight, I'll let you know what it turns out.

PS: glibc-2.3.1-38 -> 40 changes are probably not interesting, are they?
Comment 11 Jakub Jelinek 2003-02-06 06:05:13 EST
The third argument to open of course doesn't matter at all, the second argument
is O_RDONLY|O_NONBLOCK, the third argument is only used with O_CREAT bit set.
Given that there shouldn't be any difference between -5 and -38 either
at least from the ltrace output, I think -38 -> -4{0,1} shouldn't be interesting at all.
Comment 12 Ville Herva 2003-02-06 06:12:21 EST
Ok, it seems glibc-2.3.1-38 gives fd=-1,errno=0 from open() when the control/smtpgreeting does not exist - qmail expects fd=-1,errno=2(ENOENT), which is what glibc-2.3.1-5 gives.
(Of course, qmail was and is compiled against much older glibc, but this doesn't seem right anyway.)

Hold on, I'll provide more info shortly.
Comment 13 Ville Herva 2003-02-06 06:22:02 EST
Created attachment 89893 [details]
The relevant code path

The (short) code path qmail-smtpd traverses before failure.

I added some debug to it, and with glibc-2.3.1-38 it gives:
root@babbage:/scratch>echo QUIT|./qmail-smtpd.DEBUG
at control.c:64: control_readline(control/me): fd=3, errno=0
at control.c:66: expecting error_noent=2, ENOENT=2
at control.c:64: control_readline(control/smtpgreeting): fd=-1, errno=2
at control.c:66: expecting error_noent=2, ENOENT=2
at control.c:64: control_readline(control/localiphost): fd=-1, errno=2
at control.c:66: expecting error_noent=2, ENOENT=2
at control.c:64: control_readline(control/timeoutsmtpd): fd=-1, errno=2
at control.c:66: expecting error_noent=2, ENOENT=2
421 unable to read controls (#4.3.0)

The same with glibc-2.3.1-5:
root@babbage:/scratch>echo QUIT | LD_LIBRARY_PATH="/backup-versioned/2003-01-28

./qmail-smtpd.DEBUG
at control.c:64: control_readline(control/me): fd=3, errno=0
at control.c:66: expecting error_noent=2, ENOENT=2
at control.c:64: control_readline(control/smtpgreeting): fd=-1, errno=2
at control.c:66: expecting error_noent=2, ENOENT=2
at control.c:64: control_readline(control/localiphost): fd=-1, errno=2
at control.c:66: expecting error_noent=2, ENOENT=2
at control.c:64: control_readline(control/timeoutsmtpd): fd=-1, errno=2
at control.c:66: expecting error_noent=2, ENOENT=2
at control.c:64: control_readline(control/databytes): fd=-1, errno=2
at control.c:66: expecting error_noent=2, ENOENT=2
220 v.iki.fi ESMTP
221 v.iki.fi
Comment 14 Ville Herva 2003-02-06 06:25:30 EST
Uhh, I was obviously wrong about the errno, sorry... Hold on.
Comment 15 Jakub Jelinek 2003-02-06 06:28:17 EST
Oh yes, I think I know what's going on.
qmail is compiled against glibc 2.0.x and a binutils bug causes /lib/libc.so.6's
errno to be updated, not the one in qmail's .dynbss.
Comment 16 Ville Herva 2003-02-06 06:37:24 EST
Ok, the error I made with the attached debug output was that those fprintf's and compiling with -ggdb (instead of -O2) changed the way the program behaves. (And of course I was so eager about what I found that I forgot the double-check...)

You explanation makes sense. But, I'm not quite sure what is the fix? ;) 

Do you need any further debug info?
Comment 17 Ville Herva 2003-02-07 08:43:17 EST
Hmm, mkfs.vfat from dosfstools-2.8-1 (haven't upgraded yet) says this:




Incorrectly built binary which accesses errno, h_errno or _res directly. Needs 
to be fixed.




qmail doesn't...




But I'm still a little lost as to what is the correct solution. Am I out of luck 
with my glibc-2.0 linked binaries (there still are quite a bunch of them) from 
now to eternity, or is there something that can be done?
Comment 18 Jakub Jelinek 2003-02-17 09:37:42 EST
Please try latest rawhide glibc (2.3.1-46 ATM).
Comment 19 Ville Herva 2003-02-17 16:07:20 EST
With glibc-2.3.1-46, qmail seems to behave 100% right afaict from (really) short testing. If anything turns out in wider testing, I'll let you know.

Thank you very much!
Comment 20 Ville Herva 2003-02-23 16:51:56 EST
I came across this message from Phil Edwards:

http://news.gmane.org/article.php?id=13960&group=gmane.mail.qmail.general

which explains the breakage, and has a patch for qmail.

Hopefully this is all moot now that glibc is fixed.

Note You need to log in before you can comment on or make changes to this bug.