Bug 253363 - Dovecot pop3-login/imap-login crash with OOM error
Dovecot pop3-login/imap-login crash with OOM error
Status: CLOSED ERRATA
Product: Red Hat Enterprise Linux 5
Classification: Red Hat
Component: dovecot (Show other bugs)
5.1
x86_64 Linux
medium Severity medium
: ---
: ---
Assigned To: Tomas Janousek
:
: 372781 377641 427600 445597 (view as bug list)
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2007-08-18 06:17 EDT by Vladimir Mosgalin
Modified: 2010-10-22 13:52 EDT (History)
9 users (show)

See Also:
Fixed In Version: RHSA-2008-0297
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2008-05-21 10:20:13 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
Complete imap-login strace (31.96 KB, text/plain)
2007-08-18 06:19 EDT, Vladimir Mosgalin
no flags Details

  None (edit)
Description Vladimir Mosgalin 2007-08-18 06:17:17 EDT
From Bugzilla Helper:
User-Agent: Mozilla/5.0 (X11; U; Linux i686 (x86_64); en-US; rv:1.8.1.6) Gecko/20070725 Firefox/2.0.0.6

Description of problem:
When dovecot is configured to use either imap or pop3, it tries to start imap-login or pop3-login helper modules. Both fail to load; failure is the same and look like

Aug 18 01:59:34 taka dovecot: Dovecot v1.0.rc15 starting up
Aug 18 01:59:35 taka dovecot: Temporary failure in creating login processes, slowing down for now
Aug 18 01:59:35 taka dovecot: pop3-login: pop3-login: error while loading shared libraries: libsepol.so.1: failed to map segment from shared object: Cannot allocate memory
Aug 18 01:59:35 taka dovecot: pop3-login: pop3-login: error while loading shared libraries: libsepol.so.1: failed to map segment from shared object: Cannot allocate memory
Aug 18 01:59:35 taka dovecot: child 3645 (login) returned error 127
Aug 18 01:59:35 taka dovecot: child 3646 (login) returned error 127
Aug 18 01:59:35 taka dovecot: pop3-login: pop3-login: error while loading shared libraries: libsepol.so.1: failed to map segment from shared object: Cannot allocate memory
Aug 18 01:59:35 taka dovecot: child 3647 (login) returned error 127
Aug 18 02:00:35 taka dovecot: pop3-login: pop3-login: error while loading shared libraries: libsepol.so.1: failed to map segment from shared object: Cannot allocate memory
Aug 18 02:00:35 taka last message repeated 2 times


The problem isn't selinux-related, setting it to permissive mode doesn't help. When replacing imap-login with script that straces actual imap-login, last messages in strace output are

open("/lib64/libsepol.so.1", O_RDONLY)  = 4
read(4, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\340<\0\010"..., 832) = 832
fstat(4, {st_mode=S_IFREG|0755, st_size=247528, ...}) = 0
mmap(0x3608000000, 2383168, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 4, 0) = -1 ENOMEM (Cannot allocate memory)
close(4)                                = 0
writev(2, [{"/usr/libexec/dovecot/imap-login_", 32}, {": ", 2}, {"error while loading shared libra"..., 36}, {": ", 2}, {"libsepol.so.1", 13}, {": ", 2}, {"failed to map segment from share"..., 40}, {": ", 2}, {"Cannot allocate memory", 22}, {"\n", 1}], 10/usr/libexec/dovecot/imap-login_: error while loading shared libraries: libsepol.so.1: failed to map segment from shared object: Cannot allocate memory) = 152
exit_group(127)                         = ?


Version-Release number of selected component (if applicable):
dovecot-1.0-1.2.rc15.el5.x86_64

How reproducible:
Always


Steps to Reproduce:
1. Put "protocols = imap imaps" line in /etc/dovecot.conf (everything else can be left default)
2. Start dovecot service
3. Observe error in /var/log/maillog

Actual Results:


Expected Results:


Additional info:
Googling for this problem shows that it's either memory-related or limits-related. This machine has plenty amount of ram, from /proc/meminfo:

MemTotal:      2059992 kB
MemFree:       1769548 kB
Buffers:         20220 kB
Cached:         185412 kB
SwapCached:          0 kB
Active:          89916 kB
Inactive:       162548 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:      2059992 kB
LowFree:       1769548 kB
SwapTotal:     2129912 kB
SwapFree:      2129912 kB
Dirty:              36 kB
Writeback:           0 kB
AnonPages:       46848 kB
Mapped:          13600 kB
Slab:            20696 kB
PageTables:       3820 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:   3159908 kB
Committed_AS:   141928 kB
VmallocTotal: 34359738367 kB
VmallocUsed:      1704 kB
VmallocChunk: 34359736519 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

No changes in limits.conf, so memory limits for regular user are default:
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
max nice                        (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 16383
max locked memory       (kbytes, -l) 32
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
max rt priority                 (-r) 0
stack size              (kbytes, -s) 10240
cpu time               (seconds, -t) unlimited
max user processes              (-u) 16383
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

For the record, env and ulimit -a output from inside imap-login wrapper script:
export SSL_KEY_FILE=/etc/pki/dovecot/private/dovecot.pem
export SSL_CERT_FILE=/etc/pki/dovecot/certs/dovecot.pem
export RESTRICT_SETUID=97
export RESTRICT_SETGID=97
export SYSLOG_FACILITY=16
export CAPABILITY_STRING=
export SSL_KEY_PASSWORD=
export PROCESS_PER_CONNECTION=1
export LOG_TO_MASTER=1
export MAX_CONNECTIONS=1
export PWD=/var/run/dovecot
export PROCESS_UID=5932
export GREETING="Dovecot ready."
export DOVECOT_VERSION=1.0.rc15
export SHLVL=1
export RESTRICT_CHROOT=/var/run/dovecot/login
export LOG_FORMAT="%$: %s"
export SSL_PARAM_FILE=ssl-parameters.dat
export DOVECOT_MASTER=1
export LOG_FORMAT_ELEMENTS=user="<%u> method=%m rip=%r lip=%l %c"

core file size          (blocks, -c) 0
data seg size           (kbytes, -d) 32768
max nice                        (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 16383
max locked memory       (kbytes, -l) 32
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
max rt priority                 (-r) 0
stack size              (kbytes, -s) 10240
cpu time               (seconds, -t) unlimited
max user processes              (-u) 16383
virtual memory          (kbytes, -v) 32768
file locks                      (-x) unlimited

Maybe this problem is related to that 32mb virtual memory limit; however, I don't know who exactly sets it and how do I fix this.
Comment 1 Vladimir Mosgalin 2007-08-18 06:19:49 EDT
Created attachment 161790 [details]
Complete imap-login strace
Comment 2 Vladimir Mosgalin 2007-08-20 10:32:35 EDT
Looking at dovecot source, found that setrlimit call which was messing
everything up. It can be controlled through config, with login_process_size
option. By default it's commented out, which means "login_process_size = 32".
Everything works when uncommenting it and changing 32 to 64. Resulting VMM size
of pop3-login and imap-login is just a bit over 32m, so looks like bloat from
new versions of libraries triggered this. I think this option should be
uncommented and changed in default config.

Fedora might be affected, too.
Comment 3 Tomas Janousek 2007-08-20 10:43:54 EDT
Thanks for the report, good catch. I'll change it in Fedora and in the next
update of dovecot in RHEL.
Comment 4 RHEL Product and Program Management 2007-10-31 14:24:52 EDT
This request was evaluated by Red Hat Product Management for inclusion in a Red
Hat Enterprise Linux maintenance release.  Product Management has requested
further review of this request by Red Hat Engineering, for potential
inclusion in a Red Hat Enterprise Linux Update release for currently deployed
products.  This request is not yet committed for inclusion in an Update
release.
Comment 5 Martin Poole 2007-11-07 07:53:36 EST
The problem is that the default value is designed for 32bit systems.

On 64bit systems the library mappings are greatly increased.

From a 32bit system

# pmap 12427

12427:   imap-login

00110000      4K r-x--    [ anon ]
002ea000    148K r-x--  /usr/lib/libk5crypto.so.3.1
0030f000      4K rwx--  /usr/lib/libk5crypto.so.3.1
00312000    180K r-x--  /usr/lib/libgssapi_krb5.so.2.2
0033f000      4K rwx--  /usr/lib/libgssapi_krb5.so.2.2
0082d000    260K r-x--  /lib/libssl.so.0.9.8b
0086e000     16K rwx--  /lib/libssl.so.0.9.8b
0090f000     64K r-x--  /lib/libresolv-2.6.so
0091f000      4K r-x--  /lib/libresolv-2.6.so
00920000      4K rwx--  /lib/libresolv-2.6.so
00921000      8K rwx--    [ anon ]
00a48000     28K r-x--  /usr/lib/libkrb5support.so.0.1
00a4f000      4K rwx--  /usr/lib/libkrb5support.so.0.1
00ae6000      8K r-x--  /lib/libkeyutils-1.2.so
00ae8000      4K rwx--  /lib/libkeyutils-1.2.so
00afa000    108K r-x--  /lib/ld-2.6.so
00b15000      4K r-x--  /lib/ld-2.6.so
00b16000      4K rwx--  /lib/ld-2.6.so
00b19000   1336K r-x--  /lib/libc-2.6.so
00c67000      8K r-x--  /lib/libc-2.6.so
00c69000      4K rwx--  /lib/libc-2.6.so
00c6a000     12K rwx--    [ anon ]
00c9a000     12K r-x--  /lib/libdl-2.6.so
00c9d000      4K r-x--  /lib/libdl-2.6.so
00c9e000      4K rwx--  /lib/libdl-2.6.so
00ca1000     72K r-x--  /lib/libz.so.1.2.3
00cb3000      4K rwx--  /lib/libz.so.1.2.3
00d4f000      8K r-x--  /lib/libcom_err.so.2.1
00d51000      4K rwx--  /lib/libcom_err.so.2.1
07020000    576K r-x--  /usr/lib/libkrb5.so.3.3
070b0000     12K rwx--  /usr/lib/libkrb5.so.3.3
070d4000   1140K r-x--  /lib/libcrypto.so.0.9.8b
071f1000     72K rwx--  /lib/libcrypto.so.0.9.8b
07203000     16K rwx--    [ anon ]
08048000    132K r-x--  /usr/libexec/dovecot/imap-login
08069000      8K rw---  /usr/libexec/dovecot/imap-login
09457000    164K rw---    [ anon ]
b7f01000     16K rw---    [ anon ]
bffd2000     84K rw---    [ stack ]
 total     4544K



which makes the 32M default not unreasonable given a reasonably heavy use.


The problem is apparent on 64bit 

# pmap 6733
6733:   imap-login
0000000000400000    132K r-x--  /usr/libexec/dovecot/imap-login
0000000000621000      4K rw---  /usr/libexec/dovecot/imap-login
0000003659200000    104K r-x--  /lib64/ld-2.5.so
0000003659419000      4K r----  /lib64/ld-2.5.so
000000365941a000      4K rw---  /lib64/ld-2.5.so
0000003659600000   1304K r-x--  /lib64/libc-2.5.so
0000003659746000   2048K -----  /lib64/libc-2.5.so
0000003659946000     16K r----  /lib64/libc-2.5.so
000000365994a000      4K rw---  /lib64/libc-2.5.so
000000365994b000     20K rw---    [ anon ]
0000003659e00000      8K r-x--  /lib64/libdl-2.5.so
0000003659e02000   2048K -----  /lib64/libdl-2.5.so
000000365a002000      4K r----  /lib64/libdl-2.5.so
000000365a003000      4K rw---  /lib64/libdl-2.5.so
000000365a600000     80K r-x--  /usr/lib64/libz.so.1.2.3
000000365a614000   2044K -----  /usr/lib64/libz.so.1.2.3
000000365a813000      4K rw---  /usr/lib64/libz.so.1.2.3
000000365f600000    236K r-x--  /lib64/libsepol.so.1
000000365f63b000   2048K -----  /lib64/libsepol.so.1
000000365f83b000      4K rw---  /lib64/libsepol.so.1
000000365f83c000     40K rw---    [ anon ]
000000365fa00000     84K r-x--  /lib64/libselinux.so.1
000000365fa15000   2048K -----  /lib64/libselinux.so.1
000000365fc15000      8K rw---  /lib64/libselinux.so.1
000000365fc17000      4K rw---    [ anon ]
0000003661200000      8K r-x--  /lib64/libcom_err.so.2.1
0000003661202000   2044K -----  /lib64/libcom_err.so.2.1
0000003661401000      4K rw---  /lib64/libcom_err.so.2.1
0000003661600000   1172K r-x--  /lib64/libcrypto.so.0.9.8b
0000003661725000   2044K -----  /lib64/libcrypto.so.0.9.8b
0000003661924000    124K rw---  /lib64/libcrypto.so.0.9.8b
0000003661943000     16K rw---    [ anon ]
0000003661e00000     68K r-x--  /lib64/libresolv-2.5.so
0000003661e11000   2048K -----  /lib64/libresolv-2.5.so
0000003662011000      4K r----  /lib64/libresolv-2.5.so
0000003662012000      4K rw---  /lib64/libresolv-2.5.so
0000003662013000      8K rw---    [ anon ]
0000003662600000    176K r-x--  /usr/lib64/libgssapi_krb5.so.2.2
000000366262c000   2048K -----  /usr/lib64/libgssapi_krb5.so.2.2
000000366282c000      8K rw---  /usr/lib64/libgssapi_krb5.so.2.2
0000003662e00000    568K r-x--  /usr/lib64/libkrb5.so.3.3
0000003662e8e000   2048K -----  /usr/lib64/libkrb5.so.3.3
000000366308e000     16K rw---  /usr/lib64/libkrb5.so.3.3
0000003663a00000    144K r-x--  /usr/lib64/libk5crypto.so.3.1
0000003663a24000   2044K -----  /usr/lib64/libk5crypto.so.3.1
0000003663c23000      8K rw---  /usr/lib64/libk5crypto.so.3.1
0000003663e00000     32K r-x--  /usr/lib64/libkrb5support.so.0.1
0000003663e08000   2044K -----  /usr/lib64/libkrb5support.so.0.1
0000003664007000      4K rw---  /usr/lib64/libkrb5support.so.0.1
0000003664200000      8K r-x--  /lib64/libkeyutils-1.2.so
0000003664202000   2044K -----  /lib64/libkeyutils-1.2.so
0000003664401000      4K rw---  /lib64/libkeyutils-1.2.so
0000003666000000    268K r-x--  /lib64/libssl.so.0.9.8b
0000003666043000   2048K -----  /lib64/libssl.so.0.9.8b
0000003666243000     24K rw---  /lib64/libssl.so.0.9.8b
00002aaaaaaab000      4K rw---    [ anon ]
00002aaaaaaca000   1052K rw---    [ anon ]
00007fffe3653000     84K rw---    [ stack ]
ffffffffff600000   8192K -----    [ anon ]
total            42716K

and that's just after startup.


The problem is compounded because the 2048K allocations would appear to be based
on the amount of system RAM the particular machine has.

On a smaller 64bit box programs get a 1024K allocation for the libraries.

One possibility that might make sense is to use the configuration value as the
amount that the process is allowed to grow over its initial size. For 32bit this
would only increase the maximum size to 37M, and for 64bit would allow a similar
sensible amount over whatever the program is after initial load.
Comment 6 Devrim GUNDUZ 2007-11-07 18:47:41 EST
FWIW, the problem still continues in RHEL 5.1 . I had to increase that setting
in dovecot.conf to get rid of the problem.

Regards, Devrim
Comment 7 Mogens Kjaer 2007-11-08 02:06:10 EST
It would have been nice if this problem had been fixed before releasing 5.1,
noone could read mail after the automatic update this morning :-( :-(

This is a 64bit machine with 5G RAM
Comment 8 Maciej Orzechowski 2007-11-09 06:01:04 EST
please look thi:
http://www.cyberciti.biz/tips/rhel-dovecot-error-while-loading-shared-libraries.html
Comment 9 Tomas Janousek 2007-11-09 10:05:37 EST
*** Bug 372781 has been marked as a duplicate of this bug. ***
Comment 10 Tomas Janousek 2007-11-12 11:30:53 EST
*** Bug 377641 has been marked as a duplicate of this bug. ***
Comment 16 Tomas Janousek 2008-01-05 07:22:14 EST
*** Bug 427600 has been marked as a duplicate of this bug. ***
Comment 19 Dan Horák 2008-05-08 02:17:03 EDT
*** Bug 445597 has been marked as a duplicate of this bug. ***
Comment 20 errata-xmlrpc 2008-05-21 10:20:13 EDT
An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on the solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHSA-2008-0297.html

Note You need to log in before you can comment on or make changes to this bug.