From Bugzilla Helper: User-Agent: Mozilla/5.0 (X11; U; Linux i686 (x86_64); en-US; rv:1.8.1.6) Gecko/20070725 Firefox/2.0.0.6 Description of problem: When dovecot is configured to use either imap or pop3, it tries to start imap-login or pop3-login helper modules. Both fail to load; failure is the same and look like Aug 18 01:59:34 taka dovecot: Dovecot v1.0.rc15 starting up Aug 18 01:59:35 taka dovecot: Temporary failure in creating login processes, slowing down for now Aug 18 01:59:35 taka dovecot: pop3-login: pop3-login: error while loading shared libraries: libsepol.so.1: failed to map segment from shared object: Cannot allocate memory Aug 18 01:59:35 taka dovecot: pop3-login: pop3-login: error while loading shared libraries: libsepol.so.1: failed to map segment from shared object: Cannot allocate memory Aug 18 01:59:35 taka dovecot: child 3645 (login) returned error 127 Aug 18 01:59:35 taka dovecot: child 3646 (login) returned error 127 Aug 18 01:59:35 taka dovecot: pop3-login: pop3-login: error while loading shared libraries: libsepol.so.1: failed to map segment from shared object: Cannot allocate memory Aug 18 01:59:35 taka dovecot: child 3647 (login) returned error 127 Aug 18 02:00:35 taka dovecot: pop3-login: pop3-login: error while loading shared libraries: libsepol.so.1: failed to map segment from shared object: Cannot allocate memory Aug 18 02:00:35 taka last message repeated 2 times The problem isn't selinux-related, setting it to permissive mode doesn't help. When replacing imap-login with script that straces actual imap-login, last messages in strace output are open("/lib64/libsepol.so.1", O_RDONLY) = 4 read(4, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\340<\0\010"..., 832) = 832 fstat(4, {st_mode=S_IFREG|0755, st_size=247528, ...}) = 0 mmap(0x3608000000, 2383168, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 4, 0) = -1 ENOMEM (Cannot allocate memory) close(4) = 0 writev(2, [{"/usr/libexec/dovecot/imap-login_", 32}, {": ", 2}, {"error while loading shared libra"..., 36}, {": ", 2}, {"libsepol.so.1", 13}, {": ", 2}, {"failed to map segment from share"..., 40}, {": ", 2}, {"Cannot allocate memory", 22}, {"\n", 1}], 10/usr/libexec/dovecot/imap-login_: error while loading shared libraries: libsepol.so.1: failed to map segment from shared object: Cannot allocate memory) = 152 exit_group(127) = ? Version-Release number of selected component (if applicable): dovecot-1.0-1.2.rc15.el5.x86_64 How reproducible: Always Steps to Reproduce: 1. Put "protocols = imap imaps" line in /etc/dovecot.conf (everything else can be left default) 2. Start dovecot service 3. Observe error in /var/log/maillog Actual Results: Expected Results: Additional info: Googling for this problem shows that it's either memory-related or limits-related. This machine has plenty amount of ram, from /proc/meminfo: MemTotal: 2059992 kB MemFree: 1769548 kB Buffers: 20220 kB Cached: 185412 kB SwapCached: 0 kB Active: 89916 kB Inactive: 162548 kB HighTotal: 0 kB HighFree: 0 kB LowTotal: 2059992 kB LowFree: 1769548 kB SwapTotal: 2129912 kB SwapFree: 2129912 kB Dirty: 36 kB Writeback: 0 kB AnonPages: 46848 kB Mapped: 13600 kB Slab: 20696 kB PageTables: 3820 kB NFS_Unstable: 0 kB Bounce: 0 kB CommitLimit: 3159908 kB Committed_AS: 141928 kB VmallocTotal: 34359738367 kB VmallocUsed: 1704 kB VmallocChunk: 34359736519 kB HugePages_Total: 0 HugePages_Free: 0 HugePages_Rsvd: 0 Hugepagesize: 2048 kB No changes in limits.conf, so memory limits for regular user are default: core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited max nice (-e) 0 file size (blocks, -f) unlimited pending signals (-i) 16383 max locked memory (kbytes, -l) 32 max memory size (kbytes, -m) unlimited open files (-n) 1024 pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 max rt priority (-r) 0 stack size (kbytes, -s) 10240 cpu time (seconds, -t) unlimited max user processes (-u) 16383 virtual memory (kbytes, -v) unlimited file locks (-x) unlimited For the record, env and ulimit -a output from inside imap-login wrapper script: export SSL_KEY_FILE=/etc/pki/dovecot/private/dovecot.pem export SSL_CERT_FILE=/etc/pki/dovecot/certs/dovecot.pem export RESTRICT_SETUID=97 export RESTRICT_SETGID=97 export SYSLOG_FACILITY=16 export CAPABILITY_STRING= export SSL_KEY_PASSWORD= export PROCESS_PER_CONNECTION=1 export LOG_TO_MASTER=1 export MAX_CONNECTIONS=1 export PWD=/var/run/dovecot export PROCESS_UID=5932 export GREETING="Dovecot ready." export DOVECOT_VERSION=1.0.rc15 export SHLVL=1 export RESTRICT_CHROOT=/var/run/dovecot/login export LOG_FORMAT="%$: %s" export SSL_PARAM_FILE=ssl-parameters.dat export DOVECOT_MASTER=1 export LOG_FORMAT_ELEMENTS=user="<%u> method=%m rip=%r lip=%l %c" core file size (blocks, -c) 0 data seg size (kbytes, -d) 32768 max nice (-e) 0 file size (blocks, -f) unlimited pending signals (-i) 16383 max locked memory (kbytes, -l) 32 max memory size (kbytes, -m) unlimited open files (-n) 1024 pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 max rt priority (-r) 0 stack size (kbytes, -s) 10240 cpu time (seconds, -t) unlimited max user processes (-u) 16383 virtual memory (kbytes, -v) 32768 file locks (-x) unlimited Maybe this problem is related to that 32mb virtual memory limit; however, I don't know who exactly sets it and how do I fix this.
Created attachment 161790 [details] Complete imap-login strace
Looking at dovecot source, found that setrlimit call which was messing everything up. It can be controlled through config, with login_process_size option. By default it's commented out, which means "login_process_size = 32". Everything works when uncommenting it and changing 32 to 64. Resulting VMM size of pop3-login and imap-login is just a bit over 32m, so looks like bloat from new versions of libraries triggered this. I think this option should be uncommented and changed in default config. Fedora might be affected, too.
Thanks for the report, good catch. I'll change it in Fedora and in the next update of dovecot in RHEL.
This request was evaluated by Red Hat Product Management for inclusion in a Red Hat Enterprise Linux maintenance release. Product Management has requested further review of this request by Red Hat Engineering, for potential inclusion in a Red Hat Enterprise Linux Update release for currently deployed products. This request is not yet committed for inclusion in an Update release.
The problem is that the default value is designed for 32bit systems. On 64bit systems the library mappings are greatly increased. From a 32bit system # pmap 12427 12427: imap-login 00110000 4K r-x-- [ anon ] 002ea000 148K r-x-- /usr/lib/libk5crypto.so.3.1 0030f000 4K rwx-- /usr/lib/libk5crypto.so.3.1 00312000 180K r-x-- /usr/lib/libgssapi_krb5.so.2.2 0033f000 4K rwx-- /usr/lib/libgssapi_krb5.so.2.2 0082d000 260K r-x-- /lib/libssl.so.0.9.8b 0086e000 16K rwx-- /lib/libssl.so.0.9.8b 0090f000 64K r-x-- /lib/libresolv-2.6.so 0091f000 4K r-x-- /lib/libresolv-2.6.so 00920000 4K rwx-- /lib/libresolv-2.6.so 00921000 8K rwx-- [ anon ] 00a48000 28K r-x-- /usr/lib/libkrb5support.so.0.1 00a4f000 4K rwx-- /usr/lib/libkrb5support.so.0.1 00ae6000 8K r-x-- /lib/libkeyutils-1.2.so 00ae8000 4K rwx-- /lib/libkeyutils-1.2.so 00afa000 108K r-x-- /lib/ld-2.6.so 00b15000 4K r-x-- /lib/ld-2.6.so 00b16000 4K rwx-- /lib/ld-2.6.so 00b19000 1336K r-x-- /lib/libc-2.6.so 00c67000 8K r-x-- /lib/libc-2.6.so 00c69000 4K rwx-- /lib/libc-2.6.so 00c6a000 12K rwx-- [ anon ] 00c9a000 12K r-x-- /lib/libdl-2.6.so 00c9d000 4K r-x-- /lib/libdl-2.6.so 00c9e000 4K rwx-- /lib/libdl-2.6.so 00ca1000 72K r-x-- /lib/libz.so.1.2.3 00cb3000 4K rwx-- /lib/libz.so.1.2.3 00d4f000 8K r-x-- /lib/libcom_err.so.2.1 00d51000 4K rwx-- /lib/libcom_err.so.2.1 07020000 576K r-x-- /usr/lib/libkrb5.so.3.3 070b0000 12K rwx-- /usr/lib/libkrb5.so.3.3 070d4000 1140K r-x-- /lib/libcrypto.so.0.9.8b 071f1000 72K rwx-- /lib/libcrypto.so.0.9.8b 07203000 16K rwx-- [ anon ] 08048000 132K r-x-- /usr/libexec/dovecot/imap-login 08069000 8K rw--- /usr/libexec/dovecot/imap-login 09457000 164K rw--- [ anon ] b7f01000 16K rw--- [ anon ] bffd2000 84K rw--- [ stack ] total 4544K which makes the 32M default not unreasonable given a reasonably heavy use. The problem is apparent on 64bit # pmap 6733 6733: imap-login 0000000000400000 132K r-x-- /usr/libexec/dovecot/imap-login 0000000000621000 4K rw--- /usr/libexec/dovecot/imap-login 0000003659200000 104K r-x-- /lib64/ld-2.5.so 0000003659419000 4K r---- /lib64/ld-2.5.so 000000365941a000 4K rw--- /lib64/ld-2.5.so 0000003659600000 1304K r-x-- /lib64/libc-2.5.so 0000003659746000 2048K ----- /lib64/libc-2.5.so 0000003659946000 16K r---- /lib64/libc-2.5.so 000000365994a000 4K rw--- /lib64/libc-2.5.so 000000365994b000 20K rw--- [ anon ] 0000003659e00000 8K r-x-- /lib64/libdl-2.5.so 0000003659e02000 2048K ----- /lib64/libdl-2.5.so 000000365a002000 4K r---- /lib64/libdl-2.5.so 000000365a003000 4K rw--- /lib64/libdl-2.5.so 000000365a600000 80K r-x-- /usr/lib64/libz.so.1.2.3 000000365a614000 2044K ----- /usr/lib64/libz.so.1.2.3 000000365a813000 4K rw--- /usr/lib64/libz.so.1.2.3 000000365f600000 236K r-x-- /lib64/libsepol.so.1 000000365f63b000 2048K ----- /lib64/libsepol.so.1 000000365f83b000 4K rw--- /lib64/libsepol.so.1 000000365f83c000 40K rw--- [ anon ] 000000365fa00000 84K r-x-- /lib64/libselinux.so.1 000000365fa15000 2048K ----- /lib64/libselinux.so.1 000000365fc15000 8K rw--- /lib64/libselinux.so.1 000000365fc17000 4K rw--- [ anon ] 0000003661200000 8K r-x-- /lib64/libcom_err.so.2.1 0000003661202000 2044K ----- /lib64/libcom_err.so.2.1 0000003661401000 4K rw--- /lib64/libcom_err.so.2.1 0000003661600000 1172K r-x-- /lib64/libcrypto.so.0.9.8b 0000003661725000 2044K ----- /lib64/libcrypto.so.0.9.8b 0000003661924000 124K rw--- /lib64/libcrypto.so.0.9.8b 0000003661943000 16K rw--- [ anon ] 0000003661e00000 68K r-x-- /lib64/libresolv-2.5.so 0000003661e11000 2048K ----- /lib64/libresolv-2.5.so 0000003662011000 4K r---- /lib64/libresolv-2.5.so 0000003662012000 4K rw--- /lib64/libresolv-2.5.so 0000003662013000 8K rw--- [ anon ] 0000003662600000 176K r-x-- /usr/lib64/libgssapi_krb5.so.2.2 000000366262c000 2048K ----- /usr/lib64/libgssapi_krb5.so.2.2 000000366282c000 8K rw--- /usr/lib64/libgssapi_krb5.so.2.2 0000003662e00000 568K r-x-- /usr/lib64/libkrb5.so.3.3 0000003662e8e000 2048K ----- /usr/lib64/libkrb5.so.3.3 000000366308e000 16K rw--- /usr/lib64/libkrb5.so.3.3 0000003663a00000 144K r-x-- /usr/lib64/libk5crypto.so.3.1 0000003663a24000 2044K ----- /usr/lib64/libk5crypto.so.3.1 0000003663c23000 8K rw--- /usr/lib64/libk5crypto.so.3.1 0000003663e00000 32K r-x-- /usr/lib64/libkrb5support.so.0.1 0000003663e08000 2044K ----- /usr/lib64/libkrb5support.so.0.1 0000003664007000 4K rw--- /usr/lib64/libkrb5support.so.0.1 0000003664200000 8K r-x-- /lib64/libkeyutils-1.2.so 0000003664202000 2044K ----- /lib64/libkeyutils-1.2.so 0000003664401000 4K rw--- /lib64/libkeyutils-1.2.so 0000003666000000 268K r-x-- /lib64/libssl.so.0.9.8b 0000003666043000 2048K ----- /lib64/libssl.so.0.9.8b 0000003666243000 24K rw--- /lib64/libssl.so.0.9.8b 00002aaaaaaab000 4K rw--- [ anon ] 00002aaaaaaca000 1052K rw--- [ anon ] 00007fffe3653000 84K rw--- [ stack ] ffffffffff600000 8192K ----- [ anon ] total 42716K and that's just after startup. The problem is compounded because the 2048K allocations would appear to be based on the amount of system RAM the particular machine has. On a smaller 64bit box programs get a 1024K allocation for the libraries. One possibility that might make sense is to use the configuration value as the amount that the process is allowed to grow over its initial size. For 32bit this would only increase the maximum size to 37M, and for 64bit would allow a similar sensible amount over whatever the program is after initial load.
FWIW, the problem still continues in RHEL 5.1 . I had to increase that setting in dovecot.conf to get rid of the problem. Regards, Devrim
It would have been nice if this problem had been fixed before releasing 5.1, noone could read mail after the automatic update this morning :-( :-( This is a 64bit machine with 5G RAM
please look thi: http://www.cyberciti.biz/tips/rhel-dovecot-error-while-loading-shared-libraries.html
*** Bug 372781 has been marked as a duplicate of this bug. ***
*** Bug 377641 has been marked as a duplicate of this bug. ***
http://dovecot.org/list/dovecot/2007-November/026633.html
*** Bug 427600 has been marked as a duplicate of this bug. ***
*** Bug 445597 has been marked as a duplicate of this bug. ***
An advisory has been issued which should help the problem described in this bug report. This report is therefore being closed with a resolution of ERRATA. For more information on the solution and/or where to find the updated files, please follow the link below. You may reopen this bug report if the solution does not work for you. http://rhn.redhat.com/errata/RHSA-2008-0297.html