Bug 253854

Summary: cyrus-imapd leaks filedescriptors at folder rename and delete
Product: Red Hat Enterprise Linux 5 Reporter: Radek Vokál <rvokal>
Component: cyrus-imapdAssignee: Michal Hlavinka <mhlavink>
Status: CLOSED ERRATA QA Contact: Brian Brock <bbrock>
Severity: medium Docs Contact:
Priority: medium    
Version: 5.0CC: azelinka, mpoole, ovasik, schlichting
Target Milestone: ---   
Target Release: ---   
Hardware: i386   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2011-07-21 12:19:10 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 232047    
Bug Blocks:    
Attachments:
Description Flags
perl reproducer
none
Other reproducer none

Description Tomas Janousek 2007-08-22 13:21:16 UTC
+++ This bug was initially created as a clone of Bug #232047 +++

Description of problem:
If a user or an administrator renames or deletes a mailbox folder with a quota
limit the imapd process keeps an filedescriptor open for each folder and all
contained subfolders.
So if you rename user/account/a/a/a into user/account/b/a/a 3 filedescriptors
are lost. Users with big folder hierarchy run out of filedescriptor when
renaming top level folders. If an administrator moves users between different
partitions (renm user/account user/account newpartition), things get more worse
because more descriptors seems to leak out and the command aborts quite soon
leaving mailboxes spread between partitions behind.   

Version-Release number of selected component (if applicable):

cyrus-imapd-2.2.12-3.RHEL4.1

How reproducible:

Make sure that a user "rztths" exists and has a quota on his inbox. Then do:
cyradm localhost --user cyrus
IMAP Password:
localhost.localdomain> lq user/rztths
 STORAGE 392087/512000 (76.5794921875%)
localhost.localdomain> cm user/rztths/a
localhost.localdomain> cm user/rztths/a/a
localhost.localdomain> cm user/rztths/a/a/a
localhost.localdomain> lsof|grep localhost|grep ESTABLISHED
mycyradm  14245   root    3u     IPv4  372137852                  TCP
localhost.localdomain:47203->localhost.localdomain:imap (ESTABLISHED)
imapd.old 14247  cyrus    0u     IPv4  372137855                  TCP
localhost.localdomain:imap->localhost.localdomain:47203 (ESTABLISHED)
imapd.old 14247  cyrus    1u     IPv4  372137855                  TCP
localhost.localdomain:imap->localhost.localdomain:47203 (ESTABLISHED)
imapd.old 14247  cyrus    2u     IPv4  372137855                  TCP
localhost.localdomain:imap->localhost.localdomain:47203 (ESTABLISHED)
sh        15437   root    3u     IPv4  372137852                  TCP
localhost.localdomain:47203->localhost.localdomain:imap (ESTABLISHED)
grep      15439   root    3u     IPv4  372137852                  TCP
localhost.localdomain:47203->localhost.localdomain:imap (ESTABLISHED)
grep      15440   root    3u     IPv4  372137852                  TCP
localhost.localdomain:47203->localhost.localdomain:imap (ESTABLISHED)
localhost.localdomain> lsof -p 14247
COMMAND     PID  USER   FD   TYPE     DEVICE     SIZE      NODE NAME
imapd.old 14247 cyrus  cwd    DIR      253,0     4096         2 /
imapd.old 14247 cyrus  rtd    DIR      253,0     4096         2 /
imapd.old 14247 cyrus  txt    REG      253,0   976740    252127
/usr/lib/cyrus-imapd/imapd.old

<some lines deleted>

imapd.old 14247 cyrus  mem    REG      253,4  1081344  10539035
/s80/data/cyrus/config/db/__db.003
imapd.old 14247 cyrus  mem    REG      253,4  1318912  10539014
/s80/data/cyrus/config/db/__db.002
imapd.old 14247 cyrus  mem    REG      253,4    16384  10539013
/s80/data/cyrus/config/db/__db.001
imapd.old 14247 cyrus    0u  IPv4  372137855                TCP
localhost.localdomain:imap->localhost.localdomain:47203 (ESTABLISHED)
imapd.old 14247 cyrus    1u  IPv4  372137855                TCP
localhost.localdomain:imap->localhost.localdomain:47203 (ESTABLISHED)
imapd.old 14247 cyrus    2u  IPv4  372137855                TCP
localhost.localdomain:imap->localhost.localdomain:47203 (ESTABLISHED)
imapd.old 14247 cyrus    3w  FIFO        0,7              18403 pipe
imapd.old 14247 cyrus    4u  IPv4      18402                TCP
localhost.localdomain:imap (LISTEN)
imapd.old 14247 cyrus    5u  unix 0xec82bd00          372137853 socket
imapd.old 14247 cyrus    6u   REG      253,4  4180400  10539067
/s80/data/cyrus/config/mailboxes.db
imapd.old 14247 cyrus    7u  unix 0xe477d900          372137854 socket
imapd.old 14247 cyrus    8u   REG      253,4      144  10539056
/s80/data/cyrus/config/annotations.db
imapd.old 14247 cyrus    9u   REG      253,4        0  12234753
/s80/data/cyrus/config/socket/imap-0.lock
imapd.old 14247 cyrus   10u   REG      253,4       34  10541035
/s80/data/cyrus/config/proc/14247
localhost.localdomain> renm user/rztths/a user/rztths/b
localhost.localdomain> exit
[root@mail2 src]# lsof -p 14247
COMMAND     PID  USER   FD   TYPE     DEVICE     SIZE      NODE NAME
imapd.old 14247 cyrus  cwd    DIR      253,0     4096         2 /
imapd.old 14247 cyrus  rtd    DIR      253,0     4096         2 /
imapd.old 14247 cyrus  txt    REG      253,0   976740    252127
/usr/lib/cyrus-imapd/imapd.old

<some lines deleted>

imapd.old 14247 cyrus  mem    REG      253,4  1081344  10539035
/s80/data/cyrus/config/db/__db.003
imapd.old 14247 cyrus  mem    REG      253,4  1318912  10539014
/localhost.localdomain> lq user/rztths
 STORAGE 392087/512000 (76.5794921875%)
localhost.localdomain> cm user/rztths/a
localhost.localdomain> cm user/rztths/a/a
localhost.localdomain> cm user/rztths/a/a/a
localhost.localdomain> lsof|grep localhost|grep ESTABLISHED
mycyradm  14245   root    3u     IPv4  372137852                  TCP
localhost.localdomain:47203->localhost.localdomain:imap (ESTABLISHED)
imapd.old 14247  cyrus    0u     IPv4  372137855                  TCP
localhost.localdomain:imap->localhost.localdomain:47203 (ESTABLISHED)
imapd.old 14247  cyrus    1u     IPv4  372137855                  TCP
localhost.localdomain:imap->localhost.localdomain:47203 (ESTABLISHED)
imapd.old 14247  cyrus    2u     IPv4  372137855                  TCP
localhost.localdomain:imap->localhost.localdomain:47203 (ESTABLISHED)
sh        15437   root    3u     IPv4  372137852                  TCP
localhost.localdomain:47203->localhost.localdomain:imap (ESTABLISHED)
grep      15439   root    3u     IPv4  372137852                  TCP
localhost.localdomain:47203->localhost.localdomain:imap (ESTABLISHED)
grep      15440   root    3u     IPv4  372137852                  TCP
localhost.localdomain:47203->localhost.localdomain:imap (ESTABLISHED)
localhost.localdomain> lsof -p 14247
COMMAND     PID  USER   FD   TYPE     DEVICE     SIZE      NODE NAME
imapd.old 14247 cyrus  cwd    DIR      253,0     4096         2 /
imapd.old 14247 cyrus  rtd    DIR      253,0     4096         2 /
imapd.old 14247 cyrus  txt    REG      253,0   976740    252127
/usr/lib/cyrus-imapd/imapd.old

<some "mem" lines deleted>

imapd.old 14247 cyrus  mem    REG      253,4  1081344  10539035
/s80/data/cyrus/config/db/__db.003
imapd.old 14247 cyrus  mem    REG      253,4  1318912  10539014
/s80/data/cyrus/config/db/__db.002
imapd.old 14247 cyrus  mem    REG      253,4    16384  10539013
/s80/data/cyrus/config/db/__db.001
imapd.old 14247 cyrus    0u  IPv4  372137855                TCP
localhost.localdomain:imap->localhost.localdomain:47203 (ESTABLISHED)
imapd.old 14247 cyrus    1u  IPv4  372137855                TCP
localhost.localdomain:imap->localhost.localdomain:47203 (ESTABLISHED)
imapd.old 14247 cyrus    2u  IPv4  372137855                TCP
localhost.localdomain:imap->localhost.localdomain:47203 (ESTABLISHED)
imapd.old 14247 cyrus    3w  FIFO        0,7              18403 pipe
imapd.old 14247 cyrus    4u  IPv4      18402                TCP
localhost.localdomain:imap (LISTEN)
imapd.old 14247 cyrus    5u  unix 0xec82bd00          372137853 socket
imapd.old 14247 cyrus    6u   REG      253,4  4180400  10539067
/s80/data/cyrus/config/mailboxes.db
imapd.old 14247 cyrus    7u  unix 0xe477d900          372137854 socket
imapd.old 14247 cyrus    8u   REG      253,4      144  10539056
/s80/data/cyrus/config/annotations.db
imapd.old 14247 cyrus    9u   REG      253,4        0  12234753
/s80/data/cyrus/config/socket/imap-0.lock
imapd.old 14247 cyrus   10u   REG      253,4       34  10541035
/s80/data/cyrus/config/proc/14247
localhost.localdomain> renm user/rztths/a user/rztths/b
localhost.localdomain> exit
[root@mail2 src]# lsof -p 14247
COMMAND     PID  USER   FD   TYPE     DEVICE     SIZE      NODE NAME
imapd.old 14247 cyrus  cwd    DIR      253,0     4096         2 /
imapd.old 14247 cyrus  rtd    DIR      253,0     4096         2 /
imapd.old 14247 cyrus  txt    REG      253,0   976740    252127
/usr/lib/cyrus-imapd/imapd.old

<some "mem" lines deleted>

imapd.old 14247 cyrus  mem    REG      253,4  1081344  10539035
/s80/data/cyrus/config/db/__db.003
imapd.old 14247 cyrus  mem    REG      253,4  1318912  10539014
/s80/data/cyrus/config/db/__db.002
imapd.old 14247 cyrus  mem    REG      253,4    16384  10539013
/s80/data/cyrus/config/db/__db.001
imapd.old 14247 cyrus    0u   CHR        1,3               2651 /dev/null
imapd.old 14247 cyrus    1u   CHR        1,3               2651 /dev/null
imapd.old 14247 cyrus    2u   CHR        1,3               2651 /dev/null
imapd.old 14247 cyrus    3w  FIFO        0,7              18403 pipe
imapd.old 14247 cyrus    4u  IPv4      18402                TCP
localhost.localdomain:imap (LISTEN)
imapd.old 14247 cyrus    5u  unix 0xec82bd00          372137853 socket
imapd.old 14247 cyrus    6u   REG      253,4  4180880  10539067
/s80/data/cyrus/config/mailboxes.db
imapd.old 14247 cyrus    7u  unix 0xe477d900          372137854 socket
imapd.old 14247 cyrus    8u   REG      253,4      144  10539056
/s80/data/cyrus/config/annotations.db
imapd.old 14247 cyrus    9uW  REG      253,4        0  12234753
/s80/data/cyrus/config/socket/imap-0.lock
imapd.old 14247 cyrus   14u   REG      253,4       17  10539839
/s80/data/cyrus/config/quota/r/user.rztths
imapd.old 14247 cyrus   15u   REG      253,4       17  12256609
/s80/data/cyrus/config/quota/r/user.rztths
imapd.old 14247 cyrus   16u   REG      253,4       17  12257598
/s80/data/cyrus/config/quota/r/user.rztths

Please look at filedescriptors 14-16, the imapd is not connected to any user but
still has open quota-Files, all with different inode numbers. 
Actual results:

Filedescriptor 14-16 open
Expected results:

Not open
Additional info:
The bug seems to be introduced by Patch3 in the .spec-File, 
http://email.uoa.gr/download/cyrus/cyrus-imapd-2.2.7/cyrus-imapd-2.2.7-rmquota+deletemailbox-0.1.diff
I've build my own version without this Patch and everything works fine.
Looking at
http://email.uoa.gr/projects/cyrus/quota-patches/deletemailbox/
shows a newer patch for cyrus 2.2.12 but I did not try it out because I
do not need this functionallity.

-- Additional comment from schlichting on 2007-08-22 08:51 EST --
I can see the same problem on RHEL 5 with cyrus-imapd-2.3.7-1.1.el5

As before, removing Patch3 solves the problem.

Comment 1 RHEL Program Management 2007-12-03 20:45:33 UTC
This request was evaluated by Red Hat Product Management for
inclusion, but this component is not scheduled to be updated in
the current Red Hat Enterprise Linux release.  This request will
be reviewed for a future Red Hat Enterprise Linux release.

Comment 3 RHEL Program Management 2008-07-21 23:09:37 UTC
This request was evaluated by Red Hat Product Management for
inclusion, but this component is not scheduled to be updated in
the current Red Hat Enterprise Linux release. If you would like
this request to be reviewed for the next minor release, ask your
support representative to set the next rhel-x.y flag to "?".

Comment 5 Michal Hlavinka 2009-04-16 12:56:24 UTC
I've checked new version of the patch and it does not fix this problem

Comment 7 RHEL Program Management 2009-11-06 19:17:15 UTC
This request was evaluated by Red Hat Product Management for
inclusion, but this component is not scheduled to be updated in
the current Red Hat Enterprise Linux release. If you would like
this request to be reviewed for the next minor release, ask your
support representative to set the next rhel-x.y flag to "?".

Comment 8 Martin Poole 2009-11-24 18:11:54 UTC
Created attachment 373502 [details]
perl reproducer

Attached is a small prog to reproduce this.

Assumes a user 'cytest' with same password and cyrus-imapd setup accordingly.

Comment 9 Michal Hlavinka 2010-06-14 11:16:02 UTC
thanks for the reproducer, I've just looked at it but it's just running and running... I thought it's going to exhaust file descriptors or something but nothing like that happened. Using lsof I've checked the nomber of opened descriptors and it seems stable (oscillating 49 - 59).

Comment 10 Thorsten Schlichting 2010-06-14 14:14:34 UTC
Created attachment 423847 [details]
Other reproducer

My version of reproducer. Requires user cytest with INBOX and STORAGE quota on INBOX.

# cyradm --user cyrus localhost
IMAP Password:
localhost.localdomain> cm user.cytest
localhost.localdomain> sq user.cytest STORAGE 12345
localhost.localdomain> 
# ./fdleak_cyrus.pl
rename(INBOX.a.a,INBOX.a.b): i=117, System I/O error at ./fdleak_cyrus.pl line 32.

2010-06-14T16:00:59.621013+02:00 thstest imap[4679]: IOERROR: creating quota file /var/lib/imap/quota/c/user.cytest.NEW: Too many open files
2010-06-14T16:00:59.621101+02:00 thstest imap[4679]: DBERROR: error storing user.cytest: cyrusdb error

Be aware that after running this script there are directories in the filesystem that are not in the mailbox list and might need to be deleted manually. During this run user.cytest.a.b exists but is not listed:

localhost.localdomain> lm user.cytest*
user.cytest (\HasChildren)        user.cytest.a.a (\HasNoChildren)
user.cytest.a (\HasChildren)

# ll -R /var/spool/imap/c/user/cytest/a
/var/spool/imap/c/user/cytest/a:
total 40
drwx------ 2 cyrus mail 4096 Jun 14 16:00 a
drwx------ 2 cyrus mail 4096 Jun 14 16:00 b
-rw------- 1 cyrus mail    4 Jun 14 16:00 cyrus.cache
-rw------- 1 cyrus mail  165 Jun 14 16:00 cyrus.header
-rw------- 1 cyrus mail   96 Jun 14 16:00 cyrus.index

/var/spool/imap/c/user/cytest/a/a:
total 24
-rw------- 2 cyrus mail   4 Jun 14 16:00 cyrus.cache
-rw------- 1 cyrus mail 165 Jun 14 16:00 cyrus.header
-rw------- 2 cyrus mail  96 Jun 14 16:00 cyrus.index

/var/spool/imap/c/user/cytest/a/b:
total 24
-rw------- 2 cyrus mail   4 Jun 14 16:00 cyrus.cache
-rw------- 1 cyrus mail 165 Jun 14 16:00 cyrus.header
-rw------- 2 cyrus mail  96 Jun 14 16:00 cyrus.index

Comment 11 Michal Hlavinka 2010-06-15 10:58:17 UTC
thanks both of you

> Assumes a user 'cytest' with same password and cyrus-imapd setup accordingly.

seems this was not enough, creating the quota manually was needed

I can reproduce this issue fine and I'm discussing this with original author of the patch now.

Comment 13 RHEL Program Management 2010-08-09 18:42:55 UTC
This request was evaluated by Red Hat Product Management for
inclusion in the current release of Red Hat Enterprise Linux.
Because the affected component is not scheduled to be updated in the
current release, Red Hat is unfortunately unable to address this
request at this time. Red Hat invites you to ask your support
representative to propose this request, if appropriate and relevant,
in the next release of Red Hat Enterprise Linux.

Comment 15 RHEL Program Management 2011-01-11 21:12:43 UTC
This request was evaluated by Red Hat Product Management for
inclusion in the current release of Red Hat Enterprise Linux.
Because the affected component is not scheduled to be updated in the
current release, Red Hat is unfortunately unable to address this
request at this time. Red Hat invites you to ask your support
representative to propose this request, if appropriate and relevant,
in the next release of Red Hat Enterprise Linux.

Comment 16 RHEL Program Management 2011-01-11 22:26:13 UTC
This request was erroneously denied for the current release of
Red Hat Enterprise Linux.  The error has been fixed and this
request has been re-proposed for the current release.

Comment 20 errata-xmlrpc 2011-07-21 12:19:10 UTC
An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on therefore solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHBA-2011-1075.html