Bug 113492 - after expunge, dovecot hangs fetchmail if new e-mail came in
Summary: after expunge, dovecot hangs fetchmail if new e-mail came in
Keywords:
Status: CLOSED RAWHIDE
Alias: None
Product: Fedora
Classification: Fedora
Component: fetchmail
Version: 3
Hardware: i386
OS: Linux
medium
medium
Target Milestone: ---
Assignee: John Dennis
QA Contact:
URL:
Whiteboard:
Depends On:
Blocks: FC3BugWeekTracker
TreeView+ depends on / blocked
 
Reported: 2004-01-14 17:06 UTC by Alexandre Oliva
Modified: 2007-11-30 22:10 UTC (History)
4 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2004-09-30 21:16:25 UTC
Type: ---
Embargoed:


Attachments (Terms of Use)
fix when imap EXPUNGE cmd doesn't update the message count (1.33 KB, patch)
2004-05-25 16:40 UTC, Fabrice Bellet
no flags Details | Diff
possible fix for the "LF not found" problem ? (503 bytes, patch)
2004-05-25 16:55 UTC, Fabrice Bellet
no flags Details | Diff
LF not found patch 2 (3.81 KB, patch)
2004-05-25 18:18 UTC, Timo Sirainen
no flags Details | Diff
LF not found patch 3 (4.00 KB, patch)
2004-05-25 18:53 UTC, Timo Sirainen
no flags Details | Diff
fix for recentcount and count after an imap EXPUNGE (2.11 KB, patch)
2004-05-26 14:21 UTC, Fabrice Bellet
no flags Details | Diff
new fix for recentcount and count after an imap EXPUNGE (1.56 KB, patch)
2004-06-02 11:23 UTC, Fabrice Bellet
no flags Details | Diff

Description Alexandre Oliva 2004-01-14 17:06:48 UTC
From Bugzilla Helper:
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.5) Gecko/20031202

Description of problem:
Using fetchmail -N to get e-mail with ssh preauth, with the following
.fetchmailrc:

defaults
   protocol imap
   auth ssh
   to aoliva
   expunge 0
poll oliva@lsd
   via emilia.lsd.ic.unicamp.br.
   plugin "ssh -x -a -l oliva emilia /usr/libexec/dovecot/imap"

it appears that expunging downloaded e-mail after everything in the
mailbox was downloaded causes fetchmail/dovecot to go out of sync,
because fetchmail says:

reading message aoliva.ic.unicamp.br:28 of 28 (1749 header
octets) . (1201 body octets) . flushed
31 messages (28 seen) for aoliva at emilia.lsd.ic.unicamp.br.
fetchmail: timeout after 300 seconds waiting for server oliva@lsd.
fetchmail: socket error while fetching from oliva@lsd
fetchmail: Query status=2 (SOCKET)

eventually the server times out, closes the connection, and the next
round starts ok.

I've got other scary messages at expunge/reindex time, such as lack of
From: or CR characters, that I never got before switching from UW
imapd to dovecot.

Version-Release number of selected component (if applicable):
dovecot-0.99.10-6

How reproducible:
Sometimes

Steps to Reproduce:
1.Set up fetchmailrc to get e-mail from a remote mbox using ssh and
dovecot with preauth imap, expunging e-mail at the end of the download
2.Run it

Actual Results:  If new e-mail comes in during the download, errors
like the above.

Expected Results:  No such errors.

Additional info:

Comment 1 Alexandre Oliva 2004-01-15 17:49:52 UTC
This is one of the other error messages I mentioned in the report
above, that I couldn't quote at that time:

[...]
reading message aoliva.ic.unicamp.br:45 of 45 (2206 header
octets) .. (2070 body octets) .. flushed
imap(oliva): Error: Error indexing mbox file /var/mail/oliva: LF not
found where expected


Comment 2 Warren Togami 2004-05-19 10:24:42 UTC
Timo, have you seen anything like this before?

Comment 3 Timo Sirainen 2004-05-22 03:39:41 UTC
That looks a bit as if locking isn't right. MTA using only dotlocking
and appending mail while Dovecot has locked it for reading with
fcntl() and later gets confused?

Then again, mbox syncing does have some bugs. I didn't think there
were anything too simple though..


Comment 4 Alexandre Oliva 2004-05-22 05:47:09 UTC
MTA (sendmail) is using both fctnl and .lock, according to strace.

Comment 5 Fabrice Bellet 2004-05-25 16:40:21 UTC
Created attachment 100547 [details]
fix when imap EXPUNGE cmd doesn't update the message count

The fetchmail timeout with dovecot after the EXPUNGE command occurs because
dovecot doesn't report the updated number of RECENT and EXISTS messages, like
wu-imap did. A possible workaround is to issue a SELECT or EXAMINE command
after the EXPUNGE, to force an update of these values. It seems to work for me
with this attached patch

Comment 6 Fabrice Bellet 2004-05-25 16:55:13 UTC
Created attachment 100548 [details]
possible fix for the "LF not found" problem ?

I'm not sure this is the right fix, but the input->v_offset value is modified
in mbox_index_rewrite(), and sometimes, an inconsistant value coming from this
function is propagated back to the parent, causing the error described in the
bug report. Typically, the offset is no longer at the boundary between two
messages, as it should be, but in the middle of the first message for example.
This case occurs during the EXPUNGE command when the mailbox is modifed by
another process (eg sendmail). Saving and restoring the current v_offset value
seems to solve this issue.

Comment 7 Timo Sirainen 2004-05-25 18:18:22 UTC
Created attachment 100550 [details]
LF not found patch 2

Thanks for figuring out the problem. I looked it a bit and figured out that
stream->v_size will also be too small if mbox_index_rewrite() grows the file.
This patch should fix it.

Hopefully this was the cause of the quite-rare mbox corruption that I had been
hearing. I'll release 0.99.10.5 today with this and several other important
fixes.

Comment 8 Timo Sirainen 2004-05-25 18:53:04 UTC
Created attachment 100552 [details]
LF not found patch 3

There was a bug in the patch :) This one seems to work.

Comment 9 Timo Sirainen 2004-05-25 19:38:25 UTC
And about the fetchmail/EXISTS problem - sending EXISTS after EXPUNGEs is pretty 
pointless and so is re-selecting the mailbox after it. It should rather just internally keep 
state of the message count (save EXISTS value, decrease it for each EXPUNGE).

Recent counters are different though, that should be re-sent if any expunged messages 
was a \Recent message. I'm not fixing this for 0.99.10 though, it has other problems with 
recent counters as well..

Comment 10 Warren Togami 2004-05-25 19:49:58 UTC
Timo,

Please inform us when the new release is ready, and I will prepare the
official FC1 & FC2 update.  After a bit of testing we will push this.

Thanks,
Warren

Comment 11 Fabrice Bellet 2004-05-26 14:21:27 UTC
Created attachment 100585 [details]
fix for recentcount and count after an imap EXPUNGE

About the timeout bug in fetchmail, as suggested, I reworked on a possible
patch. This one updates internally the value of count (EXISTS), recentcount
(RECENT) and expungedcount (EXPUNGE), when they are not updated by the imap
server itself after the expunge command. To be safe about recentcount, I reset
this value to zero, when it's not updated by the imap server. The visible
effect of this, is that messages incoming while the mailqueue is processed are
kept in queue.

Comment 12 Warren Togami 2004-05-27 21:18:39 UTC
dovecot-0.99.10.5-1 is currently in rawhide and I am currently testing
it before releasing an update for FC1 and FC2.  Timo please let me
know if Fabrice's comment #11 should be applied or not.

Comment 13 Timo Sirainen 2004-05-27 21:26:06 UTC
#11 is a patch for fetchmail, I can't really comment on that .. except
I don't think expungedcount is a good idea, rather just decrease
"count" directly. Otherwise it'd break if server sent eg.:

* 1 EXPUNGE
* 2 EXISTS
* 1 EXPUNGE

count -= expungedcount -> 0, not 1 as it should

Comment 14 Warren Togami 2004-05-27 21:29:56 UTC
oops, sorry I should have actually read it. =)
Fabrice can you please talk to fetchmail upstream about this issue?

Time for me to prepare the update announcement...

Comment 15 Fabrice Bellet 2004-06-02 11:23:20 UTC
Created attachment 100788 [details]
new fix for recentcount and count after an imap EXPUNGE

Ah yes. I forgot this case. This new patch updates count directly. I'll report
this case on the fetchmail mailing-list.

Comment 16 Warren Togami 2004-06-25 05:11:24 UTC
dovecot update for FC1 and FC2 went out a few days ago.

Comment 17 Alexandre Oliva 2004-06-25 17:49:33 UTC
Comment #15 contains a patch for fetchmail that still hasn't been
integrated.  Reopening the bug and moving it to fetchmail.

Comment 18 Alexandre Oliva 2004-09-05 08:43:03 UTC
Ping?  This is still present in rawhide (FC3 approaching test2).

Comment 19 John Dennis 2004-09-07 14:37:17 UTC
Huh? What's still in rawhide? The dovecot-0.99.10.5-1 rpm or the bug?
Niether of which to the best of my knowledge is in rawhide. The
current package in FC3 is dovecot-0.99.10.9-1.FC3.2 which has the fix
for this bug in it (thats my understanding) and was up until 3 days
ago the current upstream release. dovecot-0.99-11 was released on
Saturday, which will go in FC3, probably today.

Comment 20 Alexandre Oliva 2004-09-07 16:32:29 UTC
The fetchmail bug.  See, this bug has been in the fetchmail component
since the dovecot problems were all fixed.  But the patch in Comment
#15, for fetchmail, is still needed to fix the reported problem.

Comment 21 Fabrice Bellet 2004-09-08 21:53:04 UTC
yep, I confirm that there's still a problem in fetchmail imap, when
the RECENT and EXISTS values are not updated and reported after an
EXPUNGE by the imap server. This bug is fixed by the patch of Comment
#15. I mentionned it on the fetchmail-friends mailing-list in June,
without much comments. It seems that the development of fetchmail now
shifted to http://fetchmail.berlios.de so I'll resubmit this patch to
the newly created fetchmail-devel list asap.

Comment 22 Fabrice Bellet 2004-09-13 10:46:56 UTC
Ok, it will be fixed upstream:
http://lists.berlios.de/pipermail/fetchmail-devel/2004-September/000141.html

Comment 23 Alexandre Oliva 2004-09-29 17:12:28 UTC
Any chance of having this patch integrated before FC3test3?

Comment 25 John Dennis 2004-09-30 21:16:25 UTC
final patch applied in fetchmail-6.2.5-6


Note You need to log in before you can comment on or make changes to this bug.