Red Hat Bugzilla – Bug 113492
after expunge, dovecot hangs fetchmail if new e-mail came in
Last modified: 2007-11-30 17:10:35 EST
From Bugzilla Helper:
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.5) Gecko/20031202
Description of problem:
Using fetchmail -N to get e-mail with ssh preauth, with the following
plugin "ssh -x -a -l oliva emilia /usr/libexec/dovecot/imap"
it appears that expunging downloaded e-mail after everything in the
mailbox was downloaded causes fetchmail/dovecot to go out of sync,
because fetchmail says:
reading message firstname.lastname@example.org:28 of 28 (1749 header
octets) . (1201 body octets) . flushed
31 messages (28 seen) for aoliva at emilia.lsd.ic.unicamp.br.
fetchmail: timeout after 300 seconds waiting for server oliva@lsd.
fetchmail: socket error while fetching from oliva@lsd
fetchmail: Query status=2 (SOCKET)
eventually the server times out, closes the connection, and the next
round starts ok.
I've got other scary messages at expunge/reindex time, such as lack of
From: or CR characters, that I never got before switching from UW
imapd to dovecot.
Version-Release number of selected component (if applicable):
Steps to Reproduce:
1.Set up fetchmailrc to get e-mail from a remote mbox using ssh and
dovecot with preauth imap, expunging e-mail at the end of the download
Actual Results: If new e-mail comes in during the download, errors
like the above.
Expected Results: No such errors.
This is one of the other error messages I mentioned in the report
above, that I couldn't quote at that time:
reading message email@example.com:45 of 45 (2206 header
octets) .. (2070 body octets) .. flushed
imap(oliva): Error: Error indexing mbox file /var/mail/oliva: LF not
found where expected
Timo, have you seen anything like this before?
That looks a bit as if locking isn't right. MTA using only dotlocking
and appending mail while Dovecot has locked it for reading with
fcntl() and later gets confused?
Then again, mbox syncing does have some bugs. I didn't think there
were anything too simple though..
MTA (sendmail) is using both fctnl and .lock, according to strace.
Created attachment 100547 [details]
fix when imap EXPUNGE cmd doesn't update the message count
The fetchmail timeout with dovecot after the EXPUNGE command occurs because
dovecot doesn't report the updated number of RECENT and EXISTS messages, like
wu-imap did. A possible workaround is to issue a SELECT or EXAMINE command
after the EXPUNGE, to force an update of these values. It seems to work for me
with this attached patch
Created attachment 100548 [details]
possible fix for the "LF not found" problem ?
I'm not sure this is the right fix, but the input->v_offset value is modified
in mbox_index_rewrite(), and sometimes, an inconsistant value coming from this
function is propagated back to the parent, causing the error described in the
bug report. Typically, the offset is no longer at the boundary between two
messages, as it should be, but in the middle of the first message for example.
This case occurs during the EXPUNGE command when the mailbox is modifed by
another process (eg sendmail). Saving and restoring the current v_offset value
seems to solve this issue.
Created attachment 100550 [details]
LF not found patch 2
Thanks for figuring out the problem. I looked it a bit and figured out that
stream->v_size will also be too small if mbox_index_rewrite() grows the file.
This patch should fix it.
Hopefully this was the cause of the quite-rare mbox corruption that I had been
hearing. I'll release 0.99.10.5 today with this and several other important
Created attachment 100552 [details]
LF not found patch 3
There was a bug in the patch :) This one seems to work.
And about the fetchmail/EXISTS problem - sending EXISTS after EXPUNGEs is pretty
pointless and so is re-selecting the mailbox after it. It should rather just internally keep
state of the message count (save EXISTS value, decrease it for each EXPUNGE).
Recent counters are different though, that should be re-sent if any expunged messages
was a \Recent message. I'm not fixing this for 0.99.10 though, it has other problems with
recent counters as well..
Please inform us when the new release is ready, and I will prepare the
official FC1 & FC2 update. After a bit of testing we will push this.
Created attachment 100585 [details]
fix for recentcount and count after an imap EXPUNGE
About the timeout bug in fetchmail, as suggested, I reworked on a possible
patch. This one updates internally the value of count (EXISTS), recentcount
(RECENT) and expungedcount (EXPUNGE), when they are not updated by the imap
server itself after the expunge command. To be safe about recentcount, I reset
this value to zero, when it's not updated by the imap server. The visible
effect of this, is that messages incoming while the mailqueue is processed are
kept in queue.
dovecot-0.99.10.5-1 is currently in rawhide and I am currently testing
it before releasing an update for FC1 and FC2. Timo please let me
know if Fabrice's comment #11 should be applied or not.
#11 is a patch for fetchmail, I can't really comment on that .. except
I don't think expungedcount is a good idea, rather just decrease
"count" directly. Otherwise it'd break if server sent eg.:
* 1 EXPUNGE
* 2 EXISTS
* 1 EXPUNGE
count -= expungedcount -> 0, not 1 as it should
oops, sorry I should have actually read it. =)
Fabrice can you please talk to fetchmail upstream about this issue?
Time for me to prepare the update announcement...
Created attachment 100788 [details]
new fix for recentcount and count after an imap EXPUNGE
Ah yes. I forgot this case. This new patch updates count directly. I'll report
this case on the fetchmail mailing-list.
dovecot update for FC1 and FC2 went out a few days ago.
Comment #15 contains a patch for fetchmail that still hasn't been
integrated. Reopening the bug and moving it to fetchmail.
Ping? This is still present in rawhide (FC3 approaching test2).
Huh? What's still in rawhide? The dovecot-0.99.10.5-1 rpm or the bug?
Niether of which to the best of my knowledge is in rawhide. The
current package in FC3 is dovecot-0.99.10.9-1.FC3.2 which has the fix
for this bug in it (thats my understanding) and was up until 3 days
ago the current upstream release. dovecot-0.99-11 was released on
Saturday, which will go in FC3, probably today.
The fetchmail bug. See, this bug has been in the fetchmail component
since the dovecot problems were all fixed. But the patch in Comment
#15, for fetchmail, is still needed to fix the reported problem.
yep, I confirm that there's still a problem in fetchmail imap, when
the RECENT and EXISTS values are not updated and reported after an
EXPUNGE by the imap server. This bug is fixed by the patch of Comment
#15. I mentionned it on the fetchmail-friends mailing-list in June,
without much comments. It seems that the development of fetchmail now
shifted to http://fetchmail.berlios.de so I'll resubmit this patch to
the newly created fetchmail-devel list asap.
Ok, it will be fixed upstream:
Any chance of having this patch integrated before FC3test3?
final patch applied in fetchmail-6.2.5-6