From Bugzilla Helper: User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.5) Gecko/20031202 Description of problem: Using fetchmail -N to get e-mail with ssh preauth, with the following .fetchmailrc: defaults protocol imap auth ssh to aoliva expunge 0 poll oliva@lsd via emilia.lsd.ic.unicamp.br. plugin "ssh -x -a -l oliva emilia /usr/libexec/dovecot/imap" it appears that expunging downloaded e-mail after everything in the mailbox was downloaded causes fetchmail/dovecot to go out of sync, because fetchmail says: reading message aoliva.ic.unicamp.br:28 of 28 (1749 header octets) . (1201 body octets) . flushed 31 messages (28 seen) for aoliva at emilia.lsd.ic.unicamp.br. fetchmail: timeout after 300 seconds waiting for server oliva@lsd. fetchmail: socket error while fetching from oliva@lsd fetchmail: Query status=2 (SOCKET) eventually the server times out, closes the connection, and the next round starts ok. I've got other scary messages at expunge/reindex time, such as lack of From: or CR characters, that I never got before switching from UW imapd to dovecot. Version-Release number of selected component (if applicable): dovecot-0.99.10-6 How reproducible: Sometimes Steps to Reproduce: 1.Set up fetchmailrc to get e-mail from a remote mbox using ssh and dovecot with preauth imap, expunging e-mail at the end of the download 2.Run it Actual Results: If new e-mail comes in during the download, errors like the above. Expected Results: No such errors. Additional info:
This is one of the other error messages I mentioned in the report above, that I couldn't quote at that time: [...] reading message aoliva.ic.unicamp.br:45 of 45 (2206 header octets) .. (2070 body octets) .. flushed imap(oliva): Error: Error indexing mbox file /var/mail/oliva: LF not found where expected
Timo, have you seen anything like this before?
That looks a bit as if locking isn't right. MTA using only dotlocking and appending mail while Dovecot has locked it for reading with fcntl() and later gets confused? Then again, mbox syncing does have some bugs. I didn't think there were anything too simple though..
MTA (sendmail) is using both fctnl and .lock, according to strace.
Created attachment 100547 [details] fix when imap EXPUNGE cmd doesn't update the message count The fetchmail timeout with dovecot after the EXPUNGE command occurs because dovecot doesn't report the updated number of RECENT and EXISTS messages, like wu-imap did. A possible workaround is to issue a SELECT or EXAMINE command after the EXPUNGE, to force an update of these values. It seems to work for me with this attached patch
Created attachment 100548 [details] possible fix for the "LF not found" problem ? I'm not sure this is the right fix, but the input->v_offset value is modified in mbox_index_rewrite(), and sometimes, an inconsistant value coming from this function is propagated back to the parent, causing the error described in the bug report. Typically, the offset is no longer at the boundary between two messages, as it should be, but in the middle of the first message for example. This case occurs during the EXPUNGE command when the mailbox is modifed by another process (eg sendmail). Saving and restoring the current v_offset value seems to solve this issue.
Created attachment 100550 [details] LF not found patch 2 Thanks for figuring out the problem. I looked it a bit and figured out that stream->v_size will also be too small if mbox_index_rewrite() grows the file. This patch should fix it. Hopefully this was the cause of the quite-rare mbox corruption that I had been hearing. I'll release 0.99.10.5 today with this and several other important fixes.
Created attachment 100552 [details] LF not found patch 3 There was a bug in the patch :) This one seems to work.
And about the fetchmail/EXISTS problem - sending EXISTS after EXPUNGEs is pretty pointless and so is re-selecting the mailbox after it. It should rather just internally keep state of the message count (save EXISTS value, decrease it for each EXPUNGE). Recent counters are different though, that should be re-sent if any expunged messages was a \Recent message. I'm not fixing this for 0.99.10 though, it has other problems with recent counters as well..
Timo, Please inform us when the new release is ready, and I will prepare the official FC1 & FC2 update. After a bit of testing we will push this. Thanks, Warren
Created attachment 100585 [details] fix for recentcount and count after an imap EXPUNGE About the timeout bug in fetchmail, as suggested, I reworked on a possible patch. This one updates internally the value of count (EXISTS), recentcount (RECENT) and expungedcount (EXPUNGE), when they are not updated by the imap server itself after the expunge command. To be safe about recentcount, I reset this value to zero, when it's not updated by the imap server. The visible effect of this, is that messages incoming while the mailqueue is processed are kept in queue.
dovecot-0.99.10.5-1 is currently in rawhide and I am currently testing it before releasing an update for FC1 and FC2. Timo please let me know if Fabrice's comment #11 should be applied or not.
#11 is a patch for fetchmail, I can't really comment on that .. except I don't think expungedcount is a good idea, rather just decrease "count" directly. Otherwise it'd break if server sent eg.: * 1 EXPUNGE * 2 EXISTS * 1 EXPUNGE count -= expungedcount -> 0, not 1 as it should
oops, sorry I should have actually read it. =) Fabrice can you please talk to fetchmail upstream about this issue? Time for me to prepare the update announcement...
Created attachment 100788 [details] new fix for recentcount and count after an imap EXPUNGE Ah yes. I forgot this case. This new patch updates count directly. I'll report this case on the fetchmail mailing-list.
dovecot update for FC1 and FC2 went out a few days ago.
Comment #15 contains a patch for fetchmail that still hasn't been integrated. Reopening the bug and moving it to fetchmail.
Ping? This is still present in rawhide (FC3 approaching test2).
Huh? What's still in rawhide? The dovecot-0.99.10.5-1 rpm or the bug? Niether of which to the best of my knowledge is in rawhide. The current package in FC3 is dovecot-0.99.10.9-1.FC3.2 which has the fix for this bug in it (thats my understanding) and was up until 3 days ago the current upstream release. dovecot-0.99-11 was released on Saturday, which will go in FC3, probably today.
The fetchmail bug. See, this bug has been in the fetchmail component since the dovecot problems were all fixed. But the patch in Comment #15, for fetchmail, is still needed to fix the reported problem.
yep, I confirm that there's still a problem in fetchmail imap, when the RECENT and EXISTS values are not updated and reported after an EXPUNGE by the imap server. This bug is fixed by the patch of Comment #15. I mentionned it on the fetchmail-friends mailing-list in June, without much comments. It seems that the development of fetchmail now shifted to http://fetchmail.berlios.de so I'll resubmit this patch to the newly created fetchmail-devel list asap.
Ok, it will be fixed upstream: http://lists.berlios.de/pipermail/fetchmail-devel/2004-September/000141.html
Any chance of having this patch integrated before FC3test3?
final patch applied in fetchmail-6.2.5-6