Created attachment 501695 [details]
offlineimap -d imap,maildir output
Description of problem:
After upgrading to F15, I found that offlineimap deletes emails in the local storage (emails are not deleted from the server). Only reproducible in mailboxes with large numbers of emails. This does not happen with version 6.2.0.
For a more verbose version see
Version-Release number of selected component (if applicable):
Run offlineimap on a mailbox with several thousand emails, in my case the command was "offlineimap -f lists-xorg" on a mailbox with just over 25k emails. When I copied the messages of this mailbox into a "test" mailbox, I was unable to reproduce the bug though. I can reproduce it in at least two other live mailboxes though.
In the log file attached, I noticed that all emails deleted have a UID of 32768 or greater. This may be a hint. I haven't checked if this is the case with other mailboxes, right now I'm rather hesitant to keep running this command without guidance on what exact commands to run.
Also note that the log file is shortened, I've tried to leave enough in to make it sensible without cutting anything of value. The original is over 8 MB. Places where information was cut are marked with [...]
there are many possible sources of that error.
In particular: Do you know the mailserver in place? Are the emails still there with another uid? (Some servers do not really give unique ids)? What happens if you sync with offlineimap to an empty directory, are those mails actually downloaded?
(In reply to comment #1)
> Do you know the mailserver in place?
The mail server is dovecot 1.2.8
fwiw, this setup hasn't changed in ages, neither the server nor the local offlineimaprc.
> Are the emails still there with another uid? (Some servers do not really give unique ids)?
I'm not quite sure how to check if the emails are still there with another UID, any instructions would be appreciated.
> What happens if you sync with offlineimap to an empty directory, are those mails actually downloaded?
Yes, it syncs. Seems to have downloaded all emails as required. But when I ran the same command again (offlineimap -f lists-xorg) on the just-downloaded folder it deleted a number of them again: Deleting 5268 messages (32891, ...
I have the logs for the second run (was silly enough to overwrite the log from the empty sync) but I don't see any real difference to the original one, seems to be just more of the same.
Ok dovecot is known to be very reliable and works well with offlineimap.
> I'm not quite sure how to check if the emails are still there with another UID,
any instructions would be appreciated.
You could access the server and simply grep your mailbox's uidlist for the deleted messages and also search for duplicates.
(see http://wiki1.dovecot.org/MailboxFormat/Maildir#IMAP_UID_mapping for details)
The question here is, why offlineimap downloads the messages first and _then_ deletes them. Somehow we need to nail the actual problem (aka, what makes those 5268 messages different from the rest), so we can inform upstream.
Created attachment 502395 [details]
I'll just attach the whole list here. UID list and the filenames are unique across the file and I'm having a hard time figuring out what makes 32891 special (or 32768 in the previous run).
Thanks for the detailed IMAP log. This is much appreciated, and it helps for sure.
OfflineImap has recently switched to IMAPLIB2, in order to be able to support IMAP IDLE and other cool stuff. But your log seems indicate that it eats data.
For example the email server sends
untagged_responses[FETCH] 5399 += ["16222 (FLAGS (\Seen) UID 32749)"]
But the subsequent
_get_untagged_response(FETCH) => ['1 (FLAGS (\\See......
does not contain the very UID, and we subsequently try to delete it locally as we think it's gone from the server. I am in contact with the imaplib2 author and will try to get this debugged ASAP.
Hi Peter, I am in contact with the imaplib2 author and he would like to investigate this. Despite the great IMAP log, there is some stuff snipped out in the [...] part. Do you still have the full log and would you be willing to share it with me (or directly the imaplib2 author)? You ca reach me at firstname.lastname@example.org. Thanks
Just to let you know that we are on it:
We fetch a list of UIDs from the server, but the list must currently not
be interrupted with unrelated responses.
THIS is send right after one of "missing UIDs"
DEBUG[imap]: 26:17.31 who-t.net reader < * OK Searched 43% of the mailbox,
so this series (handler log):
[FETCH] 10821 += ["10822 (FLAGS (\Answered \Seen) UID
[OK] 0 += ["Searched 43% of the mailbox, ETA 0:12"]
[FETCH] 0 += ["10823 (FLAGS (\Answered \Seen) UID 27261)"]
is broken into several "Responses" (note the FETCH restarts counting at
0). It breaks the series of untagged_responses just as the imaplib2 author had
suspected. Fixing this will require to fix imaplib2 and bundle a new version in a new version of offlineimap.
Just FYI, a fix is in the current "master" tree and *I* would recommend to upgrade any fedora package to use the latest master HEAD as soon as possible to avoid further potential data loss by others.
Thanks for raising this and providing us with detailed logs. It was a pleasure to fix this bug :-).
Christoph, are you going to submit new packages or do I have to try from git?
You can try it from git for now. I am going to update to 6.3.4 ASAP once it is out. Since offlineimap is a python package, I do not see much sense in building -rc*.rpms
btw, I can confirm that v6.3.4-rc3 from git has fixed this bug.
Good to hear that.
Can you please update to the latest git snapshot if the release hasn't been made yet?
6.3.4 has been released on August 19 according to the git tag upstream. Please update on F15 and 16.
Christoph Höger, do you need help with the update? I can push an update myself if needed.
offlineimap-6.3.4-1.fc16 has been submitted as an update for Fedora 16.
offlineimap-6.3.4-1.fc15 has been submitted as an update for Fedora 15.
* should fix your issue,
* was pushed to the Fedora 16 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing offlineimap-6.3.4-1.fc16'
as soon as you are able to.
Please go to the following url:
then log in and leave karma (feedback).
offlineimap-6.3.4-1.fc15 has been pushed to the Fedora 15 stable repository. If problems still persist, please make note of it in this bug report.
offlineimap-6.3.4-1.fc16 has been pushed to the Fedora 16 stable repository. If problems still persist, please make note of it in this bug report.