Bug 708898 - offlineimap 6.3.3 deletes local emails
offlineimap 6.3.3 deletes local emails
Status: CLOSED ERRATA
Product: Fedora
Classification: Fedora
Component: offlineimap (Show other bugs)
15
Unspecified Unspecified
unspecified Severity unspecified
: ---
: ---
Assigned To: Christoph Höger
Fedora Extras Quality Assurance
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2011-05-30 00:49 EDT by Peter Hutterer
Modified: 2011-11-14 17:25 EST (History)
6 users (show)

See Also:
Fixed In Version: offlineimap-6.3.4-1.fc16
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2011-11-02 20:22:18 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
offlineimap -d imap,maildir output (729.83 KB, text/plain)
2011-05-30 00:49 EDT, Peter Hutterer
no flags Details
dovecot-uidlist (969.59 KB, text/plain)
2011-06-01 18:27 EDT, Peter Hutterer
no flags Details

  None (edit)
Description Peter Hutterer 2011-05-30 00:49:18 EDT
Created attachment 501695 [details]
offlineimap -d imap,maildir output

Description of problem:
After upgrading to F15, I found that offlineimap deletes emails in the local storage (emails are not deleted from the server). Only reproducible in mailboxes with large numbers of emails. This does not happen with version 6.2.0.


For a more verbose version see
http://who-t.blogspot.com/2011/05/fedora-15-and-offlineimap-users-beware.html

Version-Release number of selected component (if applicable):
offlineimap-6.3.3-1.fc15.noarch

How reproducible:
Run offlineimap on a mailbox with several thousand emails, in my case the command was "offlineimap -f lists-xorg" on a mailbox with just over 25k emails. When I copied the messages of this mailbox into a "test" mailbox, I was unable to reproduce the bug though. I can reproduce it in at least two other live mailboxes though.

Additional info:
In the log file attached, I noticed that all emails deleted have a UID of 32768 or greater. This may be a hint. I haven't checked if this is the case with other mailboxes, right now I'm rather hesitant to keep running this command without guidance on what exact commands to run.

Also note that the log file is shortened, I've tried to leave enough in to make it sensible without cutting anything of value. The original is over 8 MB. Places where information was cut are marked with [...]
Comment 1 Christoph Höger 2011-05-31 04:58:54 EDT
Hi,

there are many possible sources of that error.

In particular: Do you know the mailserver in place? Are the emails still there with another uid? (Some servers do not really give unique ids)? What happens if you sync with offlineimap to an empty directory, are those mails actually downloaded?
Comment 2 Peter Hutterer 2011-06-01 02:21:19 EDT
(In reply to comment #1)
> Do you know the mailserver in place?
The mail server is dovecot 1.2.8

fwiw, this setup hasn't changed in ages, neither the server nor the local offlineimaprc.

> Are the emails still there with another uid? (Some servers do not really give unique ids)?

I'm not quite sure how to check if the emails are still there with another UID, any instructions would be appreciated.

>  What happens if you sync with offlineimap to an empty directory, are those mails actually downloaded?

Yes, it syncs. Seems to have downloaded all emails as required. But when I ran the same command again (offlineimap -f lists-xorg) on the just-downloaded folder it deleted a number of them again: Deleting 5268 messages (32891, ...

I have the logs for the second run (was silly enough to overwrite the log from the empty sync) but I don't see any real difference to the original one, seems to be just more of the same.
Comment 3 Christoph Höger 2011-06-01 04:30:48 EDT
Ok dovecot is known to be very reliable and works well with offlineimap. 

> I'm not quite sure how to check if the emails are still there with another UID,
any instructions would be appreciated.

You could access the server and simply grep your mailbox's uidlist for the deleted messages and also search for duplicates.
(see http://wiki1.dovecot.org/MailboxFormat/Maildir#IMAP_UID_mapping for details)

The question here is, why offlineimap downloads the messages first and _then_ deletes them. Somehow we need to nail the actual problem (aka, what makes those 5268 messages different from the rest), so we can inform upstream.
Comment 4 Peter Hutterer 2011-06-01 18:27:24 EDT
Created attachment 502395 [details]
dovecot-uidlist

I'll just attach the whole list here. UID list and the filenames are unique across the file and I'm having a hard time figuring out what makes 32891 special (or 32768 in the previous run).
Comment 5 Sebastian Spaeth 2011-06-06 15:39:40 EDT
Thanks for the detailed IMAP log. This is much appreciated, and it helps for sure.

OfflineImap has recently switched to IMAPLIB2, in order to be able to support IMAP IDLE and other cool stuff. But your log seems indicate that it eats data.

For example the email server sends 

untagged_responses[FETCH] 5399 += ["16222 (FLAGS (\Seen) UID 32749)"]

 But the subsequent 

 _get_untagged_response(FETCH) => ['1 (FLAGS (\\See......

does not contain the very UID, and we subsequently try to delete it locally as we think it's gone from the server. I am in contact with the imaplib2 author and will try to get this debugged ASAP.
Comment 6 Sebastian Spaeth 2011-06-07 05:20:45 EDT
Hi Peter, I am in contact with the imaplib2 author and he would like to investigate this. Despite the great IMAP log, there is some stuff snipped out in the [...] part. Do you still have the full log and would you be willing to share it with me (or directly the imaplib2 author)? You ca reach me at sebastian@sspaeth.de. Thanks
Comment 7 Sebastian Spaeth 2011-06-08 10:17:19 EDT
Just to let you know that we are on it:

We fetch a list of UIDs from the server, but the list must currently not
be interrupted with unrelated responses.

THIS is send right after one of "missing UIDs"

DEBUG[imap]:   26:17.31 who-t.net reader < * OK Searched 43% of the mailbox,
ETA 0:12\r\n

so this series (handler log):
[FETCH] 10821 += ["10822 (FLAGS (\Answered \Seen) UID
27260)"]
[OK] 0 += ["Searched 43% of the mailbox, ETA 0:12"]
[FETCH] 0 += ["10823 (FLAGS (\Answered \Seen) UID 27261)"]

is broken into several "Responses" (note the FETCH restarts counting at
0). It breaks the series of untagged_responses just as the imaplib2 author had
suspected. Fixing this will require to fix imaplib2 and bundle a new version in a new version of offlineimap.
Comment 8 Sebastian Spaeth 2011-06-10 07:55:31 EDT
Just FYI, a fix is in the current "master" tree and *I* would recommend to upgrade any fedora package to use the latest master HEAD as soon as possible to avoid further potential data loss by others.

Thanks for raising this and providing us with detailed logs. It was a pleasure to fix this bug :-).
Comment 9 Peter Hutterer 2011-06-21 00:32:17 EDT
Christoph, are you going to submit new packages or do I have to try from git?
Comment 10 Christoph Höger 2011-06-21 04:07:52 EDT
You can try it from git for now. I am going to update to 6.3.4 ASAP once it is out. Since offlineimap is a python package, I do not see much sense in building -rc*.rpms
Comment 11 Peter Hutterer 2011-07-10 19:18:10 EDT
btw, I can confirm that v6.3.4-rc3 from git has fixed this bug.
Comment 12 Sebastian Spaeth 2011-07-11 03:21:02 EDT
Good to hear that.
Comment 13 Amit Shah 2011-10-13 04:08:58 EDT
Can you please update to the latest git snapshot if the release hasn't been made yet?
Comment 14 Amit Shah 2011-10-13 04:11:07 EDT
6.3.4 has been released on August 19 according to the git tag upstream.  Please update on F15 and 16.
Comment 15 Rahul Sundaram 2011-10-13 05:18:17 EDT
Christoph Höger,  do you need help with the update?  I can push an update myself if needed.
Comment 16 Fedora Update System 2011-10-19 17:14:32 EDT
offlineimap-6.3.4-1.fc16 has been submitted as an update for Fedora 16.
https://admin.fedoraproject.org/updates/offlineimap-6.3.4-1.fc16
Comment 17 Fedora Update System 2011-10-19 17:23:36 EDT
offlineimap-6.3.4-1.fc15 has been submitted as an update for Fedora 15.
https://admin.fedoraproject.org/updates/offlineimap-6.3.4-1.fc15
Comment 18 Fedora Update System 2011-10-19 22:25:55 EDT
Package offlineimap-6.3.4-1.fc16:
* should fix your issue,
* was pushed to the Fedora 16 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing offlineimap-6.3.4-1.fc16'
as soon as you are able to.
Please go to the following url:
https://admin.fedoraproject.org/updates/FEDORA-2011-14630
then log in and leave karma (feedback).
Comment 19 Fedora Update System 2011-11-02 20:22:18 EDT
offlineimap-6.3.4-1.fc15 has been pushed to the Fedora 15 stable repository.  If problems still persist, please make note of it in this bug report.
Comment 20 Fedora Update System 2011-11-14 17:25:58 EST
offlineimap-6.3.4-1.fc16 has been pushed to the Fedora 16 stable repository.  If problems still persist, please make note of it in this bug report.

Note You need to log in before you can comment on or make changes to this bug.