Red Hat Bugzilla – Bug 1306846
[IMAPx] Message download in chunks sometimes broken (for GMail)
Last modified: 2016-02-16 09:50:21 EST
Created attachment 1123301 [details]
How a bad message looks like.
Description of problem:
Evolution occasionally misses email content from Gmail.
Version-Release number of selected component (if applicable):
Steps to Reproduce:
1. Set up Evolution to use IMAP for Gmail.
1. Receive email with Evolution.
From time to time the email title and sender is shown, but the header and content is empty. If the email has an attachment, the attachment (Base64 encoded) becomes the body of the email.
Email shows up correctly.
I highly suspect this is related to a race condition in threading. Does evolution download the message in threads, or somewhere try to decode different parts of the message in threads?
I see this in Fedora 22 version of Evolution as well, but not in the Fedora 21 version of Evolution.
Also, the local cached version of the message is broken too. I can see the email from Gmail web client correctly.
Evolution will fetch email in chunks, so that it doesn't hog the IMAP connection with a huge download and can still do other things while it's downloading. There have been issues with broken IMAP servers failing to properly handle the "send 20KiB from offset 40KiB" requests, especially Microsoft Exchange which isn't really an email server at all and is actually faking the RFC5322-based mail content on the fly. I wouldn't expect Google to be that broken though.
Can you run evolution with CAMEL_DEBUG=imapx and capture the output when the offending message is being downloaded? And attach the actual .mbox file of a bad message, please? Preferably from the original download in your ~/.cache/evolution/mail directory, since I think Evolution still corrupts the whitespace in the headers when you 'save to file'.
Thanks for a bug report. Fedora 21 has evolution-data-server 3.12.11, Fedora 22 has evolution-data-server 3.16.5. Both are using the same way of message download, in chunks of ~32KB, if the message size is larger than this. Fedora 23 uses a similar code, but not exactly the same. I'm wondering whether the reason could be related to some background library.
Anyway, using CAMEL_DEBUG=imapx will make it too chatty, it would be enough to catch the io part only, which is when you run evolution as this:
$ CAMEL_DEBUG=imapx:io evolution &>log.txt
To be able to re-download the message from the server you need to delete it from the cache. The delete can be done also when the evolution is running, just select another message, delete the broken one, and select the new one again. The interesting part of the log is the one when you select the broken message again, when it is downloading. Ideally disable any other IMAP account, because the logging will be done for each of them, thus to not overlap the output with anything unrelated.
> Evolution will fetch email in chunks, so that it doesn't hog the IMAP
> connection with a huge download and can still do other things while it's
The part after 'so' from the above quoted text is not true anymore (3.18.x speaking). messages are downloaded synchronously from the server in one connection, while other tasks can be done in another connection.
I would try to decrease concurrent connections, in account Properties, Receiving Options tab. It can have its influence on this thing. Use like 2 connections for the Google server.
Seen exactly same issue against imap gmail with an older version of evolution (3.8.5) ; only 2 cases , one having an attachment base64 encoded in the body (long download time, apparently "freezing" evolution).
I hope I was able to capture the log file as described (see attached log2.txt)
I was trying to purge the cache of the bad message, but could not locate the message id. Is there a way for me to find the message id from the UI?
Created attachment 1123641 [details]
Bad email from cache.
I was able to find the message with grep.
Created attachment 1123643 [details]
I could not reproduce this particular message.
I tried 5 times with the clear cache method, and each time the message was received correctly.
I will keep evolution running in log mode and upload the log when another message is hit.
Meanwhile, an interesting observation is that the correct message is of the same length. I've attached the file too.
The zeros in the corrupt file is only in range:
0 ~ 00040000
or the first 262144 bytes. The rest of the two files appear to be identical.
Thanks for the update. I just added  an option for the IMAPx accounts named "use-multi-fetch", which defaults to 'true' (current behaviour). Set it to 'false' to disable message download in chunks. You can do that by:
a) locate correct .source file eithe rin ~/.config/evolution/sources or
~/.cache/evolution/sources/ which references the right server and contains
section [Imapx Backend]. Locate in this section key named UseMultiFetch or
add a new one in that section, if not there yet, with value 'false', thus
it'll look like:
b) restart evolution, or ideally also other background evolution processes,
for example evolution-source-registry (re-login or restart will do it most
Since then the messages will be always downloaded in one call, not in chunks.
It's not meant to be a fix for this issue, it's just a quick workaround for it. The option is available since 3.19.91+ of the evolution-data-server.
(In reply to Herve Lemaitre from comment #3)
> I hope I was able to capture the log file as described (see attached
Thanks. The log has missing lines with command:
UID FETCH 987 (BODY.PEEK<111.555>)
(where the '987' is a message UID and the "111.555" is a byte range of the message to download). A small-enough message, like less than 32KB, is downloaded at once, without the byte range, thus the command looks like this:
UID FETCH 987 (BODY.PEEK)
This is the same for both current git master and Herve's 3.8.5.
(In reply to Feng Yu from comment #7)
> I will keep evolution running in log mode and upload the log when another
> message is hit.
Thanks. It can be some thread interleaving, though 3.18.x (not 3.8.5) has this done synchronously, in one thread only. The log will show, I hope. My GMail account downloads messages fine, but I have slower connection, and/or I connect to a different GMail server, which can have influence too.
Please note there is not needed the whole log, I'd like to see only the "UID FETCH" with "BODY.PEEK" commands (as described above). Ideally keep the log for more/later digging from it, once you'll reproduce this. Thanks in advance.
I just noticed an upstream bug report  about the same. I'm closing this in favour of it, but feel free to update this bug report with more information, if you do not have an account in GNOME's bugzilla. Please do not reopen this bug report, I will provide information in the upstream bug report, if needed. If you can, then preferably join the upstream bug report.
I also suggest to update to the 3.18.5 release, because it can be also a problem with IDLE implementation in pre-3.18.5 of the evolution-data-server. I suggest to update to it once it is available (it had been released only yesterday). You can also try to disable "Listen for server change notifications" in the account Properties->Receiving Options tab, but you lose functionality, which is not always desired.