Bug 1306846 - [IMAPx] Message download in chunks sometimes broken (for GMail)
[IMAPx] Message download in chunks sometimes broken (for GMail)
Status: CLOSED UPSTREAM
Product: Fedora
Classification: Fedora
Component: evolution (Show other bugs)
23
Unspecified Unspecified
unspecified Severity unspecified
: ---
: ---
Assigned To: Milan Crha
Fedora Extras Quality Assurance
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2016-02-11 16:43 EST by Feng Yu
Modified: 2016-02-16 09:50 EST (History)
6 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2016-02-16 09:50:21 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
How a bad message looks like. (36.75 KB, image/png)
2016-02-11 16:43 EST, Feng Yu
no flags Details
Bad email from cache. (1.62 MB, application/octet-stream)
2016-02-12 16:23 EST, Feng Yu
no flags Details
Correct message. (1.62 MB, text/plain)
2016-02-12 16:29 EST, Feng Yu
no flags Details


External Trackers
Tracker ID Priority Status Summary Last Updated
GNOME Desktop 761096 None None None 2016-02-16 09:50 EST

  None (edit)
Description Feng Yu 2016-02-11 16:43:36 EST
Created attachment 1123301 [details]
How a bad message looks like.

Description of problem:

Evolution occasionally misses email content from Gmail.

Version-Release number of selected component (if applicable):
3.18.4

How reproducible:

Ocassionally

Steps to Reproduce:
1. Set up Evolution to use IMAP for Gmail.
1. Receive email with Evolution.
2.
3.

Actual results:

From time to time the email title and sender is shown, but the header and content is empty. If the email has an attachment, the attachment (Base64 encoded) becomes the body of the email.

Expected results:

Email shows up correctly.

Additional info:

I highly suspect this is related to a race condition in threading. Does evolution download the message in threads, or somewhere try to decode different parts of the message in threads? 

I see this in Fedora 22 version of Evolution as well, but not in the Fedora 21 version of Evolution. 

Also, the local cached version of the message is broken too. I can see the email from Gmail web client correctly.
Comment 1 David Woodhouse 2016-02-12 03:44:48 EST
Evolution will fetch email in chunks, so that it doesn't hog the IMAP connection with a huge download and can still do other things while it's downloading. There have been issues with broken IMAP servers failing to properly handle the "send 20KiB from offset 40KiB" requests, especially Microsoft Exchange which isn't really an email server at all and is actually faking the RFC5322-based mail content on the fly. I wouldn't expect Google to be that broken though.

Can you run evolution with CAMEL_DEBUG=imapx and capture the output when the offending message is being downloaded? And attach the actual .mbox file of a bad message, please? Preferably from the original download in your ~/.cache/evolution/mail directory, since I think Evolution still corrupts the whitespace in the headers when you 'save to file'.
Comment 2 Milan Crha 2016-02-12 05:16:28 EST
Thanks for a bug report. Fedora 21 has evolution-data-server 3.12.11, Fedora 22 has evolution-data-server 3.16.5. Both are using the same way of message download, in chunks of ~32KB, if the message size is larger than this. Fedora 23 uses a similar code, but not exactly the same. I'm wondering whether the reason could be related to some background library.

Anyway, using CAMEL_DEBUG=imapx will make it too chatty, it would be enough to catch the io part only, which is when you run evolution as this:

   $ CAMEL_DEBUG=imapx:io evolution &>log.txt

To be able to re-download the message from the server you need to delete it from the cache. The delete can be done also when the evolution is running, just select another message, delete the broken one, and select the new one again. The interesting part of the log is the one when you select the broken message again, when it is downloading. Ideally disable any other IMAP account, because the logging will be done for each of them, thus to not overlap the output with anything unrelated.

> Evolution will fetch email in chunks, so that it doesn't hog the IMAP
> connection with a huge download and can still do other things while it's
> downloading

The part after 'so' from the above quoted text is not true anymore (3.18.x speaking). messages are downloaded synchronously from the server in one connection, while other tasks can be done in another connection.

I would try to decrease concurrent connections, in account Properties, Receiving Options tab. It can have its influence on this thing. Use like 2 connections for the Google server.
Comment 3 Herve Lemaitre 2016-02-12 12:08:06 EST
hi
Seen exactly same issue against imap gmail with an older version of evolution (3.8.5) ; only 2 cases , one having an attachment base64 encoded in the body (long download time, apparently "freezing" evolution).

I hope I was able to capture the log file as described (see attached log2.txt)
Comment 5 Feng Yu 2016-02-12 16:21:20 EST
I was trying to purge the cache of the bad message, but could not locate the message id. Is there a way for me to find the message id from the UI?
Comment 6 Feng Yu 2016-02-12 16:23 EST
Created attachment 1123641 [details]
Bad email from cache.

I was able to find the message with grep.
Comment 7 Feng Yu 2016-02-12 16:29 EST
Created attachment 1123643 [details]
Correct message.

I could not reproduce this particular message. 

I tried 5 times with the clear cache method, and each time the message was received correctly.

I will keep evolution running in log mode and upload the log when another message is hit.

Meanwhile, an interesting observation is that the correct message is of the same length. I've attached the file too.
Comment 8 Feng Yu 2016-02-12 16:34:28 EST
The zeros in the corrupt file is only in range:

0 ~ 00040000

or the first 262144 bytes. The rest of the two files appear to be identical.
Comment 9 Milan Crha 2016-02-15 12:15:52 EST
Thanks for the update. I just added [1] an option for the IMAPx accounts named "use-multi-fetch", which defaults to 'true' (current behaviour). Set it to 'false' to disable message download in chunks. You can do that by:
a) locate correct .source file eithe rin ~/.config/evolution/sources or
   ~/.cache/evolution/sources/ which references the right server and contains
   section [Imapx Backend]. Locate in this section key named UseMultiFetch or
   add a new one in that section, if not there yet, with value 'false', thus
   it'll look like:
      UseMultiFetch=false
b) restart evolution, or ideally also other background evolution processes,
   for example evolution-source-registry (re-login or restart will do it most
   easily).

Since then the messages will be always downloaded in one call, not in chunks.

It's not meant to be a fix for this issue, it's just a quick workaround for it. The option is available since 3.19.91+ of the evolution-data-server.

[1] https://git.gnome.org/browse/evolution-data-server/commit/?id=6ace0dd
Comment 10 Milan Crha 2016-02-15 12:33:15 EST
(In reply to Herve Lemaitre from comment #3)
> I hope I was able to capture the log file as described (see attached
> log2.txt)

Thanks. The log has missing lines with command:
   UID FETCH 987 (BODY.PEEK[]<111.555>)
(where the '987' is a message UID and the "111.555" is a byte range of the message to download). A small-enough message, like less than 32KB, is downloaded at once, without the byte range, thus the command looks like this:
   UID FETCH 987 (BODY.PEEK[])

This is the same for both current git master and Herve's 3.8.5.

(In reply to Feng Yu from comment #7)
> I will keep evolution running in log mode and upload the log when another
> message is hit.

Thanks. It can be some thread interleaving, though 3.18.x (not 3.8.5) has this done synchronously, in one thread only. The log will show, I hope. My GMail account downloads messages fine, but I have slower connection, and/or I connect to a different GMail server, which can have influence too.

Please note there is not needed the whole log, I'd like to see only the "UID FETCH" with "BODY.PEEK[]" commands (as described above). Ideally keep the log for more/later digging from it, once you'll reproduce this. Thanks in advance.
Comment 11 Milan Crha 2016-02-16 09:50:21 EST
I just noticed an upstream bug report [1] about the same. I'm closing this in favour of it, but feel free to update this bug report with more information, if you do not have an account in GNOME's bugzilla. Please do not reopen this bug report, I will provide information in the upstream bug report, if needed. If you can, then preferably join the upstream bug report.

I also suggest to update to the 3.18.5 release, because it can be also a problem with IDLE implementation in pre-3.18.5 of the evolution-data-server. I suggest to update to it once it is available (it had been released only yesterday). You can also try to disable "Listen for server change notifications" in the account Properties->Receiving Options tab, but you lose functionality, which is not always desired.

[1] https://bugzilla.gnome.org/show_bug.cgi?id=761096

Note You need to log in before you can comment on or make changes to this bug.