Bug 1085750

Summary:	evolution using excessive network bandwidth on imaps connection
Product:	[Fedora] Fedora	Reporter:	Steve Whitehouse <swhiteho>
Component:	evolution	Assignee:	Matthew Barnes <mbarnes>
Status:	CLOSED UPSTREAM	QA Contact:	Fedora Extras Quality Assurance <extras-qa>
Severity:	unspecified	Docs Contact:
Priority:	unspecified
Version:	20	CC:	fabiano, lucilanga, mbarnes, mcrha, swhiteho
Target Milestone:	---
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2014-04-14 09:41:43 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Steve Whitehouse 2014-04-09 09:09:45 UTC

Description of problem:

After a recent upgrade from f14 to f20, I received a message from my ISP relating to exceeding my download limit for the month. The total downloaded was just over 17G in about 2.5 days (as I'd been away for some of the time).

A quick investigation with tcpdump showed that evolution was responsible for almost all of this. Bearing in mind that I have to pay quite expensive fees for bandwidth during peak times (8am to 6pm) this means that evolution is not usable as an email client for me until this is fixed.

I'm using evolution with two imaps mail accounts. One has quite a large number of messages, circa 300k whereas the other is relatively small.

Having asked on the irc channel yesterday, I was pointed at some instructions to allow debugging of the imaps connection, and this appears to show that when evolution is "idle" it fills the time by requesting sets of summary info (size and headers) for bunches of messages. I ran a test yesterday, outside of peak hours, and in a few minutes evolution had downloaded over 300G of mainly
message headers.

Taking a random message number, and grepping the log file, it appears that it is requesting the same message over and over again. While it is sending these requests, it is filling my broadband connection to capacity.

If we assume that the average message has 1k of headers, then 300k messages should result in approx 307M of data, so 300G seems excessive. I was measuring the traffic using some iptables accounting rules I'd added specially to track it.

Also, just to confirm, the upgrade from f14, was basically a reinstall, but keeping /home, so there is no chance of old packages still sitting around.

Once I've checked through the log file generated by evolution for any private info, then I'll attach that to this bug.

Version-Release number of selected component (if applicable):

How reproducible:

100%

Steps to Reproduce:
1. Start evolution
2.
3.

Actual results:

Network maxed out with repeated requests for headers & size of messages.

Expected results:

Some small requests to update the local state according to the remote mail box, and then the network connection should be more or less idle if there is no local activity, and little email being delivered to the remote mail box.

Additional info:

Comment 2 Matthew Barnes 2014-04-09 12:14:26 UTC

It looks like you used CAMEL_DEBUG=imapx:command to generate the log file, as it's missing some critical details about the IMAP responses being received.

The constant refetching seems to be the result of the log message:

   "Eep, after QRESYNC we're out of sync."

Which (AFAICT) always follows messages about unseen flag mismatches:

   "Not updating unseen count for new message XXX"

Which can occur in the event of a stale UIDNEXT value.

I need to verify that your server is including an updated UIDNEXT value in response to a SELECT+QRESYNC command.

Can you generate a new log file using CAMEL_DEBUG=imapx:io,imapx:command ?

Comment 3 Steve Whitehouse 2014-04-09 12:32:55 UTC

Yes, that is not a problem, but I will wait until after 6pm, since it will cause a lot of network traffic.

Comment 5 Milan Crha 2014-04-10 06:11:52 UTC

(In reply to Matthew Barnes from comment #2)
>    "Eep, after QRESYNC we're out of sync."
> 
> Which (AFAICT) always follows messages about unseen flag mismatches:
> 
>    "Not updating unseen count for new message XXX"
> 
> Which can occur in the event of a stale UIDNEXT value.

I think it's obvious from Matthew's post, but just in case, as a workaround, disabling Quick Resync in account preferences may stop doing this.

Comment 6 Steve Whitehouse 2014-04-10 09:00:40 UTC

I will try that out later on (again after 6pm, for obvious reasons)

Comment 8 Milan Crha 2014-04-11 05:36:00 UTC

> So it appears thats not the solution :(

I see in the log that evolution recognized almost 470K messages in one of your folders and it begun to download message headers for it. These are used in the message list in the view (and many other places). It's only the initial download, once it is done the download should get down.

Comment 9 Steve Whitehouse 2014-04-11 07:45:22 UTC

Bearing in mind that there are only circa 300k message in that folder, if it has seen 470k messages, then it has downloaded some of them twice. The problem is that it does not stop downloading the messages :(

I did check that by taking a random message ID which I could see had been downloaded twice in evo3.log. I don't really want to let evolution run on for too long since it is costing me money each time I do this test, however if it is useful for debugging, I can provide a longer trace.

Comment 10 Milan Crha 2014-04-14 09:41:43 UTC

That's OK, there is no need for longer long. I just noticed that you are right, evolution downloads headers for messages twice, which is clearly wrong. I opened an upstream bug report [1] for this part.

I also filled bug [2] upstream, which is for the initial QResync issue.

I'm closing this bug report in a favour of the two upstream bugs. Feel free to CC there and watch any updates on it, and/or comment here with more concerns.

[1] https://bugzilla.gnome.org/show_bug.cgi?id=728167
[2] https://bugzilla.gnome.org/show_bug.cgi?id=728168

Comment 11 Steve Whitehouse 2014-04-14 10:32:24 UTC

Thanks - I've added myself to the cc list of those two upstream bugs