Bug 1618662 - Evolution takes ages retrieving IMAP messages before I can do anything
Summary: Evolution takes ages retrieving IMAP messages before I can do anything
Keywords:
Status: CLOSED NOTABUG
Alias: None
Product: Fedora
Classification: Fedora
Component: evolution
Version: 28
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: Milan Crha
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-08-17 09:10 UTC by Louis van Dyk
Modified: 2018-08-22 09:10 UTC (History)
7 users (show)

Fixed In Version:
Clone Of:
Environment:
Last Closed: 2018-08-22 09:10:16 UTC
Type: Bug
Embargoed:


Attachments (Terms of Use)
Backtrace (23.56 KB, text/plain)
2018-08-21 00:54 UTC, Louis van Dyk
no flags Details
Output from starting from the command line (3.48 MB, application/x-gzip)
2018-08-21 01:01 UTC, Louis van Dyk
no flags Details
Image after inbox-searchfolder-inbox (172.16 KB, image/png)
2018-08-21 01:03 UTC, Louis van Dyk
no flags Details
Image top output (185.31 KB, image/png)
2018-08-21 01:03 UTC, Louis van Dyk
no flags Details
Image after inbox and open another message (271.22 KB, image/png)
2018-08-21 01:05 UTC, Louis van Dyk
no flags Details

Description Louis van Dyk 2018-08-17 09:10:01 UTC
Description of problem:
I host my mail server on my home PC using sendmail.  My spool file is rather large (about 15 GB).  On this same PC I open Evolution.  Lately, it before I can do anything else, I have to wait for it to "retrieve" messages from my IMAP mailbox.  At the bottom it displays this message:
Retrieving message "nnnnnnnn" in "Home : INBOX"
It takes anything from 1 second to 40 seconds and then moves on to another message number.  Sometimes in frustration I click the red dot to stop it - where it stays cancelling for a while first.  Today it eventually finished, and filtered the messages as normal.
It appears to me to be worse than it used to be because it didn't used to get in my way before.  I'm not sure what's changed.
I also don't understand why it takes 40 seconds to retrieve a message from the same PC it's running on - there is absolutely no network involved to cause any latency.  My /var/spool/mail/louis file is on a SSD too!

Version-Release number of selected component (if applicable):
evolution-3.28.5-1.fc28.x86_64
evolution-data-server-3.28.5-1.fc28.x86_64
evolution-data-server-langpacks-3.28.5-1.fc28.noarch
evolution-ews-3.28.5-1.fc28.x86_64
evolution-ews-langpacks-3.28.5-1.fc28.noarch
evolution-help-3.28.5-1.fc28.noarch
evolution-langpacks-3.28.5-1.fc28.noarch


How reproducible:
Typically happens when I open Evolution or sometimes when I click on one of the search VFolders at the bottom.


Steps to Reproduce:
1. As above
2.
3.

Actual results:


Expected results:
I expect the caching process to be much quicker, especially when running on the same machine as the mail daemon.  If Evolution allowed me to do stuff while it was syncing itself in the background, that too would be preferable.

Additional info:

Comment 1 Milan Crha 2018-08-20 16:39:53 UTC
Thanks for a bug report. It it's IMAP, then open the account Properties and in Receiving Options change the number of concurrent connections, where the default is like 3, which means up to three connections can be made to your (local) IMAP server/service, which allows some parallelism.

I'm afraid the long delay is on the server/service side, evolution just waiting for the response, but I can be wrong. Try to run evolution from the command line as this:

   $ CAMEL_DEBUG=imapx:io evolution

and then watch what it does when it is stuck on the message download. Preferably do that before changing the concurrent connections, because I'd like to have this in the original state. If it is related to message filtering, like moving it to the correct folder, then the upcoming 3.30 has an improvement in the message filtering when it comes to move to some other folder, which speeds the moves significantly.

Back to your issue: if you have installed also evolution-data-server-debuginfo package, of the same version as the evolution-data-server package, then you can see what evolution does (ideally also with evolution-debuginfo installed), when you run this gdb command:

   $ gdb --batch --ex "t a a bt" -pid=`pidof evolution` &>bt.txt

Please check the bt.txt for any private information, like passwords, email address, server addresses,... I usually search for "pass" at least (quotes for clarity only), before sharing it anywhere.

This might shed a bit of light on it.

Comment 2 Louis van Dyk 2018-08-21 00:53:32 UTC
Interestingly, I had already set the number of connections to 5 some time back.  I have left it like that.

Attached will be the backtrace, a copy of the console when running it - I ran it twice - you will see the separations with a number of carriage returns, and a few screen shots.

The one line of weird characters was when I tried to scroll back with the mouse wheel when it was stuck downloading.

I've also given you a screenshot of what I see in the first time (opened, clicked on search folders and then on Inbox again), and in the second time (I only tried to open a message).  The message waited until all the retrieves were done before it showed any message text.

What DID strike me as interesting was that as I was looking at the messages it was stalling on, it seems that each time I open Evolution, it gets stuck on many of the same numbers each time.

Comment 3 Louis van Dyk 2018-08-21 00:54:35 UTC
Created attachment 1477384 [details]
Backtrace

Comment 4 Louis van Dyk 2018-08-21 01:01:59 UTC
Created attachment 1477386 [details]
Output from starting from the command line

Comment 5 Louis van Dyk 2018-08-21 01:03:12 UTC
Created attachment 1477387 [details]
Image after inbox-searchfolder-inbox

Comment 6 Louis van Dyk 2018-08-21 01:03:56 UTC
Created attachment 1477388 [details]
Image top output

Comment 7 Louis van Dyk 2018-08-21 01:05:03 UTC
Created attachment 1477389 [details]
Image after inbox and open another message

Comment 8 Milan Crha 2018-08-21 13:42:49 UTC
Thanks for the update. I see your IMAP folder has more than half million of messages, which is impressive and it looks like there is no problem with the IMAP as such. The backtrace shows that the refresh of the search folder is ongoing, it is opening each message to check the expression:

   (and
      (match-all (header-starts-with ""  " "))
   )

And it's that "header-starts-with", which requires the message to be loaded. Imagine evolution loading each of those half million messages due to it, whenever a refresh of the search folder is invoked.

What is that search folder is supposed to do, please? Basic "Match All" condition simply selects every message in the folder, without a need to load it into the memory.

Comment 9 Milan Crha 2018-08-21 13:46:00 UTC
(In reply to Milan Crha from comment #8)
> I see your IMAP folder has more than half million of messages, ...

Oops, no, it's a UID, not a sequence number. The Inbox has 96410 messages, not half of the million. Still, it's plenty of messages to be loaded into the memory.

Comment 10 Louis van Dyk 2018-08-21 14:29:27 UTC
:)  Thank you!!  

The header is to get rid of some SPAM I get.  Senders that start their name with a space are always SPAM and I get them enough to be a nuisance, and hence the rule.  It only needs to do that on inbound messages though ...

On looking at it, I recognise that I should have made that SPECIFIC header = From, and not ANY header.  I will fix that and test again.  

I later placed a Message Filter on all incoming mail like that (using specific header) and so I actually no longer need this particular search folder as it's now expected to always be empty.

On the topic of SPAM, I have Spamassassin installed and have linked the Milter to Comodo Antivirus for Linux.  Yet, I still get too much of it.  Do you know of a good guide for Spamassassin - I suspect my installation attempt may be flawed.  I keep dumping stuff into Evolution's Junk folder, and hoping that Spamassassin checks it out ...

Comment 11 Milan Crha 2018-08-22 09:10:16 UTC
(In reply to Louis van Dyk from comment #10)
> On the topic of SPAM, I have Spamassassin installed and have linked the
> Milter to Comodo Antivirus for Linux.  Yet, I still get too much of it.  Do
> you know of a good guide for Spamassassin - I suspect my installation
> attempt may be flawed.  I keep dumping stuff into Evolution's Junk folder,
> and hoping that Spamassassin checks it out ...

See https://help.gnome.org/users/evolution/stable/mail-spam.html.en

Some users claim spamassassin is better for servers, and sometimes also used by the servers, thus they use Bogofilter on the local machine instead.

You should install evolution-spamassassin (or evolution-bogofilter) in order to train it when you mark messages as Junk or Not Junk. With those installed you'll see appropriate options in the Edit->Preferences->Mail Preferences->Junk tab, at the bottom, "Junk filtering software" option, or the specific options for the one junk filter plugin you've installed/selected.

I'm closing this bug report for now, but feel free to update it. I can reopen it when needed. Thanks.


Note You need to log in before you can comment on or make changes to this bug.