Bug 183219

Summary: checking mail now seems to use SELECT instead of STATUS
Product: [Fedora] Fedora Reporter: David Woodhouse <dwmw2>
Component: evolution-data-serverAssignee: Matthew Barnes <mbarnes>
Severity: medium Docs Contact:
Priority: medium    
Version: 10CC: fortran, frank, mattdm, mcrha, stu
Target Milestone: ---Keywords: Regression
Target Release: ---   
Hardware: All   
OS: Linux   
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2009-06-29 02:23:17 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Description Flags
Partial attempt at a fix.
This time without the most obvious bugs. none

Description David Woodhouse 2006-02-27 16:19:57 UTC
In FC4, Evolution would use STATUS to check for mail in folders without actually
opening them. It did this poorly, as is the subject of other bugs. But now in
rawhide (evo-data-server 1.5.91) it's actually got worse -- it now actually
opens every folder by using the SELECT command, instead of just using STATUS.

Once it's changed the open folder by issuing SELECT, it then needs to re-open
the folder which _should_ be open, and needs to re-fetch all the flags for that
folder. That's amazingly inefficient (or just broken, if it doesn't do it).

Comment 1 David Woodhouse 2006-02-27 19:24:55 UTC
OMFG it seems not only to be using SELECT but _also_ actually fetching all
headers for all mails in all folders. That's just insane -- adding to FC5Blocker.

Comment 2 David Woodhouse 2006-02-27 19:41:19 UTC
It's taken ten minutes so far, and it's only on the third folder of about
30-odd. What used to be a two-line transaction with the mail server (STATUS and
the response) is now a multi-megabyte download taking many minutes. 

I think I'll go home and come back in the morning to see if it's finished
starting up. This needs to get fixed if we're not going to just drop Evolution
from the distribution. 

Has anyone actually been using Evolution with IMAP recently? I'd have kept a
closer eye on it if I'd known it wasn't getting any testing.

Comment 3 David Woodhouse 2006-02-28 13:32:14 UTC
It's also fetching all flags each time it selects each folder, so even though it
is actually vaguely responsive after leaving it overnight, it's still a massive
regression from before. It's hardly usable.

The patches at
http://david.woodhou.se/evolution-2.5.91-check-only-active-folders.patch and
do help somewhat, by making it do this only for _active_ folders, not for
_every_ folder including multiple years worth of mail archives -- but it's still
very broken.

Comment 4 David Woodhouse 2006-02-28 15:04:36 UTC
Created attachment 125398 [details]
Partial attempt at a fix.

This goes some way to fixing the problem, by making it use STATUS for any
non-current folder instead of SELECT and completely updating the folder info.

It's not sufficient though -- the IMAP code's idea of 'current_folder' isn't
necessarily always correct, and the unseen counts aren't being shown even for
the non-current folders, when the STATUS command observes that they've changed.
It may be that the GUI code is fetching the counts from the cached folder info,
not just the summary which my new function is updating. 

Still investigating...

Comment 5 David Woodhouse 2006-02-28 15:33:53 UTC
Created attachment 125400 [details]
This time without the most obvious bugs.

This fixes the obvious bugs in the first attempt, but the behaviour is still
the same.

Comment 6 David Woodhouse 2006-02-28 17:19:59 UTC
This is probably the right approach, but there are two problems yet to be solved.

Firstly, the GUI code is probably using camel_folder_get_unread_message_count()
to  get the unread message count, and that's ignoring the summary info which my
patch updates -- it's making it up from the cached message index. We should
probably fix that in imap_getv(), and then the message counts should work.

Secondly, the check against imap_store->current_folder isn't the right thing to
do. What we actually want to do is use SELECT if it's a folder for which the GUI
actually cares about the message index, and use STATUS only when the GUI cares
only about the unseen count.

The GUI can have more than one folder 'open' at a time, in multiple windows, and
it can use some folders as source for vFolders, etc. -- all these folders would
fall into the first category where we should use 'SELECT'.

We should probably add a folder flag to show that the GUI has a folder 'open',
and then the code in my patch can use SELECT for any folder which is 'open', and
STATUS for any folder which is not.

Something for discussion with evo hackers upstream. I lack the patience for such

Comment 7 David Woodhouse 2006-03-15 22:06:20 UTC
This bug is still present in what looks like it's going to be the final FC5
release. It means that evolution is fairly much unusable on remote mailservers
with any sizeable amount of mail.

Is there any prospect of a fix? 

Could we revert to Evolution 2.4? Does that predate the changes which broke it?

Comment 8 Matthias Clasen 2006-07-26 01:14:57 UTC
What is the status of this bug ?
It still sits on the FC5 Blocker, which does not do it any good.
Does it need to be moved to FC6Target or FC6Blocker ?

Comment 9 David Woodhouse 2006-07-26 01:34:50 UTC
We need an erratum for FC5 ASAP, and it should also be on FC6Blocker.

Comment 10 Matthew Barnes 2006-07-26 01:49:03 UTC
I could use some technical assistance with this one, as it's pretty far over my
head at this point.  I'm afraid my knowledge of IMAP is only elementary until I
can find time to study it further.

David, are any of the above patches still applicable?

Comment 11 David Woodhouse 2006-07-26 01:57:23 UTC
Applicable still as a proof of concept, yes, but not really ready for submission.

At one point, a Novell Evolution hacker (shres) picked them up and was trying to
beat them into shape. He got laid off though, and the patches are apparently
lost on his old workstation.

The IMAP part of it isn't the hard part -- it's making Evolution deal with the
fact that it doesn't have all the flags and headers, for folders which aren't
used as vFolder sources. The blocker is the flag which marks a folder as 'open'
on the GUI side, meaning that we actually _do_ need all headers &c for it.

Comment 12 Matthew Barnes 2006-07-31 01:48:03 UTC
Realistically, I doubt a solution is going to emerge in time for Fedora Core 6.
 It sounds like this is going to require a large chunk of time to understand the
problem, develop a solution, and test it well enough to release.  Not to mention
upstream coordination.

While I think most Evolution users would acknowledge that the IMAP support needs
improvement, it seems to work well enough for many -- including myself -- to use
on a daily basis.  So I think my time would be better spent fixing crashers and
security vulnerabilities before FC6 is released.

So unless someone else can devote time to this, I recommend that this be moved
to FC6Target instead of FC6Blocker.

Comment 13 Milan Crha 2007-10-22 11:24:23 UTC
Hi David, can you review this with latest Evolution (2.12.1) in F8T3, please?
As far as I know, there has been done some changes in imap provider of camel,
even not many. Also, it can depend on a provider you are running, there are 3
imap providers in camel. (But I don't expect you use other than that default.
Just in case.) Also, can you check your setup in Edit->Preferences, Edit imap
account -> tab "IMAP Headers"? Sometimes can help in tab "Receiving Options"
disable "Check for new messages in all folders", but I don't think we want to go
in this way.

Comment 14 Frank Crawford 2008-03-22 12:28:35 UTC
Folks, has any activity gone on with this?  This appears to be an issue I am
having with Evolution since FC6 sometime, up to F8, with Microsoft Exchange as
an IMAP server and evolution-2.12.3-3.fc8.

The mailbox size is > 2G (many, many individual messages) and over a hundred
separate folders.  The "Check for new messages in all folders" is set to off,
but the problem still occurs.

Running gdb appears to indicate that evolution is scanning inactive mailboxes
for something, and eventually hangs for long periods of time.

I can supply more details and gdb traces if is will help.

Comment 15 David Woodhouse 2008-05-13 15:58:24 UTC
Hm, why moved to Fedora 9? This should stay on rawhide; it's been open for over
two years already; there's no reason to believe it'll be fixed by F10, is there?

Comment 16 Matthew Barnes 2008-05-13 18:14:16 UTC
It was just part of a mass move.  I'm fine with moving this one back to Rawhide.

Comment 17 Bug Zapper 2008-05-14 02:06:06 UTC
Changing version to '9' as part of upcoming Fedora 9 GA.
More information and reason for this action is here:

Comment 19 Bug Zapper 2008-11-26 01:48:35 UTC
This bug appears to have been reported against 'rawhide' during the Fedora 10 development cycle.
Changing version to '10'.

More information and reason for this action is here:

Comment 20 Matthew Barnes 2009-06-29 02:23:17 UTC
Closing this since an investigation is already well underway upstream.
Please see [1] for further updates.

[1] http://bugzilla.gnome.org/show_bug.cgi?id=336076