Bug 1650665 - Copy messages within mail filters immediately
Summary: Copy messages within mail filters immediately
Keywords:
Status: CLOSED NEXTRELEASE
Alias: None
Product: Fedora
Classification: Fedora
Component: evolution-data-server
Version: 29
Hardware: x86_64
OS: Linux
unspecified
unspecified
Target Milestone: ---
Assignee: Milan Crha
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-11-16 19:01 UTC by Paulo Zanoni
Modified: 2018-11-27 07:37 UTC (History)
8 users (show)

Fixed In Version: evolution-data-server-3.30.3
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2018-11-27 07:37:42 UTC


Attachments (Terms of Use)
Filter (14.18 KB, text/html)
2018-11-16 20:36 UTC, Paulo Zanoni
no flags Details

Description Paulo Zanoni 2018-11-16 19:01:42 UTC
Description of problem:

After an upgrade to Fedora 29, my Evolution filters stopped working as intended. The "copy message to folder" rule option really seems broken, although the "move to folder" rule seems to work. Evolution definitely seems to interpet rules in a different way now.

I have a very complex set of rules. I use EWS to connect to an Exchange server, then my rules assign points to message (-3 to 3), copy messages to different *local* folders based on points and senders, then finally move messages to folders in the Exchange server. It is the "copy message to local folders" that doesn't seem to be working.

Here's a simplified version of my rule set:

1. If message is to me, copy to local/me, assign 3 points, set label "tome".
2. If message is to mailing list X, copy to local/list-x, assign 1 point.
3. If message is to mailing list Y, copy to local/list-y, assign 1 point.
4. If message has 1 point, move to exchange/lists, stop processing.
5. If message has 3 points, move to exchange/tome, stop processing.
6. Apply to all messages: copy to local/me, move to exchange/tome, stop processing. (this to applies to anything without 1 or 3 points)

I did check the output of "CAMEL_DEBUG=filters evolution" and everything that it prints seems to be the correct thing. The problem is that messages simply do not get copied, although the log says they were copied.

I also tried adding a few rules as a way to debug things, and to my surprise adding new debug rules made the other non-related rules simply start working at some point. Adding even more rules made it stop working again. So this doesn't seem to be a trivial "copying to local doesn't work", but more of "something in my rules makes copying to local stop working, but it works if I add even more rules". Evolution definitely seems to be confused by the rules.

So what I did is: I simply deleted every filter and recreated everything from scratch. Now I'm in a state where messages get copied properly, but if I change my rules things may simply stop: for example, when I add the debug rules in my next paragraph, my other rules stop working.

One thing that I noticed while debugging is that how labels get applied is different on F29: if I set a label *after* copying a message, the copied message has the label. This was not the case on F28 and definitely contradicts Evolution documentation [0].

[0]: https://help.gnome.org/users/evolution/stable/mail-filters-not-working.html.en

TL;DR:
- Filters were working nice on F28, after an upgrade to F29 I don't get my messages copied from the EWS account to my local folders anymore.
- Filter log messages say the messages are being copied, when in fact they're not
- Adding more filter rules made things start working somehow.
- Filters definitely behave differently now, as can be demonstrated by setting a label after copying a message: the copied message also has the label.

Version-Release number of selected component (if applicable):

[user@machine ~]$ grep PRETTY /etc/os-release 
PRETTY_NAME="Fedora 29 (Workstation Edition)"
[user@machine ~]$ rpm -qa | grep -i evolution | sort
evolution-3.30.2-2.fc29.x86_64
evolution-data-server-3.30.2-1.fc29.x86_64
evolution-data-server-langpacks-3.30.2-1.fc29.noarch
evolution-ews-3.30.2-1.fc29.x86_64
evolution-ews-langpacks-3.30.2-1.fc29.noarch
evolution-help-3.30.2-2.fc29.noarch
evolution-langpacks-3.30.2-2.fc29.noarch


How reproducible:
Always, but changing rules affects reproducibility in a non-trivial way.


Steps to Reproduce:
1. Have working filters
2. Upgrade to F29
3. Notice non-working filters.
4. Also see the files I'll attach.

Actual results:
- Messages don't get copied to local folders anymore.
- Labels being set after a rule that copies the message get retroactively applied to the message that was copied before.


Expected results:
- Copying messages should actually copy the messages.
- If a rule copies a message somehwere, then the next rule sets a label, the label should not appear in the copy that was made by a previous rule.

Additional info:
- I'll attach filters.xml files later, explaining which one is which.

Comment 1 Paulo Zanoni 2018-11-16 20:36:58 UTC
Created attachment 1506549 [details]
Filter

Here's the filter I'm currently using (this is not what I was using on F28, since I deleted all my F28 filters and recreated them on F29 in an attempt to see if the problem went away). I manually edited the filter to anonymize it so I won't leak anything I shouldn't. Hopefully I was able to just edit the domains and names.

So the thing about this filter is,  if I use it:

- Messages that arrive on "0 debug copied" have the "important" label, while they really shouldn't because they were moved to "0 debug copied" in a previous rule.

- Messages that were supposed to arrive on "0 - In" (local) simply never arrive it. But they do get copied to "0 Inbox" (exchange) folder with the "meu" label, and the debug logs do say they were copied.

- The first 3 rules are created for debugging. They are supposed to be self-contained and not interfere with the other rules due to the "stop" rule that gets applied to those messages and the fact that nothing else acts on score -2 or the xdebugx subject line. The interesting thing is that if I remove those rules, then the messages that were supposed to arrive "0 - In" (local) now will start arriving it! Although they will arrive with the "meu" label, which is something they shouldn't.

I hope that helps trying to reproduce the problem.

Thanks in advance.

Comment 2 Milan Crha 2018-11-19 10:28:36 UTC
Thanks for a bug report. The 3.30.x has a change in filtering, where it piles copy/move operations into bulks and does that at the end, not after each message. It speeds filtering significantly. On the other hand, it misbehaves for your use case, which is completely correct (your use case is completely valid, not the broken behaviour).

As long as the move-to-folder also means to stop processing of other rules, it might be sufficient to do copy-to-folder immediately and bulk only the move-to-folder.

Comment 3 Milan Crha 2018-11-19 11:03:55 UTC
Here [1] is a test build of evolution-data-server, which does message copy immediately, not in a batch with message move. That should address all the above issues.

Could you give it a try, please?

You can download evolution-data-server and evolution-data-server-langpacks packages from [1] and then run as root:

   $ dnf update ./evolution-data-server*.rpm

from a directory where you saved the packages. Then it's enough to run evolution as a regular user.

[1] https://koji.fedoraproject.org/koji/taskinfo?taskID=30993508

Comment 4 Paulo Zanoni 2018-11-19 18:36:38 UTC
Hello

I installed your package and after a quick test it seems to be working as intended.

That said, I'm almost not going to use this machine this week, so I'll only have a more certain confirmation that everything is indeed working next week, once I go back to heavy emailing.

Feel free to push the update as-is and close the bug if you want. I can open another ticket (or reopen this one) if the problem remains.

Thanks,
Paulo

Comment 5 Milan Crha 2018-11-19 20:05:53 UTC
Thanks for a quick response. If you do not mind, then I'll wait for the next week and the heavier filtering. The next upstream release is planned on December 10th, thus there is plenty of time to confirm the fix.

Comment 6 Paulo Zanoni 2018-11-26 23:14:03 UTC
After another day of usage, the filters seem to be working fine.

A problem I'm observing is that maybe there's some memory leak going on. After just a few hours of usage Evolution is already eating 17% of my 8gb of memory. I can't confirm that this is caused by the version that fixed the filters or if this is already in the F29 version of the package (I'm not very willing to go back to the plain F29 version because having broken filters is a major issue I want to avoid). If you have more packages available to test, feel free to submit them. I'll keep monitoring Evolution memory usage for now.

Thanks a lot for the fix,
Paulo

Comment 7 Milan Crha 2018-11-27 07:37:42 UTC
Thanks for the confirmation. I committed the change upstream for 3.31.3+ [2] and for 3.30.3+ [3].

With respect of the possible memory leak, better to start a new bug report for it, to not mix different things in one report (I doubt the memory leak is caused/introduced by this change).

Valgrind can be used to debug memory leaks. The set of commands look like this:

   $ export G_SLICE=always-malloc
   $ export GIGACAGE_ENABLED=0
   $ valgrind --leak-check=full --num-callers=30 --show-leak-kinds=definite \
       evolution &>log.txt

but make sure you've installed debuginfo and debugsource packages for evolution-data-server (from the [1] at comment #3) and at least for other evolution packages you've installed (rpm -qa | grep evolution) of exactly the same version as those binary packages, thus the valgrind log is usable. It's rather useless without those debug symbols.

Please note that valgrind slows things down significantly and causes higher CPU usage, due to all the memory checking, thus it's not meant for any longer use, at least in case of Evolution. The longer you make it run the better, but I fully understand that the slowness will be so huge that you might consider Evolution under valgrind useless.

[2] https://gitlab.gnome.org/GNOME/evolution-data-server/commit/63741937d
[3] https://gitlab.gnome.org/GNOME/evolution-data-server/commit/c0dc8c5d1


Note You need to log in before you can comment on or make changes to this bug.