Bug 1565343 - Crash when meeting invitation iCalendar part could not be parsed
Summary: Crash when meeting invitation iCalendar part could not be parsed
Keywords:
Status: CLOSED NEXTRELEASE
Alias: None
Product: Fedora
Classification: Fedora
Component: evolution-ews
Version: 27
Hardware: x86_64
OS: Linux
unspecified
urgent
Target Milestone: ---
Assignee: Milan Crha
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
: 1571400 1577994 1605417 (view as bug list)
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-04-09 21:21 UTC by psg_nm
Modified: 2018-07-23 14:55 UTC (History)
4 users (show)

Fixed In Version: evolution-ews-3.28.2
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2018-04-18 09:14:03 UTC
Type: Bug
Embargoed:


Attachments (Terms of Use)
Evolution related messages from "journalctl -r" (46.57 KB, text/plain)
2018-04-09 21:21 UTC, psg_nm
no flags Details
Another 'journalctl -r' for experiment on April 10, 2018 (30.35 KB, text/plain)
2018-04-10 15:43 UTC, psg_nm
no flags Details
test-rh1565343.c (2.12 KB, text/plain)
2018-04-16 17:36 UTC, Milan Crha
no flags Details

Description psg_nm 2018-04-09 21:21:00 UTC
Created attachment 1419569 [details]
Evolution related messages from "journalctl -r"

Description of problem:

I have been using Evolution on my Fedora machine with Fedora 25, 26, and 27 for some time.  I updated my system on March 26, 2018 and Evolution started crashing after about 10-20 minutes of uptime.  I have been updating my machine since then with no improvement in behavior.  I am using the evolution-ews plugin to connect Evolution to a Microsoft Exchange server.

Version-Release number of selected component (if applicable):

evolution.x86_64                               3.26.6-1.fc27
evolution-data-server.x86_64                   3.26.6-1.fc27
evolution-data-server-langpacks.noarch         3.26.6-1.fc27
evolution-ews.x86_64                           3.26.6-1.fc27
evolution-ews-langpacks.noarch                 3.26.6-1.fc27
evolution-help.noarch                          3.26.6-1.fc27
evolution-langpacks.noarch                     3.26.6-1.fc27


How reproducible:

It seems to be very reproducible at this point.  I cannot run Evolution for more than 10-20 minutes before it crashes.

Steps to Reproduce:
1. Open Evolution with a connection to the Exchange server.
2. Any usage (no known specific operation seems to cause crash).

Actual results:

Evolution crashes with messages like (a few messages from journalctl -r that seemed especially relevant):

Apr 09 10:25:41 mymachine abrt-notification[5713]: Process 3437 (evolution) crashed in ews_folder_get_message_sync()
Apr 09 10:25:32 mymachine kernel: pool[5457]: segfault at 0 ip 00007effdd2b69fd sp 00007eff17ffe990 error 4 in libcamelews-priv.so[7effdd2a3000+2a000]
Apr 09 10:25:32 mymachine audit[3437]: ANOM_ABEND auid=1000 uid=1000 gid=1000 ses=2 subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 pid=3437 comm="pool" exe="/usr/bin/evolution" sig=11 res=1
Apr 09 10:25:18 mymachine evolution[3437]: g_string_insert_len: assertion 'len == 0 || val != NULL' failed
Apr 09 10:25:18 mymachine evolution[3437]: camel_internet_address_encode_address: assertion 'addr' failed

See the attachment for a more complete log.

Expected results:

Stable operation for days, of course.

Additional info:

My Exchange Inbox is quite large (almost 50,000 messages), which might be relevant.  The other piece of relevant information is that this seemed to occur after a "dnf update" I did on March 26th.  The update might be a coincidence or it might be relevant.  Here is the "sudo dnf history info" for the particular update that happened right before I started seeing the behavior:

$ sudo dnf history info 142
Transaction ID : 142
Begin time     : Mon 26 Mar 2018 12:26:55 PM MDT
Begin rpmdb    : 2867:3751b6f48dab9c90741a2eeaf9ad564f04eb5406
End time       : Mon 26 Mar 2018 12:27:10 PM MDT (15 seconds)
End rpmdb      : 2867:7a22286adbdfa5785d2297a86943bc4630e9368d
User           : <redacted>
Return-Code    : Success
Command Line   : update
Transaction performed with:
    Installed     dnf-2.7.5-2.fc27.noarch  @updates
    Installed     rpm-4.14.1-1.fc27.x86_64 @updates
Packages Altered:
    Upgraded caja-actions-1.8.3-3.fc27.x86_64          @fedora
    Upgrade               1.8.3-4.fc27.x86_64          @updates
    Upgraded caja-actions-doc-1.8.3-3.fc27.noarch      @fedora
    Upgrade                   1.8.3-4.fc27.noarch      @updates
    Upgraded llvm-libs-5.0.1-4.fc27.i686               @updates
    Upgraded llvm-libs-5.0.1-4.fc27.x86_64             @updates
    Upgrade            5.0.1-5.fc27.i686               @updates
    Upgrade            5.0.1-5.fc27.x86_64             @updates
    Upgraded nspr-4.18.0-1.fc27.i686                   @updates
    Upgraded nspr-4.18.0-1.fc27.x86_64                 @updates
    Upgrade       4.19.0-1.fc27.i686                   @updates
    Upgrade       4.19.0-1.fc27.x86_64                 @updates
    Upgraded nspr-devel-4.18.0-1.fc27.x86_64           @updates
    Upgrade             4.19.0-1.fc27.x86_64           @updates
    Upgraded nss-3.35.0-1.1.fc27.i686                  @updates
    Upgraded nss-3.35.0-1.1.fc27.x86_64                @updates
    Upgrade      3.36.0-1.0.fc27.i686                  @updates
    Upgrade      3.36.0-1.0.fc27.x86_64                @updates
    Upgraded nss-softokn-3.35.0-1.0.fc27.i686          @updates
    Upgraded nss-softokn-3.35.0-1.0.fc27.x86_64        @updates
    Upgrade              3.36.0-1.0.fc27.i686          @updates
    Upgrade              3.36.0-1.0.fc27.x86_64        @updates
    Upgraded nss-softokn-freebl-3.35.0-1.0.fc27.i686   @updates
    Upgraded nss-softokn-freebl-3.35.0-1.0.fc27.x86_64 @updates
    Upgrade                     3.36.0-1.0.fc27.i686   @updates
    Upgrade                     3.36.0-1.0.fc27.x86_64 @updates
    Upgraded nss-sysinit-3.35.0-1.1.fc27.x86_64        @updates
    Upgrade              3.36.0-1.0.fc27.x86_64        @updates
    Upgraded nss-tools-3.35.0-1.1.fc27.x86_64          @updates
    Upgrade            3.36.0-1.0.fc27.x86_64          @updates
    Upgraded nss-util-3.35.0-1.0.fc27.i686             @updates
    Upgraded nss-util-3.35.0-1.0.fc27.x86_64           @updates
    Upgrade           3.36.0-1.0.fc27.i686             @updates
    Upgrade           3.36.0-1.0.fc27.x86_64           @updates
    Upgraded osinfo-db-20180311-1.fc27.noarch          @updates
    Upgrade            20180318-1.fc27.noarch          @updates
    Upgraded qt5-qtwebengine-5.10.1-1.fc27.x86_64      @updates
    Upgrade                  5.10.1-4.fc27.x86_64      @updates
    Upgraded sky-2.1.7090-1.fc27.x86_64                @telred-fedora-27
    Upgrade      2.1.7105-1.fc27.x86_64                @telred-fedora-27
Scriptlet output:
   1 Running as unit: run-r24568e81f3404b4fbfa67f339cc1f3b2.service
   2 Running as unit: run-re03851cbc14a4d479dbf5211e3983e0a.service

Comment 1 psg_nm 2018-04-09 23:46:01 UTC
To test my assertion that it didn't seem to matter what I did with Evolution when it crashes, I tried one more test where all I did was start Evolution and authenticate with the Exchange server. The application crashed after about 15 minutes--I didn't open any e-mail, contacts, calendars, etc.  Of course, the application did download new e-mail from the Exchange server (and maybe calendar events??), so it did do something when the application started.

Comment 2 Milan Crha 2018-04-10 07:06:17 UTC
Thanks for a bug report. The update itself didn't bring anything directly related, at least on the first look. The crash happens when evolution-ews downloads one of the messages for offline usage. It can be that the message has some structure which confuses evolution-ews, but your log also shows quite many issues with the folder summary (those GError-s). It's not obvious which of the accounts you've configured (it can be the EWS account, but also the On This Computer) has the problem, but if it's the EWS account, then I'd suggest to try to delete the local cache and let it create it from scratch, which means downloading all the messages again. I would get more information from the bactrace, but there are missing debuginfo packages, or at least for evolution-ews, which would show what line it was and in case of the ABRT report also the local variables, which might help. That the crash happens after some time after start possibly depends on the refresh interval you've set for the account. It's in the account properties, Receiving Options tab.

To make evolution-ews re-download the local summary from scratch, do this:
a) close evolution
b) move away: ~/.cache/evolution/mail/<ews-accouint-uid>
   I want to move it away, thus you can eventually return it back, if anything
   goes really wrong
c) run evolution from a terminal, thus you'll see any runtime issues there.

I do not know how the local summary could get broken, but according to the journalctl log it is broken.

Let me know whether it helped, please.

Comment 3 psg_nm 2018-04-10 15:40:35 UTC
I did as you asked.  Evolution still crashed.  Here are some of the terminal messages (which look a little familiar):


$ evolution

(evolution:4194): camel-CRITICAL **: camel_internet_address_encode_address: assertion 'addr' failed

(evolution:4194): GLib-CRITICAL **: g_string_insert_len: assertion 'len == 0 || val != NULL' failed

(evolution:4194): camel-CRITICAL **: camel_internet_address_encode_address: assertion 'addr' failed

(evolution:4194): GLib-CRITICAL **: g_string_insert_len: assertion 'len == 0 || val != NULL' failed
Segmentation fault (core dumped)



This time Evolution died within just a few minutes of starting as it was synchronizing the folders.

I have created another attachment with the Evolution-specific results from 'journalctl -r' from this experiment.

If I can install some debug libraries, please let me know how I should do it so I can help you determine the source of the problem.

Also, I only have Evolution connected to Exchange right now, so there aren't any other mail accounts used by Evolution.

Comment 4 psg_nm 2018-04-10 15:43:43 UTC
Created attachment 1419960 [details]
Another 'journalctl -r' for experiment on April 10, 2018

Moved the Exchange data from .cache and started evolution again.  It died after only a few minutes as it was syncing messages from Exchange.

Comment 5 Milan Crha 2018-04-11 08:55:51 UTC
Thanks for the update. Those runtime warnings are new to me. I do not know whether they are related to the crash you are facing or not, but it's possible they are.

To install the debug information packages run as root this:

   # dnf install evolution-data-server-debuginfo evolution-debuginfo \
     evolution-ews-debuginfo --enablerepo=updates-debuginfo

and make sure the versions being downloaded match those you've installed:

   $ rpm -q evolution-data-server evolution evolution-ews

In case you've installed any old debuginfo packages use `dnf update` instead of `dnf install` in the above command.

With that installed, I'd like to see what the server returned too, because it can be something what the evolution-ews doesn't expect. You can do that with the below gdb command, which I'd use to avoid the journalctl usage, because it's not necessary for this testing. Run the command as a regular user, not as root:

   $ EWS_DEBUG=2 gdb evolution --ex r --ex "bt full"

This runs evolution and prints a backtrace when gdb stops. It can stop earlier than on the crash, in which case use gdb command "c" to continue and when it stops again use the gdb command "bt full" to print the backtrace. Then you can either "c" to continue or "q" to quit gdb. The backtrace can contain private information, like server addresses, user emails and such, thus make sure you "censor" them before sharing anywhere. Similarly the exact EWS debugging output contains raw communication between the server and evolution-ews, including content of the messages it downloaded. I'm not interested in the whole EWS log, only some part before the crash happened, and that being "censored" too. I'd like to distinguish between filled and unfilled values though, thus ideally "censor" by replacing letters with an 'x' or such (like from "Milan" making "xxx"). If you'd be unsure or there would be too much to censor, feel free to send it packed to my bugzilla email only, with a reference to this bug report in the subject, thus I'd not overlook it in my spam folder.

I believe you could be able to reproduce the crash when you click the Send/Receive button at the top toolbar, which invokes the message download like with the periodical update. The crash should happen as soon as the refresh (and download for offline usage) reaches the offending message.

Comment 6 psg_nm 2018-04-12 19:38:22 UTC
I have been able to install the debuginfo packages and run gdb to find the error.  I have sent the sanitized output directly to your bugzilla email.

Thanks!

Comment 7 Milan Crha 2018-04-13 07:41:43 UTC
Thanks, it looks all fine. It seems as evolution-ews failed to parse the iCalendar attachment/part for some reason, but the code expects it always working, thus it crashed short time afterwards. Would it be possible to send me that offending message
   /home/xxx/.cache/evolution/mail/xxx.xxx.x.xxx/folders/
   Deleted Items/mimecontent/JZGDHZ
as an attachment, please? I promise I'll use it only for debugging purposes and will delete it as soon as I'll have it done. I'd like to see whether the iCalendar part is somehow broken or what failed there. It would be particularly easy to just skip all of this in the code, but I'd like to recover as much from the message as possible, similarly like I did recently here:
   https://bugzilla.gnome.org/show_bug.cgi?id=794815
which was about a similar problem (invalid UTF-8 letters in the response from the server caused incomplete calendar content download).

According to the path of the file, the message is in the Deleted Items folder, thus entering it and right-clicking it and pick Refresh may eventually trigger the crash too. I hope you do not have any automatic expunge policy set n the server, because it might purge the message from the Deleted Items when triggered.

Comment 8 psg_nm 2018-04-16 17:06:14 UTC
I just sent you the uudecoded iCalendar invitation via e-mail.  I had to censor it, but hopefully it is useful.  As I mentioned in the e-mail, I would be happy to run some other tests locally, but I really doubt that it can be cleared for sharing (my apologies).

Comment 9 Milan Crha 2018-04-16 17:36:06 UTC
Created attachment 1422656 [details]
test-rh1565343.c

Thanks for it. I understand the event contains private information, thus it's fine you redacted it. Unfortunately, it seems the decode/censor/something corrected it. I attached a little test program which opens a message you give it a path to (or if you correct the path in the file itself then you can run it without any argument), and it'll try to decode each subpart as iCalendar component and then prints some information about it, including the decoded iCalendar content. The first line of the file contains a comment with a command line to compile it. It requires evolution-data-server-devel and libical-devel packages being installed.
The interesting lines are:
   trying data of len:0 valid utf-8:0 comp:(nil) data:

Comment 10 Milan Crha 2018-04-18 09:14:03 UTC
Thanks for the update (received by email). I've been able to reproduce it too and I realized it had been fixed only recently, here:
https://bugzilla.gnome.org/show_bug.cgi?id=791475#c4
Thus will be part of evolution-data-server 3.28.2+. 

I fixed the crash on the evolution-ews side as well, now that I know it is not an issue on the exchange server, but on the evolution* side instead. Thank you for your help with this.

Created commit f497204 in ews master (3.29.2+) [1]
Created commit 183f9ba in ews gnome-3-28 (3.28.2+)

[1] https://git.gnome.org/browse/evolution-ews/commit/?id=f497204

Comment 11 Milan Crha 2018-04-25 08:11:59 UTC
*** Bug 1571400 has been marked as a duplicate of this bug. ***

Comment 12 Milan Crha 2018-05-15 10:11:23 UTC
*** Bug 1577994 has been marked as a duplicate of this bug. ***

Comment 13 Milan Crha 2018-07-23 14:55:22 UTC
*** Bug 1605417 has been marked as a duplicate of this bug. ***


Note You need to log in before you can comment on or make changes to this bug.