May I please get the output of "rpm -q zarafa"? And may you please create a 
coredump with a full backtrace? Does this issue happen with a specific e-mail
or more or less at a random point?

[root@null zarafa]# rpm -q zarafa; cat /etc/redhat-release 
zarafa-6.30.10-2.fc12.x86_64
Fedora release 12 (Constantine)

It seems to happen with a specific mail - I can reproduce the problem if I re-run imapsync.

I'll try to produce a better backtrace!

strace:

[pid 25715] sendto(13, "POST /zarafa HTTP/1.1\r\nHost: loc"..., 834, 0, NULL, 0) = 834
[pid 25715] recvfrom(13, "HTTP/1.1 200 OK\r\nServer: gSOAP/2"..., 65536, 0, NULL, NULL) = 707
[pid 25715] open("/usr/lib64/gconv/ISO8859-1.so", O_RDONLY) = 14
[pid 25715] read(14, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\200\4\0\0\0\0\0\0"..., 832) = 832
[pid 25715] fstat(14, {st_mode=S_IFREG|0755, st_size=13008, ...}) = 0
[pid 25715] mmap(NULL, 2105376, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 14, 0) = 0x7f516dd57000
[pid 25715] mprotect(0x7f516dd58000, 2097152, PROT_NONE) = 0
[pid 25715] mmap(0x7f516df58000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 14, 0x1000) = 0x7f516df58000
[pid 25715] close(14)                   = 0
[pid 25715] mprotect(0x7f516df58000, 4096, PROT_READ) = 0
[pid 25715] --- SIGSEGV (Segmentation fault) @ 0 (0) ---
[pid 25715] write(3, "Mon 22 Mar 2010 11:46:02 AM CET:"..., 65) = 65
[pid 25715] futex(0x7f5175de1600, FUTEX_WAKE_PRIVATE, 2147483647) = 0
[pid 25715] write(3, "Mon 22 Mar 2010 11:46:02 AM CET:"..., 103) = 103
[pid 25715] write(3, "Mon 22 Mar 2010 11:46:02 AM CET:"..., 99) = 99
[pid 25715] write(3, "Mon 22 Mar 2010 11:46:02 AM CET:"..., 94) = 94
[pid 25715] write(3, "Mon 22 Mar 2010 11:46:02 AM CET:"..., 220) = 220
[pid 25715] write(3, "Mon 22 Mar 2010 11:46:02 AM CET:"..., 170) = 170
[pid 25715] write(3, "Mon 22 Mar 2010 11:46:02 AM CET:"..., 173) = 173
[pid 25715] write(3, "Mon 22 Mar 2010 11:46:02 AM CET:"..., 155) = 155
[pid 25715] write(3, "Mon 22 Mar 2010 11:46:02 AM CET:"..., 171) = 171
[pid 25715] write(3, "Mon 22 Mar 2010 11:46:02 AM CET:"..., 171) = 171
[pid 25715] write(3, "Mon 22 Mar 2010 11:46:02 AM CET:"..., 161) = 161
[pid 25715] write(3, "Mon 22 Mar 2010 11:46:02 AM CET:"..., 153) = 153
[pid 25715] write(3, "Mon 22 Mar 2010 11:46:02 AM CET:"..., 179) = 179
[pid 25715] write(3, "Mon 22 Mar 2010 11:46:02 AM CET:"..., 125) = 125
[pid 25715] write(3, "Mon 22 Mar 2010 11:46:02 AM CET:"..., 111) = 111
[pid 25715] write(3, "Mon 22 Mar 2010 11:46:02 AM CET:"..., 99) = 99
[pid 25715] write(3, "Mon 22 Mar 2010 11:46:02 AM CET:"..., 96) = 96
[pid 25715] write(3, "Mon 22 Mar 2010 11:46:02 AM CET:"..., 144) = 144
[pid 25715] kill(0, SIGSEGV)            = 0
[pid 25700] <... select resumed> )      = ? ERESTARTNOHAND (To be restarted)
[pid 25715] rt_sigreturn(0 <unfinished ...>
[pid 25700] --- SIGSEGV (Segmentation fault) @ 0 (0) ---
Process 25700 detached
[pid 25715] <... rt_sigreturn resumed> ) = 0
[pid 25715] +++ killed by SIGSEGV +++
[pid 25724] +++ killed by SIGSEGV +++


For some reason it doesn't produce a core dump...
I'll try to run it in GDB next.

Run from GDB (anonymised):
Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7ffff7fe7710 (LWP 29824)]
__strlen_sse2 () at ../sysdeps/x86_64/strlen.S:31
31              pcmpeqb (%rdi), %xmm2
(gdb) bt
#0  __strlen_sse2 () at ../sysdeps/x86_64/strlen.S:31
#1  0x00007ffff7944136 in length (this=<value optimized out>, lpicEvent=<value optimized out>, lpIcalItem=<value optimized out>, lplstMsgProps=<value optimized out>, lplstIcalRecip=<value optimized out>)
    at /usr/include/c++/4.4.3/bits/char_traits.h:263
#2  assign (this=<value optimized out>, lpicEvent=<value optimized out>, lpIcalItem=<value optimized out>, lplstMsgProps=<value optimized out>, lplstIcalRecip=<value optimized out>)
    at /usr/include/c++/4.4.3/bits/basic_string.h:970
#3  operator= (this=<value optimized out>, lpicEvent=<value optimized out>, lpIcalItem=<value optimized out>, lplstMsgProps=<value optimized out>, lplstIcalRecip=<value optimized out>)
    at /usr/include/c++/4.4.3/bits/basic_string.h:514
#4  VConverter::HrAddRecipients (this=<value optimized out>, lpicEvent=<value optimized out>, lpIcalItem=<value optimized out>, lplstMsgProps=<value optimized out>, lplstIcalRecip=<value optimized out>)
    at vconverter.cpp:941
#5  0x00007ffff7946f71 in VConverter::HrICal2MAPI (this=0x7fffe8172880, lpEventRoot=0x7fffe820f960, lpEvent=0x7fffe8210f10, lpPrevItem=<value optimized out>, lppRet=0x7ffff7fe5558) at vconverter.cpp:167
#6  0x00007ffff7947c89 in VEventConverter::HrICal2MAPI (this=<value optimized out>, lpEventRoot=<value optimized out>, lpEvent=<value optimized out>, lpPrevItem=<value optimized out>, lppRet=0x7ffff7fe5558)
    at vevent.cpp:71
#7  0x00007ffff7939b2b in ICalToMapiImpl::ParseICal (this=0x7fffe817c6c0, strIcal=<value optimized out>, strCharset=<value optimized out>, strServerTZparam=<value optimized out>, lpMailUser=<value optimized out>,
    ulFlags=<value optimized out>) at ICalToMAPI.cpp:232
#8  0x00007ffff7b9d063 in VMIMEToMAPI::disectBody (this=0x7fffe8167a40, VMHeader=0x7fffe81e2078, VMBody=0x7fffe81e20a8, lpMessage=0x7fffe81f3210, onlyBody=<value optimized out>, filterDouble=<value optimized out>)
    at VMIMEToMAPI.cpp:1542
#9  0x00007ffff7b9c81b in VMIMEToMAPI::disectBody (this=0x7fffe8167a40, VMHeader=<value optimized out>, VMBody=0x7fffe81fafd8, lpMessage=0x7fffe81f3210, onlyBody=<value optimized out>,
    filterDouble=<value optimized out>) at VMIMEToMAPI.cpp:1384
#10 0x00007ffff7b9bb8f in VMIMEToMAPI::fillMAPIMail (this=0x7fffe8167a40, VMMessage=<value optimized out>, lpMessage=0x7fffe81f3210) at VMIMEToMAPI.cpp:370
#11 0x00007ffff7b9c0e6 in VMIMEToMAPI::convertVMIMEToMAPI (this=0x7fffe8167a40, input=
    "Return-Path: <XXX.XXX>\r\nReceived: from mail.XX.XX.XX ([unix socket])\r\n\tby mail.XX.XX.XX (Cyrus v2.1.16) with LMTP; Fri, 05 Mar 2010 13:45:14 +0100\r\nX-Sieve: CMU Sieve 2.2\r\n---Type <return> to continue, or q <return> to quit---
"..., lpMessage=0x7fffe81f3210) at VMIMEToMAPI.cpp:164
#12 0x00007ffff7ba7dcb in IMToMAPI (lpSession=<value optimized out>, lpMsgStore=<value optimized out>, lpAddrBook=<value optimized out>, lpMessage=0x7fffe81f3210, input=
    "Return-Path: <XXX.XXX>\r\nReceived: from mail.XX.XX.XX ([unix socket])\r\n\tby mail.XX.XX.XX (Cyrus v2.1.16) with LMTP; Fri, 05 Mar 2010 13:45:14 +0100\r\nX-Sieve: CMU Sieve 2.2\r\n"..., dopt=..., lpLogger=<value optimized out>) at inetmapi.cpp:180
#13 0x0000000000429080 in IMAP::HrCmdAppend (this=0x7ffff7fe6c60, strTag=<value optimized out>, strFolder="INBOX", strData=<value optimized out>, strFlags="(\\Seen NonJunk)", strTime=<value optimized out>)
    at IMAP.cpp:1311
#14 0x000000000040fdf5 in HandlerIMAP (lpArg=<value optimized out>) at Gateway.cpp:647
#15 0x00007ffff5d40a3a in start_thread (arg=<value optimized out>) at pthread_create.c:297
#16 0x00007ffff5a9f67d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#17 0x0000000000000000 in ?? ()

FYI. I just checked with my colleague; The mail is an iCal invitation with an iCal attachment. Hope this information helps.

Are you able to provide the e-mail causing the crash as mbox or something like
this? Of course the e-mail still should crash the gateway after anonymisation.

Created attachment 401793 [details]
Mail message causing the zarafa-gateway crash

I got the mail forwarded, but I guess the original mail should haven't changed...

Any progress?

Duplicate of upstream Zarafa ticket #5477:

libical should compiled with ./configure --disable-icalerrors-are-fatal; user 
also needs patch libical-configure-undef-fatal-errors.diff in  libical (in our 
source package).

http://developer.zarafa.com/download/libical-configure-undef-fatal-errors.diff

Debarshi, as the libical maintainer, may you us help here? Or maybe Huzaifa?

I patched libical locally with the proposed patch and added the configure option. Restarted the service and started synchronizing various accounts. So far no segfaults!

I could do it in CVS as well, if you don't have time and want me to do it for you.

Thanks for the feedback, but I'm not sure whether I should touch libical with
a non-upstream patch without having talked to the package maintainer before -
even I'm a provenpackager who just could do this.

Debarshi, Huzaifa? May you give me some feedback regarding comment #9 and #10?

From Libical 0.43 errors are set to be non-fatal by default. See the following snippet from src/libical/icalerror.c:117:

#if ICAL_ERRORS_ARE_FATAL == 1
int icalerror_errors_are_fatal = 1;
#else
int icalerror_errors_are_fatal = 0;
#endif

The icalerror_errors_are_fatal global variable, which can also be set by the applications, determine whether calls to icalerror_set_errno are fatal or not.

Therefore Zarafa is probably triggering something else. And looking at the libical code there are some places which use 'ifndef ICAL_ERRORS_ARE_FATAL' and not '#if ICAL_ERRORS_ARE_FATAL == 1'.

Debarshi,
 the only thing I can say is that zarafa-gateway doesn't die any more. Since over a week it's running fine now.

Can you try out the following libical change?

Patch: http://rishi.fedorapeople.org/libical-0.43-errors-are-fatal.patch
SRPM: http://rishi.fedorapeople.org/libical-0.43-6.fc12.src.rpm

It would be nice to run this past Zarafa's upstream. If this has the same effect as the one they have published at http://developer.zarafa.com/download/libical-configure-undef-fatal-errors.diff then we can notify libical's upstream.

Debarshi,
 I'll try your libical in a hurry.

However, I would like to add two more SIGSEGVs:

server.log - quite clear why it failed, but it should not *DIE*(!):
Fri 09 Apr 2010 09:37:12 AM CEST: Unable to get database connection: Too many connections
Fri 09 Apr 2010 09:37:12 AM CEST: 0x0000000047367b /usr/bin/zarafa-server(_Z7sigsegvi+0x4b) [0x47367b]
Fri 09 Apr 2010 09:37:12 AM CEST: 0x007f69c9db50f0 /lib64/libpthread.so.0(+0xf0f0) [0x7f69c9db50f0]
Fri 09 Apr 2010 09:37:12 AM CEST: 0x00000000498a9f /usr/bin/zarafa-server(_ZN18ECStoreObjectTable4LoadEv+0x1ff) [0x498a9f]
Fri 09 Apr 2010 09:37:12 AM CEST: 0x0000000055124d /usr/bin/zarafa-server(_ZN20ECGenericObjectTable9QueryRowsEP4soapjjPP6rowSet+0x4d) [0x55124d]
Fri 09 Apr 2010 09:37:12 AM CEST: 0x000000004f011c /usr/bin/zarafa-server(_Z18ns__tableQueryRowsP4soapyjjjP22tableQueryRowsResponse+0x42c) [0x4f011c]
Fri 09 Apr 2010 09:37:12 AM CEST: 0x00000000607d05 /usr/bin/zarafa-server(_Z29soap_serve_ns__tableQueryRowsP4soap+0xb5) [0x607d05]
Fri 09 Apr 2010 09:37:12 AM CEST: 0x0000000060e354 /usr/bin/zarafa-server(_Z10soap_serveP4soap+0xa4) [0x60e354]
Fri 09 Apr 2010 09:37:12 AM CEST: 0x00000000479828 /usr/bin/zarafa-server(_ZN22ECSoapServerConnection15process_requestEPv+0x78) [0x479828]
Fri 09 Apr 2010 09:37:12 AM CEST: 0x007f69c9daca3a /lib64/libpthread.so.0(+0x6a3a) [0x7f69c9daca3a]
Fri 09 Apr 2010 09:37:12 AM CEST: 0x007f69c9b0b67d /lib64/libc.so.6(clone+0x6d) [0x7f69c9b0b67d]
Fri 09 Apr 2010 09:37:12 AM CEST: When reporting this traceback, please include Linux distribution name, system architecture and Zarafa version.


monitor.log - It's obvious that this happened because the server process was gone. But again, there's no reason to *DIE*(!):
Fri 09 Apr 2010 09:37:13 AM CEST: Unable to receive stats table data, error 0x80040115
Fri 09 Apr 2010 09:37:13 AM CEST: Caught SIGSEGV (11), traceback:
Fri 09 Apr 2010 09:37:13 AM CEST: 0x00000000406357 /usr/bin/zarafa-monitor(_Z7sigsegvi+0x57) [0x406357]
Fri 09 Apr 2010 09:37:13 AM CEST: 0x007f9d624cc0f0 /lib64/libpthread.so.0(+0xf0f0) [0x7f9d624cc0f0]
Fri 09 Apr 2010 09:37:13 AM CEST: 0x0000000040a3ef /usr/bin/zarafa-monitor(_ZN14ECQuotaMonitor17CheckCompanyQuotaEP11_sECCompany+0x71f) [0x40a3ef]
Fri 09 Apr 2010 09:37:13 AM CEST: 0x0000000040a7ab /usr/bin/zarafa-monitor(_ZN14ECQuotaMonitor10CheckQuotaEv+0x2db) [0x40a7ab]
Fri 09 Apr 2010 09:37:13 AM CEST: 0x0000000040ab9d /usr/bin/zarafa-monitor(_ZN14ECQuotaMonitor6CreateEPv+0x16d) [0x40ab9d]
Fri 09 Apr 2010 09:37:13 AM CEST: 0x007f9d624c3a3a /lib64/libpthread.so.0(+0x6a3a) [0x7f9d624c3a3a]
Fri 09 Apr 2010 09:37:13 AM CEST: 0x007f9d6222267d /lib64/libc.so.6(clone+0x6d) [0x7f9d6222267d]
Fri 09 Apr 2010 09:37:13 AM CEST: When reporting this traceback, please include Linux distribution name, system architecture and Zarafa version.

Oliver, any news regarding the libical changes from Debarshi?

Sorry. I totally forgot to update this bug.

I've recompiled libical-0.43-6.fc12.src.rpm on my system and ran a few imapsync processes already and didn't see any problems so far.

So the ical stuff seems to be solved with >WorksForMe< :-)

Debarshi, how do we proceed now? From my point of understanding we need to
get your patch into upstream and into Fedora (including EPEL). Right?

(In reply to comment #20)
> Debarshi, how do we proceed now? From my point of understanding we need to
> get your patch into upstream and into Fedora (including EPEL). Right?    

Can we run this past the Zarafa developers? Then I can handle the Libical end of things.

Debarshi, the Quality Assurance and Release Manager of Zarafa told me, that
the developers don't want to replace the patch (as they can't be sure, that
it will have the same effect). But it would make a difference, if the patch
makes it into upstream.

Debarshi, any news here?

Ping? Debarshi, any news here? May we please send your patch to libical
upstream and apply it to all active Fedora branches and especially to the
RHEL 6 beta?

This message is a reminder that Fedora 12 is nearing its end of life.
Approximately 30 (thirty) days from now Fedora will stop maintaining
and issuing updates for Fedora 12.  It is Fedora's policy to close all
bug reports from releases that are no longer maintained.  At that time
this bug will be closed as WONTFIX if it remains open with a Fedora 
'version' of '12'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version prior to Fedora 12's end of life.

Bug Reporter: Thank you for reporting this issue and we are sorry that 
we may not be able to fix it before Fedora 12 is end of life.  If you 
would still like to see this bug fixed and are able to reproduce it 
against a later version of Fedora please change the 'version' of this 
bug to the applicable version.  If you are unable to change the version, 
please add a comment here and someone will do it for you.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events.  Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.

The process we are following is described here: 
http://fedoraproject.org/wiki/BugZappers/HouseKeeping

I'm very sure, this report is still valid, moving to Rawhide.

libical-0.46-2.el4 has been submitted as an update for Fedora EPEL 4.
https://admin.fedoraproject.org/updates/libical-0.46-2.el4

libical-0.46-2.fc14 has been submitted as an update for Fedora 14.
https://admin.fedoraproject.org/updates/libical-0.46-2.fc14

libical-0.46-2.el5 has been submitted as an update for Fedora EPEL 5.
https://admin.fedoraproject.org/updates/libical-0.46-2.el5

libical-0.46-2.fc13 has been submitted as an update for Fedora 13.
https://admin.fedoraproject.org/updates/libical-0.46-2.fc13

libical-0.46-2.el4 has been pushed to the Fedora EPEL 4 testing repository.  If problems still persist, please make note of it in this bug report.
 If you want to test the update, you can install it with 
 su -c 'yum --enablerepo=updates-testing update libical'.  You can provide feedback for this update here: https://admin.fedoraproject.org/updates/libical-0.46-2.el4

libical-0.46-2.el5 has been pushed to the Fedora EPEL 5 stable repository.  If problems still persist, please make note of it in this bug report.

libical-0.46-2.el4 has been pushed to the Fedora EPEL 4 stable repository.  If problems still persist, please make note of it in this bug report.

libical-0.46-2.fc14 has been pushed to the Fedora 14 stable repository.  If problems still persist, please make note of it in this bug report.