Sorry for late reply.
It would be great if you could try with cyrus-sasl-2.1.23-14 to bisect where to find a problem. You can achieve it be removing
 * cyrus-sasl-2.1.23-ad_compat.patch
from your srpm

The only GSSAPI related changes between 13 and 15 are bz994242 and  bz825863. I don't see the problem there, but it wouldn't be the first time upstream would introduce regression, but we still don't have any other reports than from you.

Backtrace would be helpful. I see the error is coming from 

 * _plug_decode() at plugins/plugin_common.c:659

which is probably called from

 * gssapi_decode() at plugins/gssapi.c:497
 * ???

If it is still actual, can you try to build the recent RPM without these patches:

 * cyrus-sasl-2.1.23-release-server_creds.patch
or 
 * cyrus-sasl-2.1.23-ad_compat.patch

it is enough to comment out the lines

 * %patch40 -p1 -b .release-server_creds
or
 * %patch41 -p1 -b .ad_compat

and build the packages over.


Unfortunately, your description of your environment is quite vague to reproduce in my environment.

Jakub, I will try to test this and provide more information/config files within the next few weeks when I get some time. 

--Matt

Jakub, I have rebuilt using the cyrus-sasl-2.1.23-15.el6.src.rpm package.

If I install the RPM from that build it breaks again of course with the message,
lmtp[26204]: encoded packet size too big (4156 > 4096)

If I comment out only %patch41, and leave %patch42 uncommented, it works!
So it is definitely something in patch41 -p1 -b .ad_compat

I will attach my imapd.conf and cyrus.conf and any other relevant files I can think of. Let me know how I could generate a backtrace of those plugins.

Created attachment 1104828 [details]
backend imapd.conf

Created attachment 1104829 [details]
cyrus.conf

Created attachment 1104830 [details]
smtpd.conf sasl2

Created attachment 1104831 [details]
pam.d/lmtp,imap,smtp

Comment on attachment 1104831 [details]
pam.d/lmtp,imap,smtp

And saslauthd is configured to use pam

Created attachment 1104837 [details]
frontend imapd.conf

Thanks for coming back with your findings. The %patch41 is way too complicated and I am worried there might be some parts missing. Thought, for rhel6 it is going to be hard to come with new changes. Would it be possible for you to give it a try to your use case with RHEL7? There is newer version of cyrus-sasl, but also it might be also cyrus-imap bug.

Jakub,
So I am kind of surprised this is still a problem, but I have actually just reproduced it in CentOS7 with cyrus-sasl-2.1.26-19.2

rpm -qa | grep cyrus
cyrus-imapd-2.4.17-8.el7_1.x86_64
cyrus-sasl-lib-2.1.26-19.2.el7.x86_64
cyrus-imapd-utils-2.4.17-8.el7_1.x86_64
cyrus-sasl-2.1.26-19.2.el7.x86_64
cyrus-sasl-plain-2.1.26-19.2.el7.x86_64
cyrus-sasl-gssapi-2.1.26-19.2.el7.x86_64

First I built a standalone server with all the same auth mechanisms as before, saslauthd/pam_krb5 and did not get the error.

I then built a second server, set it as the backend, and the initial server as a frontend/mupdate server. Then I configured all the correct principals in kerberos, etc. 
Then started with a new mboxlist, created a mailbox with cyradm and confirmed that it got proxied to the backend. I now get the error again.

cyrus-centos7-backend lmtp[15655]: encoded packet size too big (4156 > 4096)

and on the frontend:

postfix/lmtp[3052]: 31CDF633FA: to=<matt.hoyle@cyrus-centos7>, relay=cyrus-centos7[/var/lib/imap/socket/lmtp], delay=0.19, delays=0.07/0.01/0.03/0.09, dsn=4.3.0, status=deferred (host cyrus-centos7[/var/lib/imap/socket/lmtp] said: 451 4.3.0 System I/O error (in reply to end of DATA command))


I am using the same configs as what I attached earlier, just different hostnames.

If I keep everything the same, but delete the matt.hoyle mailbox in cyradm, then comment out mupdate_server and serverlist in imapd.conf (therefore making it a standalone server again), and create the mailbox in cyradm again, the error goes away.

before: user.matt^hoyle 1 cyrus-centos7-backend!default matt.hoyle        lrswipkxtecda  <--- proxied mailbox and cannot send mail that is over 4k

after: user.matt^hoyle 0 default matt.hoyle    lrswipkxtecda   <--- standalone mailbox and same email sends fine

Yes, you are right. This feature (ad_compat) is available also in RHEL7 since it came from upstream around 2010 (commit [1] and related). Though it is certainly wrong that it breaks interoperability with existing setups.

Reading further, seems related to bugs [2] and [3] even though they are marked as fixed (but the resolution does not look much confident).

I tried to set up the virtual machines to reproduce your configuration, but without success. Still "I configured all the correct principals in kerberos" and cyradm stuff are things that I am not well versed. Is it local principal or remote? Are there used krb5 configuration files? Postfix is configured how? cyradm is where?

Yes. I never configured cyrus-imapd so I probably miss some details. You might try to drop a message to cyrus-imap or cyrus-sasl mailing list and there might be somebody who will be able to help you (and might see some obvious flaw that I miss). Or if you might try to prepare virtual machines/docker files/something ready to use/step-to-step how-to and I might go into that with some mass destruction tool like strace or gdb from cyrus-sasl point of view.


[1] https://cgit.cyrus.foundation/cyrus-sasl/commit/plugins/gssapi.c?id=cccc5a5a87a74cd434fbdf5e87c4158e21ebcf19
[2] https://bugzilla.cyrusimap.org/show_bug.cgi?id=2314
[3] https://bugzilla.cyrusimap.org/show_bug.cgi?id=2457

Thanks for the information. I wonder if you should change this bug to RHEL7 since that seems more relevant now? I looked through those bugs, I tried reverting just the changes in [1] but it did not seem to fix it.

I downloaded the SRPMs for cyrus-sasl 2.1.23-15 and 2.1.26-19.2. I then compared patch41 from 2.1.23-15 to gssapi.c in 2.1.26-19 and modified the c file to try and revert everything that patch did (since that patch is no longer a patch in 2.1.26). I found a set of changes that makes the problem go away, I created a patch with the changes, built the RPMs and just installed cyrus-sasl-gssapi and that fixes it. 
I am going to attach the patch I created for you to look through, I am also going to try and narrow down those changes.

Would you be able to give me some hints on using gdb with this? I have attached it to the cyrus master pid, but not really sure where I should set breakpoints or anything. I also setup debugging for lmtp using gdb, strace, and ltrace commands. I did not see anything useful in strace or ltrace, and gdb did not seem to generate the backtrace.

I will attempt to write up some instructions for you to reproduce this along with step-by-step for kerberos. Just give me a few days to do that.

Created attachment 1123162 [details]
My patch to revert ad_compat changes

So I figured out gdb a little bit. I set a breakpoint in plugin_common.c at line 658 when it compares text->size > text->in_maxbuf
I'm not sure if any of this will help any, but figured I'd pass it along. Let me know if there is any other info that could be useful.

Breakpoint 4, _plug_decode (text=text@entry=0x1063948, input=0x1061834 "\005\004\006\377", inputlen=4092, 
    output=output@entry=0x1063978, outputsize=outputsize@entry=0x106398c, outputlen=0x7ffc92fe6e64, 
    decode_pkt=decode_pkt@entry=0x7ff0dfbd51b0 <gssapi_decode_packet>, rock=rock@entry=0x10638f0)
    at ../plugins/plugin_common.c:658
658                     if (text->size > text->in_maxbuf) {
(gdb) p text->size
$22 = 4156
(gdb) p text->in_maxbuf
$23 = 4096

(gdb) bt
#0  _plug_decode (text=text@entry=0x1063948, input=0x1061834 "\005\004\006\377", inputlen=4092, output=output@entry=0x1063978, outputsize=outputsize@entry=0x106398c, outputlen=0x7ffc92fe6e64, 
    decode_pkt=decode_pkt@entry=0x7ff0dfbd51b0 <gssapi_decode_packet>, rock=rock@entry=0x10638f0) at ../plugins/plugin_common.c:658
#1  0x00007ff0dfbd5726 in gssapi_decode (context=0x10638f0, input=<optimized out>, inputlen=<optimized out>, output=0x7ffc92fe6e68, outputlen=<optimized out>) at gssapi.c:511
#2  0x00007ff0e689359d in sasl_decode (conn=0x105cb70, input=<optimized out>, inputlen=inputlen@entry=4096, output=output@entry=0x7ffc92fe6e68, outputlen=outputlen@entry=0x7ffc92fe6e64) at common.c:643
#3  0x0000000000446b62 in prot_sasldecode (s=s@entry=0x102d310, n=n@entry=4096) at prot.c:204
#4  0x0000000000447aec in prot_fill (s=s@entry=0x102d310) at prot.c:727
#5  0x00000000004488e9 in prot_getc (s=s@entry=0x102d310) at prot.c:1704
#6  0x000000000040c728 in parseheader (skipheaders=0x7ffc92fe7180, contents=<synthetic pointer>, headname=<synthetic pointer>, fout=0x106f360, fin=0x102d310) at spool.c:157
#7  spool_fill_hdrcache (fin=0x102d310, fout=fout@entry=0x106f360, cache=0x1035690, skipheaders=skipheaders@entry=0x7ffc92fe7180) at spool.c:361
#8  0x000000000040ad04 in savemsg (m=<optimized out>, func=0x75cb00 <mylmtp>, cd=0x7ffc92fe7930) at lmtpengine.c:725
#9  lmtpmode (func=func@entry=0x75cb00 <mylmtp>, pin=0x102d310, pout=<optimized out>, fd=fd@entry=0) at lmtpengine.c:1302
#10 0x0000000000406ead in service_main (argc=argc@entry=1, argv=argv@entry=0x1024010, envp=envp@entry=0x7ffc92fea8e0) at lmtpd.c:305
#11 0x0000000000406540 in main (argc=<optimized out>, argv=<optimized out>, envp=0x7ffc92fea8e0) at service.c:582


I also set a breakpoint for gssapi_decode. When it gets to the point before failing, I checked the size of inputlen and it is 4096, does it add bytes to that after calling _plug_decode?

(gdb) p inputlen
$8 = 4096
(gdb) bt
#0  gssapi_decode (context=0x1067a60, input=0x1061830 "", inputlen=4096, output=0x7ffc92fe6e68, outputlen=0x7ffc92fe6e64) at gssapi.c:507
#1  0x00007ff0e689359d in sasl_decode (conn=0x105cb70, input=<optimized out>, inputlen=inputlen@entry=4096, output=output@entry=0x7ffc92fe6e68, outputlen=outputlen@entry=0x7ffc92fe6e64) at common.c:643
#2  0x0000000000446b62 in prot_sasldecode (s=s@entry=0x102d310, n=n@entry=4096) at prot.c:204
#3  0x0000000000447aec in prot_fill (s=s@entry=0x102d310) at prot.c:727
#4  0x00000000004488e9 in prot_getc (s=s@entry=0x102d310) at prot.c:1704
#5  0x000000000040c728 in parseheader (skipheaders=0x7ffc92fe7180, contents=<synthetic pointer>, headname=<synthetic pointer>, fout=0x10661d0, fin=0x102d310) at spool.c:157
#6  spool_fill_hdrcache (fin=0x102d310, fout=fout@entry=0x10661d0, cache=0x1035690, skipheaders=skipheaders@entry=0x7ffc92fe7180) at spool.c:361
#7  0x000000000040ad04 in savemsg (m=<optimized out>, func=0x75cb00 <mylmtp>, cd=0x7ffc92fe7930) at lmtpengine.c:725
#8  lmtpmode (func=func@entry=0x75cb00 <mylmtp>, pin=0x102d310, pout=<optimized out>, fd=fd@entry=0) at lmtpengine.c:1302
#9  0x0000000000406ead in service_main (argc=argc@entry=1, argv=argv@entry=0x1024010, envp=envp@entry=0x7ffc92fea8e0) at lmtpd.c:305
#10 0x0000000000406540 in main (argc=<optimized out>, argv=<optimized out>, envp=0x7ffc92fea8e0) at service.c:582

(gdb) info args
context = 0xedb640
input = 0xedd770 ""
inputlen = 4096
output = 0x7ffca694a388
outputlen = 0x7ffca694a384

First of all I would run the updated version (with ad_compat) only on one of the machines, to make eliminate at least half of the patch and debugging.

The first part is quite obvious, because it is visible in the original error message. In plug_decode there is some recalculation of the sizes to something else (you see in backtrace, that inputlen=4092 even in the input of the function). But I suspect more the sender part, that it sends too big packet.

It would be interesting to see the same backtrace of working and failing cases.

Jakub, I have finally narrowed down the issue to one line. I'm not sure if the issue could pop up again in certain conditions. But at least for me I can deliver the email with this one change to gssapi.c:

diff -Npru cyrus-sasl-2.1.26.orig/plugins/gssapi.c cyrus-sasl-2.1.26/plugins/gssapi.c
--- cyrus-sasl-2.1.26.orig/plugins/gssapi.c     2016-02-12 16:41:49.715000000 -0500
+++ cyrus-sasl-2.1.26/plugins/gssapi.c  2016-02-12 16:42:37.938000000 -0500
@@ -1016,8 +1016,7 @@ gssapi_server_mech_ssfcap(context_t *tex
     }

     /* build up our security properties token */
-    if (text->requiressf != 0 &&
-       (text->qop & (LAYER_INTEGRITY|LAYER_CONFIDENTIALITY))) {
+    if (text->qop & (LAYER_INTEGRITY|LAYER_CONFIDENTIALITY)) {
        if (params->props.maxbufsize > 0xFFFFFF) {
            /* make sure maxbufsize isn't too large */
            /* maxbufsize = 0xFFFFFF */

So maybe something wrong about the way requiressf is getting set? Everything else in the code is from the latest version cyrus-sasl, this was the only change I made.

Here are some steps to reproduce in the same environment as me. I created 3 VMs of CentOS7-minimal. One for kerberos, one for cyrus frontend/mupdate and postfix, one for backend.

You can follow this guide to first configure kerberos on server and clients (imap servers).
https://www.centos.org/docs/5/html/5.1/Deployment_Guide/s1-kerberos-server.html

To reproduce you don't need to worry about user authentication and can skip TLS, you will simply be able to send from the frontend using a program like mailx, and just have the proper kerberos tickets for the frontend and backend.
To configure pam on the clients, you can either specify pam_krb5.so in pam.d/mupdate pam.d/smtp, etc. as I did in the attached pam files. Or you can use authconfig-tui and select "Use Kerberos" which will configure pam.d/password-auth, then all your pam.d files for mupdate, lmtp, imap, and smtp can just use password-auth like so:
#%PAM-1.0
auth       include      password-auth
account    include      password-auth

Once your kerberos server is running, you can use the command kadmin.local on the kerberos server to create the principals. You should also create an admin user to connect to the kerberos server, the principal would be "<user>/admin"
You will need host tickets as well as user tickets, I usually match the user tickets to the hostname minus the domain name, so if your fqdns are frontend.local and backend.local, your principals should look like this:

backend@LOCAL
frontend@LOCAL
imap/backend.local@LOCAL
imap/frontend.local@LOCAL
lmtp/backend.local@LOCAL
lmtp/frontend.local@LOCAL
mupdate/frontend.local@LOCAL
<user>/admin@LOCAL

For example host tickets get created with, "addprinc imap/backend.local"
User ticket gets created with, "addprinc backend"

Now you need to add the clients tickets to frontend and backend by using ktadd command with the admin user you created.
kadmin -p <user>/admin -q "ktadd imap/backend.local"
Add the remaining tickets, lmtp/backend.local and backend.
Then do the same on frontend server add frontend's tickets.

Once that is done there will be a file /etc/krb5.keytab, change the ownership to this:
$ ll /etc/krb5.keytab 
-rw-r----- 1 cyrus saslauth 1498 Feb 10 21:01 /etc/krb5.keytab

Your cyrus.conf will get this ticket with an auth command under START,
auth    cmd="/usr/bin/kinit -k backend"
on frontend:    auth    cmd="/usr/bin/kinit -k frontend"

Look at the attached imapd config, replacing username with just "frontend", you can also remove TLS config.

To make things easy, I would just install cyrus-imapd and sasl from the repos to first reproduce the error. You can then build just the sasl rpms from srpm and use rpm --force -iv on the new rpms created. I also had to install the new debuginfo rpm created with the patched cyrus-sasl-gssapi rpm to get the debug symbols.

These are the sasl rpms needed:
rpm -qa | grep sasl
cyrus-sasl-plain-2.1.26-19.2.el7.centos.x86_64
cyrus-sasl-lib-2.1.26-19.2.el7.centos.x86_64
cyrus-sasl-devel-2.1.26-19.2.el7.centos.x86_64
cyrus-sasl-gssapi-2.1.26-19.2.el7.centos.x86_64
cyrus-sasl-2.1.26-19.2.el7.centos.x86_64
cyrus-sasl-debuginfo-2.1.26-19.2.el7.centos.x86_64

If pam is configured correctly, edit /etc/sysconfig/saslauthd and set MECH=pam, then start it with systemctl. Do this on both frontend and backend.
If gssapi is installed from srpm, you can use "/usr/sbin/saslauthd -a pam" to start it. Then start cyrus master on frontend first so mupdate is running, then start it on the backend.
Services running on frontend:
imap    cmd="proxyd" listen="imap" prefork=5
lmtpunix  cmd="lmtpproxyd" listen="/var/lib/imap/socket/lmtp" prefork=1
mupdate   cmd="/usr/local/cyrus/bin/mupdate -m" listen=3905 prefork=1

on backend:
imap    cmd="imapd" listen="imap" prefork=5
lmtp   cmd="lmtpd" listen="lmtp" prefork=0
and under START on backend:
mupdatepush   cmd="ctl_mboxlist -m"

Now configure master.cf and main.cf for postfix on the frontend.
For this test I believe you only need to worry about lmtp, feel free to enable cyrus delivery to though to compare, I found the bug with either method.
lmtp      unix  -       -       n       -       -       lmtp
cyrus     unix  -       n       n       -       -       pipe
  user=cyrus argv=/usr/cyrus/bin/deliver -e -r ${sender} -m ${extension} ${user}

Then in main.cf the important things are myhostname: frontend.local, mydomain: local. Then make sure mydestination has $myhostname and $mydomain in it
Also add your subnet of the VMs to mynetworks.
Then configure mailbox_transport:
mailbox_transport = lmtp:unix:/var/lib/imap/socket/lmtp
#mailbox_transport = cyrus  # Can also switch to this to test both

Then at the end I configured restrictions like this (Note: this is just for testing, not actually good settings to be using):
smtpd_client_restrictions = permit_mynetworks,
        permit_sasl_authenticated,
        permit

smtpd_relay_restrictions = permit_mynetworks,
        permit_sasl_authenticated,
        reject_unauth_destination

smtpd_recipient_restrictions = permit_mynetworks,
        reject_unauth_destination,
        permit


You still need to configure your mailbox user. On the frontend you can type "cyradm frontend"
Then at the cyradm prompt, type "create user/jakub"
Then type "lm" to list the mailbox to confirm.

Then on the backend you can use ctl_mboxlist -d to make sure the mailbox list got propagated to the backend.

At this point you can test mail if all went well. Create a file under 4k bytes and one over 4kbytes, I usually just fill it with random text, then send with mailx from the frontend.
"mailx -s "Small mail" jakub < smallletter"

"mailx -s "Large mail" jakub < largeletter"

Check the log on the backend, depending on how syslog is setup it will either be in /var/log/secure or /var/log/messages about the encoded packet message.

If you have any issues with the setup I can try to answer any questions. But hopefully you will be able to verify the bug after this.

(In reply to mtt.hoyle from comment #19)
> So maybe something wrong about the way requiressf is getting set? Everything
> else in the code is from the latest version cyrus-sasl, this was the only
> change I made.

Thank you for verbose analysis. It is helpful that we know where is the problem coming from.

The   requiressf   is set few lines above based on   params->props.min_ssf   and   params->external_ssf   parameters (and they are set based on other variables from connection). They should be set by calling application? Can you peek into that with gdb, if there are not some strange values, for example uninitialised/garbage if you don't specify them explicitly by calling application?

Or even some previous source of these values, for example   conn->external.ssf  in   sasl_setprop()   ?

I will give a try to build the environment to test, but probably not today.

Finally I found some time to set up this configuration. I stumbled on some places because of some broken configuration from earlier times playing around these systems (cyrus-sasl configuration, iptables, firewalls, kerberos) and now I finally see your error message. I also learned few things about these cyrus-imap and kerberos (cool stuff! Thanks for nice guide!).

So far I played with RHEL6 backend and RHEL7 frontend and really, updating the package (removing your code chunk) on backend makes the error go away. I will do some more debugging later, since it is getting late.

The commit probably introducing your problematic feature:
https://cgit.cyrus.foundation/cyrus-sasl/commit/?id=f16192a2ef647dd2282104bc146efd8d08095532

I played with that few more hours, but frankly I am getting lost in the meaning of all these variables and branches on both client and server.
Also I ran out of ideas where to look and what for to look since I was unable to capture the place where the actual encryption of SASL data stream is happening (either in cyrus deliver or postfix lmtp).
Just putting few notes what I dig up from the gdb on the server (backend) side:

from  cyrus-imap  (lmtpd) in  gssapi_server_mech_step():

(gdb) p *params
$2 = {service = 0x7f0554ac5920 "lmtp", appname = 0x7f0554ac5c90 "Cyrus", 
  serverFQDN = 0x7f0554ac5a80 "backend.local", user_realm = 0x0, 
  iplocalport = 0x7f0554ac4654 "192.168.122.168;24", 
  ipremoteport = 0x7f0554ac4a75 "192.168.122.232;38189", servicelen = 4, applen = 5, slen = 13, 
  urlen = 0, iploclen = 18, ipremlen = 21, log_level = 7, utils = 0x7f0554ac5aa0, callbacks = 0x0, 
  props = {min_ssf = 0, max_ssf = 256, maxbufsize = 4096, security_flags = 16, property_names = 0x0, 
    property_values = 0x0}, external_ssf = 0, transition = 0, 
  canon_user = 0x7f0552915420 <_sasl_canon_user>, propctx = 0x7f0554ac5c50, spare_ptr1 = 0x0, 
  spare_ptr2 = 0x0, spare_ptr3 = 0x0, spare_ptr4 = 0x0, spare_fptr1 = 0, spare_fptr2 = 0, 
  spare_int1 = 0, spare_int2 = 0, spare_int3 = 0, flags = 0, param_version = 0}

(gdb) p *text   # (from arguments conn_context)
$4 = {state = 2, gss_ctx = 0x7f0554ad8a80, client_name = 0x7f0554adecd0, server_name = 0x7f0554ac3800, 
  server_creds = 0x0, client_creds = 0x0, limitssf = 256, requiressf = 0, qop = 7 '\a', 
  utils = 0x7f0554ac5aa0, decode_context = {utils = 0x0, needsize = 0, sizebuf = "\000\000\000", 
    size = 0, buffer = 0x0, cursize = 0, in_maxbuf = 0}, encode_buf = 0x0, decode_buf = 0x0, 
  decode_once_buf = 0x0, encode_buf_len = 0, decode_buf_len = 0, decode_once_buf_len = 0, 
  enc_in_buf = 0x0, out_buf = 0x7f0554ade900 "`\201\231\006\t*\206H\206\367\022\001\002\002\002", 
  out_buf_len = 156, authid = 0x7f0554ad94f0 "frontend", user = 0x0}

Notably text->qop = 7 = LAYER_NONE | LAYER_INTEGRITY | LAYER_CONFIDENTIALITY;


With current version (upstream) we get while decoding:

Breakpoint 1, _plug_decode (text=0x7f47ff96ea58, input=0x7f47ff9584d4 "\005\004\006\377", 
    inputlen=4092, output=0x7f47ff96ea88, outputsize=0x7f47ff96ea9c, outputlen=0x7fff5841b42c, 
    decode_pkt=0x7f47f7b9de60 <gssapi_decode_packet>, rock=0x7f47ff96ea10)
    at ../plugins/plugin_common.c:658
658			if (text->size > text->in_maxbuf) {
(gdb) p *text
$2 = {utils = 0x7f47ff95cac0, needsize = 0, sizebuf = "\000\000\020<", size = 4156, 
  buffer = 0x7f47ff992370 "\005\004\006\377", cursize = 0, in_maxbuf = 4096}

With your proposed fix the text does not get over the buffer size:

Breakpoint 1, _plug_decode (text=0x7f9218daea58, input=0x7f9218d984a4 "\005\004\006\377", 
    inputlen=4092, output=0x7f9218daea88, outputsize=0x7f9218daea9c, outputlen=0x7ffca48a6e6c, 
    decode_pkt=0x7f9211713e60 <gssapi_decode_packet>, rock=0x7f9218daea10)
    at ../plugins/plugin_common.c:658
658			if (text->size > text->in_maxbuf) {
(gdb) p *text
$1 = {utils = 0x7f9218d9ca90, needsize = 0, sizebuf = "\000\000\020", size = 4096, 
  buffer = 0x7f9218dd2430 "\005\004\006\377", cursize = 0, in_maxbuf = 4096}


Notable difference in the "decoded" sizebuffer, which has in the wrong case "<" (60 decimal) instead of \000 in the end.

I'm glad you were able to setup an environment to reproduce it. I kind of got stick around the same place. I was starting gdb from different places and set it to follow child processes, but couldn't find anything more meaningful than what you found.

I noticed your email to the cyrus-sasl mailing list, seems there were no responses? Is there any way to get in contact with any of the developers responsible for the patch that introduced this?

I have a contact to the people preparing the new bugfix release of cyrus-sasl [1] (changes are already applied in git). The problem is not applying the patch, but to understand why to apply it. Still, your setup looks like the only one having this problem and it goes away after removing "random" chunk of code. But this chunk might be useful in other cases. It might be even problem of initialization in Postfix/Cyrus-IMAP. But we were not able to prove any of these theories.


I might have some time in future to dig into that further. I will keep you informed.

[1] https://cgit.cyrus.foundation/cyrus-sasl/log/

Created attachment 1151938 [details]
Test patch to allow gssapi to work for >4k messages

I agree about not pushing this patch without really knowing more about it.

I have seen bug reports with the same message when using GSSAPI in a murder. They were reported years ago for a different version, so it seems to have come and gone. I would have to look more to see if it was caused by the same variable.

I am surprised there are not more reports though. The cyrus documentation recommends using GSSAPI/Kerberos-v5 when using a murder cluster. The way I understood this was I could not directly use GSSAPI because that means every user would have to authenticate via kerberos. We use LDAP for our users and GSSAPI for authentication between the cyrus servers for authenticating to each other via tickets. The only way I saw to make this work for both LDAP and GSSAPI was to use saslauthd and pam, so that I could configure PAM to check both ldap and krb5.
I would think quite a few other people would have the same setup, are people avoiding GSSAPI all together I wonder? 
This may not be the right place to ask. Maybe I'll send out a message to the cyrus-imap mailing list to see what people do.

When Red Hat shipped 6.8 on May 10, 2016 RHEL 6 entered Production Phase 2.
https://access.redhat.com/support/policy/updates/errata#Production_2_Phase
That means only "Critical and Important Security errata advisories (RHSAs) and Urgent Priority Bug Fix errata advisories (RHBAs) may be released". 

Moving to RHEL7, which is also affected by this issue and is more probable to receive the fix in some form.

This is https://github.com/cyrusimap/cyrus-sasl/issues/402, and it's likely that the patch in this bug is incorrect.

This is already fixed upstream:

https://github.com/cyrusimap/cyrus-sasl/issues/402

Matt,
would you have some time to verify that the latest release (cyrus-sasl-2.1.27-rc2.tar.gz) from 

    https://www.cyrusimap.org/releases/

resolves the problem for you? If so, we would like to fix that also in RHEL7.

You can either try to build it from source or use my copr repository with latest build:

https://copr.fedorainfracloud.org/coprs/jjelen/cyrus-sasl/

I will try to set up the testing environment again, if I will have time, but I can't promise that.

Jakub, Yes I will try to verify this sometime next week.