Bug 858197

using both libqb and corosync master branch from git as of today.

        logger_subsys {
                subsys: TOTEM
                debug: off
        }

corosync-cmapctl -s logging.logger_subsys.TOTEM.debug str trace

It appears that there is memory corruption and list entries duplication in call site handling.

libtotem does not link with libqb nor uses libqb directly.

corosync/exec/main.c defines _logsys_log_printf as:

static void
_logsys_log_printf(int level, int subsys,
                const char *function_name,
                const char *file_name,
                int file_line,
                const char *format, ...)
{
        va_list ap;

        va_start(ap, format);
        qb_log_from_external_source_va(function_name, file_name,
                                    format, level, file_line,
                                    subsys, ap);
        va_end(ap);
}

libtotem (and all parts of totem) are passed that function as logging function.
Including other bits such as subsys.

Every call libtotem does to log_printf is an alias to totem_log_printf that jumps back to qb_log_from_external_source_va.
Also to be noted that TOTEM "filename" for filtering is defined as totem.

All other subsystems are using qb_log directly. This is the first and most important difference.

qb_log_from_external_source_va -> calls into qb_log_callsite_get that
calls into qb_log_dcs_get.

For some reasons, that I can´t really figure out yet, qb_log_dcs_get, does allocated different callsite for the same file and I suspect that there is a memory leak and memory corruption. I don´t exclude that what we do in corosync is wrong, but....

diff --git a/exec/main.c b/exec/main.c
index c21f47f..140845f 100644
--- a/exec/main.c
+++ b/exec/main.c
@@ -827,11 +827,17 @@ _logsys_log_printf(int level, int subsys,
                const char *format, ...)
 {
        va_list ap;
+       struct qb_log_callsite *cs = qb_log_callsite_get(function_name,
+                                                        file_name,
+                                                        format,
+                                                        level,
+                                                        file_line,
+                                                        0);
+
+       fprintf(stderr, "DEBUG: callsite: %p %s\n", cs, file_name);
 
        va_start(ap, format);
-       qb_log_from_external_source_va(function_name, file_name,
-                                   format, level, file_line,
-                                   subsys, ap);
+       qb_log_real_va_(cs, ap);
        va_end(ap);
 }

changing _logsys_log_printf as in this patch shows:
(short snippet)
Sep 19 13:49:26 trace   [TOTEM ] Received ringid(192.168.2.97:304) seq 1
DEBUG: callsite: 0x7fec263c3320 /home/fabbione/work/cluster/corosync/corosync/exec/totemsrp.c
Sep 19 13:49:26 trace   [YKD   ] Received ringid(192.168.2.97:304) seq 2
DEBUG: callsite: 0x7fec263be110 /home/fabbione/work/cluster/corosync/corosync/exec/totemsrp.c
DEBUG: callsite: 0x7fec263be138 /home/fabbione/work/cluster/corosync/corosync/exec/totemsrp.c

that doesn´t look correct at all. subsystem has become YKD from TOTEM (note i don´t load anything in corosync.conf) after changing debug=off to debug=trace.

and it´s odd to see that new cs are allocated (this is just a short summary of a full run).

Adding debugging log to libqb somehow confirms that new_dcs are allocated from time to time.

Should be fixed by:
https://github.com/asalkeld/libqb/commit/6e130725e4eadee54f4725ba2959a6be88a221a4
https://github.com/asalkeld/libqb/commit/a623af969628e14e1ad76e05e3ac37eac6c83bf9
https://github.com/asalkeld/libqb/commit/435d1dbce2073c18f2eeb318c2ed8782979ccae6
https://github.com/asalkeld/libqb/commit/5e955579cb0a3396e1732e017619a135a216a616

(I'll make a release soon)

I think the logsys.c can be largely removed if we use the following function: qb_log_filter_fn_set().

libqb-0.14.3-2.fc17 has been submitted as an update for Fedora 17.
https://admin.fedoraproject.org/updates/libqb-0.14.3-2.fc17

libqb-0.14.3-2.fc18 has been submitted as an update for Fedora 18.
https://admin.fedoraproject.org/updates/libqb-0.14.3-2.fc18

Package libqb-0.14.3-2.fc18:
* should fix your issue,
* was pushed to the Fedora 18 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing libqb-0.14.3-2.fc18'
as soon as you are able to.
Please go to the following url:
https://admin.fedoraproject.org/updates/FEDORA-2012-17170/libqb-0.14.3-2.fc18
then log in and leave karma (feedback).

libqb-0.14.3-2.fc17 has been pushed to the Fedora 17 stable repository.  If problems still persist, please make note of it in this bug report.