RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1261830 - crm_report should detect mixed corosync and pacemaker messages in single log
Summary: crm_report should detect mixed corosync and pacemaker messages in single log
Keywords:
Status: CLOSED WONTFIX
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: pacemaker
Version: 7.3
Hardware: Unspecified
OS: Unspecified
medium
high
Target Milestone: rc
: ---
Assignee: Ken Gaillot
QA Contact: cluster-qe@redhat.com
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2015-09-10 09:38 UTC by Jaroslav Kortus
Modified: 2019-11-04 19:03 UTC (History)
5 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-11-04 19:03:41 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)

Description Jaroslav Kortus 2015-09-10 09:38:08 UTC
Description of problem:
Some lines of corosync.log do have timestamps prepended, while some others do not. This later confuses crm_report and it discards that log file completely.

Version-Release number of selected component (if applicable):
corosync-2.3.4-7.el7.x86_64

How reproducible:
100%

Steps to Reproduce:
1. set up a cluster via pcs
2.
3.

Actual results:
corosync.log contains log entries with no timestamps, followed by timestamped ones
crm_report does not contain corosync.log from all nodes

Expected results:
timestamps everywhere
crm_report collecting the log files

Additional info:
My current example:

[root@virt-085 cluster]# cat corosync.log
[8651] virt-085.cluster-qe.lab.eng.brq.redhat.com corosyncnotice  [MAIN  ] Corosync Cluster Engine ('2.3.4'): started and ready to provide service.
[8651] virt-085.cluster-qe.lab.eng.brq.redhat.com corosyncinfo    [MAIN  ] Corosync built-in features: dbus systemd xmlconf snmp pie relro bindnow
[8651] virt-085.cluster-qe.lab.eng.brq.redhat.com corosyncnotice  [TOTEM ] Initializing transport (UDP/IP Unicast).
[8651] virt-085.cluster-qe.lab.eng.brq.redhat.com corosyncnotice  [TOTEM ] Initializing transmit/receive security (NSS) crypto: none hash: none
[8651] virt-085.cluster-qe.lab.eng.brq.redhat.com corosyncnotice  [TOTEM ] The network interface [10.34.71.85] is now up.
[8651] virt-085.cluster-qe.lab.eng.brq.redhat.com corosyncnotice  [SERV  ] Service engine loaded: corosync configuration map access [0]
[8651] virt-085.cluster-qe.lab.eng.brq.redhat.com corosyncinfo    [QB    ] server name: cmap
[8651] virt-085.cluster-qe.lab.eng.brq.redhat.com corosyncnotice  [SERV  ] Service engine loaded: corosync configuration service [1]
[8651] virt-085.cluster-qe.lab.eng.brq.redhat.com corosyncinfo    [QB    ] server name: cfg
[8651] virt-085.cluster-qe.lab.eng.brq.redhat.com corosyncnotice  [SERV  ] Service engine loaded: corosync cluster closed process group service v1.01 [2]
[8651] virt-085.cluster-qe.lab.eng.brq.redhat.com corosyncinfo    [QB    ] server name: cpg
[8651] virt-085.cluster-qe.lab.eng.brq.redhat.com corosyncnotice  [SERV  ] Service engine loaded: corosync profile loading service [4]
[8651] virt-085.cluster-qe.lab.eng.brq.redhat.com corosyncnotice  [QUORUM] Using quorum provider corosync_votequorum
[8651] virt-085.cluster-qe.lab.eng.brq.redhat.com corosyncnotice  [SERV  ] Service engine loaded: corosync vote quorum service v1.0 [5]
[8651] virt-085.cluster-qe.lab.eng.brq.redhat.com corosyncinfo    [QB    ] server name: votequorum
[8651] virt-085.cluster-qe.lab.eng.brq.redhat.com corosyncnotice  [SERV  ] Service engine loaded: corosync cluster quorum service v0.1 [3]
[8651] virt-085.cluster-qe.lab.eng.brq.redhat.com corosyncinfo    [QB    ] server name: quorum
[8651] virt-085.cluster-qe.lab.eng.brq.redhat.com corosyncnotice  [TOTEM ] adding new UDPU member {10.34.71.79}
[8651] virt-085.cluster-qe.lab.eng.brq.redhat.com corosyncnotice  [TOTEM ] adding new UDPU member {10.34.71.85}
[8651] virt-085.cluster-qe.lab.eng.brq.redhat.com corosyncnotice  [TOTEM ] adding new UDPU member {10.34.71.86}
[8651] virt-085.cluster-qe.lab.eng.brq.redhat.com corosyncnotice  [TOTEM ] A new membership (10.34.71.85:20) was formed. Members joined: 2
[8651] virt-085.cluster-qe.lab.eng.brq.redhat.com corosyncnotice  [QUORUM] Members[1]: 2
[8651] virt-085.cluster-qe.lab.eng.brq.redhat.com corosyncnotice  [MAIN  ] Completed service synchronization, ready to provide service.
[8651] virt-085.cluster-qe.lab.eng.brq.redhat.com corosyncnotice  [TOTEM ] A new membership (10.34.71.79:24) was formed. Members joined: 1 3
[8651] virt-085.cluster-qe.lab.eng.brq.redhat.com corosyncnotice  [QUORUM] This node is within the primary component and will provide service.
[8651] virt-085.cluster-qe.lab.eng.brq.redhat.com corosyncnotice  [QUORUM] Members[3]: 1 2 3
[8651] virt-085.cluster-qe.lab.eng.brq.redhat.com corosyncnotice  [MAIN  ] Completed service synchronization, ready to provide service.
Set r/w permissions for uid=189, gid=189 on /var/log/cluster/corosync.log
Sep 10 10:40:56 [8678] virt-085.cluster-qe.lab.eng.brq.redhat.com pacemakerd:   notice: mcp_read_config:        Configured corosync to accept connections from group 189: OK (1)
Sep 10 10:40:56 [8678] virt-085.cluster-qe.lab.eng.brq.redhat.com pacemakerd:   notice: main:   Starting Pacemaker 1.1.13 (Build: 44eb2dd):  generated-manpages agent-manpages ncurses libqb-logging libqb-ipc upstart systemd nagios  corosync-native atomic-attrd acls
Sep 10 10:40:56 [8678] virt-085.cluster-qe.lab.eng.brq.redhat.com pacemakerd:     info: main:   Maximum core file size is: 18446744073709551615
Sep 10 10:40:56 [8678] virt-085.cluster-qe.lab.eng.brq.redhat.com pacemakerd:     info: qb_ipcs_us_publish:     server name: pacemakerd


crm_report symptom:
[root@virt-086 ~]# crm_report --from "2011-01-01 00:00:00" report-out
virt-086:   Calculated node list: virt-079 virt-085 virt-086 
virt-086:   Collecting data from virt-079 virt-085 virt-086  (01/01/2011 12:00:00 AM to 09/10/2015 11:30:58 AM)
virt-079:   WARN: Unknown time format used by: 
virt-079:   WARN: Unknown time format used by: 
virt-079:   WARN: Unknown time format used by: 
virt-079:   WARN: Unknown time format used by: 
virt-079:   WARN: Unknown time format used by: 
virt-079:   WARN: Unknown time format used by: 
virt-079:   WARN: Unknown time format used by: 
virt-079:   WARN: Unknown time format used by: 
virt-079:   WARN: Unknown time format used by: 
virt-079:   WARN: Unknown time format used by: 
virt-079:   WARN: Unknown time format used by: 
virt-079:   WARN: Unknown time format used by: 
virt-079:   WARN: Unknown time format used by: 
virt-079:   WARN: Unknown time format used by: 
virt-079:   WARN: Unknown time format used by: 
virt-079:   WARN: Unknown time format used by: 
virt-079:   WARN: Unknown time format used by: 
virt-079:   WARN: Unknown time format used by: 
virt-079:   WARN: Unknown time format used by: 
virt-079:   WARN: Unknown time format used by: 
virt-079:   WARN: Skipping bad logfile '/var/log/cluster/corosync.log': Could not determine log dates

Comment 3 Jan Friesse 2016-01-25 16:22:49 UTC
Capacity constrained, moving to 7.4 (7.3 is mostly about Qdevice).

Comment 6 Jan Friesse 2018-07-10 12:00:38 UTC
I've kept opened this BZ because of possible corosync.log overwrite (both corosync and pacemaker opens this file), but this is not going to be fixed in RHEL-7 (on the other hand it's already fixed in upstream Pacemaker 2.0).

Fixing date is little tricky. We could include patch in corosync to make "timestamp:on" default, but I believe it would be better to be fixed in pcs (so to add line timestamp: on to generated corosync.conf).

Another option may be to "fix" crm_report (if possible) to detect if line begins with date or not.

Ken, any idea if "fixing" crm_report would be doable? If so, could you please reassign this BZ?

Comment 8 Ken Gaillot 2018-07-10 14:23:13 UTC
(In reply to Jan Friesse from comment #6)
> Another option may be to "fix" crm_report (if possible) to detect if line
> begins with date or not.
> 
> Ken, any idea if "fixing" crm_report would be doable? If so, could you
> please reassign this BZ?

Agreed.

crm_report is currently highly inefficient because it's a shell script, when something like python would be much more suitable at this stage of its complexity. The current date-searching code is a prime example of where language facilities would help. It's something I want to do every time I look at its code :) so this bz will be a good reason to put it on the radar.

Comment 9 Ken Gaillot 2019-11-04 19:03:41 UTC
Due to limited developer time, we're going to have to let this one go. In RHEL 8, the pacemaker log is separate from the corosync log, so the pacemaker log will always be collected. Also, when crm_report is run by sosreport, sosreport handles the log collection, avoiding the issue.


Note You need to log in before you can comment on or make changes to this bug.