RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1210906 - Add /var/log/mysqld.log to sos mysql/mariadb plugin
Summary: Add /var/log/mysqld.log to sos mysql/mariadb plugin
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: sos
Version: 7.1
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: rc
: ---
Assignee: Pavel Moravec
QA Contact: David Kutálek
URL:
Whiteboard:
Depends On:
Blocks: 1232720 1237208 1238200
TreeView+ depends on / blocked
 
Reported: 2015-04-10 21:36 UTC by Michael Bayer
Modified: 2015-11-19 12:17 UTC (History)
12 users (show)

Fixed In Version: sos-3.2-20.el7
Doc Type: Bug Fix
Doc Text:
When using the sosreport utility on a Pacemaker node, one of the MariaDB MySQL server log-files was not properly collected. With this update, the underlying code has been corrected, and the log-file is now collected as expected.
Clone Of:
: 1232720 1237208 1238200 (view as bug list)
Environment:
Last Closed: 2015-11-19 12:17:35 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2015:2118 0 normal SHIPPED_LIVE sos bug fix and enhancement update 2015-11-19 10:50:37 UTC

Description Michael Bayer 2015-04-10 21:36:26 UTC
Description of problem:

We frequently get SOS reports where we'd like to see the status of mariadb-galera running under pacemaker, and current installer configurations log this status in the file /var/log/mysqld.log; however, this file is not present.

While there is a /var/log/mariadb/mariadb.log that we are already getting, this is typically only populated during the initial server configuration phase; the actual galera bootstrapping / clustering ends up in /var/log/mysqld.log since Mariadb is run under Pacemaker OCF resource agents for the full run, instead of via systemd which is only during setup.

looking in https://github.com/sosreport/sos/blob/master/sos/plugins/mysql.py#L35, I see that there's a file add for /var/log/mysql/mysqld.log, as well as the mariadb.log file, and then an option "all_logs" that would via wildcard add /var/log/mysqld*, but when we get an SOS report as you see attached to https://bugzilla.redhat.com/show_bug.cgi?id=1209833, this logfile isn't there.  In sos.log I don't see anything indicating this flag being used, though I'm not sure if this is something I'd see there in any case.

Whether or not the --all_logs option is something that can be turned on for the folks sending us these reports (I'm not sure what front-end they use to run this), adding the "/var/log/mysqld.log" path either to the Mysql plugin or perhaps the RedHatMysql plugin subclass I see at https://github.com/sosreport/sos/blob/master/sos/plugins/mysql.py#L79 would be a straightforward way of making sure it's present; it seems like if this plugin wants to capture the "primary" MySQL/MariaDB log then on an RH system with pacemaker running it should definitely be getting /var/log/mysqld.log.


Version-Release number of selected component (if applicable):

sos_logs/ui.log for these particular files reports "sosreport (version 3.2)".

Comment 2 Pavel Moravec 2015-04-11 15:00:05 UTC
My understanding is that this file is currently taken when adding --all-logs option, like in upstream:

setup:
# yum install mysql sos
..
# rpm -qa | egrep '(sql|maria|sos)'
mariadb-5.5.41-2.el7_0.x86_64
sqlite-3.7.17-4.el7.x86_64
mariadb-libs-5.5.41-2.el7_0.x86_64
sos-3.2-15.el7_1.1.noarch
# touch /var/log/mysql.log # just to artificially generate the file


check default plugin:
# sosreport -o mysql
..
Your sosreport has been generated and saved in:
  /var/tmp/sosreport-pmoravec-rhel7.gsslab.brq.redhat.com-20150411165050.tar.xz
..
# tar tvJf /var/tmp/sosreport-pmoravec-rhel7.gsslab.brq.redhat.com-20150411165050.tar.xz | grep mysql.log
#


check plugin with --all-logs option:
# sosreport -o mysql --all-logs
..
Your sosreport has been generated and saved in:
  /var/tmp/sosreport-pmoravec-rhel7.gsslab.brq.redhat.com-20150411165219.tar.xz
..
# tar tvJf /var/tmp/sosreport-pmoravec-rhel7.gsslab.brq.redhat.com-20150411165219.tar.xz | grep mysql.log
-rw-r--r-- root/root         0 2015-04-11 16:43 sosreport-pmoravec-rhel7.gsslab.brq.redhat.com-20150411165219/var/log/mysql.log
#

See in your installed downstream sosreport file /usr/lib/python2.7/site-packages/sos/plugins/mysql.py, line 45:

        if self.get_option("all_logs"):
            self.add_copy_spec([
                "/var/log/mysql*",
                "/var/log/mariadb*"
            ])

So my understanding is the sosreports without /var/log/mysql.log were run without option --all-logs (or maybe were run on some older sos version that did not collect the logfile).


Michael, do you agree with that? Is the current behaviour acceptable?

Comment 3 Michael Bayer 2015-04-11 22:36:33 UTC
So let's look at the intent of the mysql package and the --all-logs flag.

Without --all-logs, the MySQL plugin will in all cases grab the following files:


/var/log/mysql/mysqld.log
/var/log/mariadb/mariadb.log

With --all-logs, it additionally adds full directories:

/var/log/mysql*
/var/log/mariadb


I don't know where any of this is documented (there's nothing at http://sos.readthedocs.org/en/latest/index.html), so I can only guess as to the intent of this flag.  However, the intent seems fairly clear:


1. the mysql plugin should grab all *primary* log files by default.     this is why it hardcodes paths to mariadb.log, mysqld.log.

2. The mysql plugin should additionally grab entire directories which contain *archived* / rotated log files when --all-logs is set.    This is why it is hardcoding directory names and using wildcards.

In this case, we are looking for a *primary* log file, not archived / date-rotated log files.   That is, we want the mysqld.log.  It's just as hardcoded right now, the mysql plugin is using the wrong path for a Red Hat Pacemaker installation.   So the proposal is just to add this alternate path for the primary log file.    We'd rather not receive full directories of archived / log-rotated files nor do we have a reliable way to communicate the use of this flag to all customer and QE agents.

Comment 4 Bryn M. Reeves 2015-04-13 11:35:31 UTC
That's correct; --all-logs is for the case where you really do want *all* logs (current, rotated, any random garbage the admin left lying around in there..). It's explicitly expected to scoop up whole directories (where applicable) and warns the user that this may significantly inflate the report size in the man page and online usage info:

       --all-logs
              Tell  plugins  to  collect all possible log data ignoring any size 
              limits and including logs in non-default locations. This option
              may significantly increase the size of reports.


It is not intended to be used to collect "routine" current log files (although in many cases it may be used as a temporary workaround in the case that a current log file is absent from the plugin).

I'd like to understand better why mysql.log exists in different places but regardless it seems that this should be added to the base collection (at least for the RedHatPlugin classes).

Comment 5 Michael Bayer 2015-04-13 15:13:36 UTC
> I'd like to understand better why mysql.log exists in different places 

It's something we want to correct for on the other end as well, but currently under Pacemaker with MariaDB-Galera, Pacemaker uses OCF resource agents to start and stop Mariadb, bypassing the usual systemd system, and sets up an explicit log path which currently defaults to /var/log/mysqld.log (see https://github.com/ClusterLabs/resource-agents/blob/master/heartbeat/mysql-common.sh#L21).  This also overrides the /var/log/mariadb/mariadb.log location that is normally in our my.cnf setup.

Comment 6 Pavel Moravec 2015-04-17 15:51:42 UTC
Upstream issue: https://github.com/sosreport/sos/issues/554

Comment 7 Pavel Moravec 2015-05-31 12:01:28 UTC
Committed to upstream as https://github.com/sosreport/sos/commit/f8ab86cbd2302812cd9160bec4478cb4f84e1442

Comment 16 errata-xmlrpc 2015-11-19 12:17:35 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2015-2118.html


Note You need to log in before you can comment on or make changes to this bug.