Bug 1538066 - Fails to start after upgrade to 10.1.30: /run/mariadb missing
Summary: Fails to start after upgrade to 10.1.30: /run/mariadb missing
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: mariadb
Version: 26
Hardware: x86_64
OS: Linux
unspecified
high
Target Milestone: ---
Assignee: Michal Schorm
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-01-24 12:18 UTC by Cesar Eduardo Barros
Modified: 2018-02-13 18:46 UTC (History)
15 users (show)

Fixed In Version: mariadb-10.1.30-2.fc26
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2018-01-28 21:15:26 UTC
Type: Bug


Attachments (Terms of Use)

Description Cesar Eduardo Barros 2018-01-24 12:18:39 UTC
Description of problem:

After upgrading to 3:10.1.30-1.fc26.x86_64 and rebooting, mariadb fails to start.

The /var/log/mariadb/mariadb.log file contains the following error message:

[ERROR] mysqld: Can't create/write to file '/run/mariadb/mariadb.pid' (Errcode: 2 "No such file or directory")
[ERROR] Can't start server: can't create PID file: No such file or directory

The /run/mariadb directory does not exist.


Version-Release number of selected component (if applicable):

mariadb-server-10.1.30-1.fc26.x86_64


How reproducible:

Tried only once.


Steps to Reproduce:
1. Upgrade from 10.1.29-1.fc26.x86_64 to 10.1.30-1.fc26.x86_64
2. Reboot
3. systemctl start mariadb


Actual results:

The start of the mariadb.service unit fails. There are no useful messages in "systemctl status mariadb" or the journal.

The message above can be found in mariadb's own log file.


Expected results:

The /run/mariadb directory should have been created on boot.


Additional info:

The commit at http://pkgs.fedoraproject.org/cgit/rpms/mariadb.git/commit/?h=f26&id=f71e91b3792f79b9d582a1ca1798fc6551c148dd copies a new file to tmpfiles.d, apparently overwriting an earlier file (http://pkgs.fedoraproject.org/cgit/rpms/mariadb.git/tree/mysql.tmpfiles.d.in?h=f26) which should have created the /run/mariadb directory on every boot. (See lines 825 and 876 of the spec file.)

The /usr/lib/tmpfiles.d/mariadb.conf file contains only

d /var/lib/mysql 0755 mysql mysql -

I believe it should also contain a line creating /run/mariadb, with the correct permissions.

Comment 1 John DeDourek 2018-01-24 15:57:10 UTC
Manually creating /run/mariadb and changing ownership to mysql.mysql allows mariadb to start normally.  However, on the next reboot, the directory again disappears.

Comment 2 Tim Elliott 2018-01-24 19:24:35 UTC
I'm seeing a similar problem in F27 after upgrading to mariadb.x86_64, version 3:10.2.10-2.fc27.

In my case, /var/log/mariadb/mariadb.log shows:

  [ERROR] mysqld: Can't create/write to file '/run/mysqld/mariadb.pid' (Errcode: 2 "No such file or directory")

And I had to manually create /run/mysqld and chown to mysql.mysql for mariadb to start up again.

Comment 3 Daniel Laczi 2018-01-24 21:22:09 UTC
(In reply to Tim Elliott from comment #2)
> I'm seeing a similar problem in F27 after upgrading to mariadb.x86_64,
> version 3:10.2.10-2.fc27.
> 
> In my case, /var/log/mariadb/mariadb.log shows:
> 
>   [ERROR] mysqld: Can't create/write to file '/run/mysqld/mariadb.pid'
> (Errcode: 2 "No such file or directory")
> 
> And I had to manually create /run/mysqld and chown to mysql.mysql for
> mariadb to start up again.

This might help you: https://bugzilla.redhat.com/show_bug.cgi?id=1483331

Comment 4 Daniel Laczi 2018-01-24 22:40:53 UTC
(In reply to John DeDourek from comment #1)
> Manually creating /run/mariadb and changing ownership to mysql.mysql allows
> mariadb to start normally.  However, on the next reboot, the directory again
> disappears.

Thanks.

Quick workaround that works for me:

echo -e '#!/bin/bash\n\n#see https://bugzilla.redhat.com/show_bug.cgi?id=1538066\nmkdir /run/mariadb && chown mysql.mysql /run/mariadb\n' >> /usr/local/bin/mariadbfix.sh && chown root:root /usr/local/bin/mariadbfix.sh && chmod 744 /usr/local/bin/mariadbfix.sh && echo -e '[Unit]\nDescription=Fix for mariadb, see https://bugzilla.redhat.com/show_bug.cgi?id=1538066\nBefore=mariadb.service\n\n[Service]\nExecStart=/usr/local/bin/mariadbfix.sh\n\n[Install]\nWantedBy=multi-user.target\n' >> /usr/lib/systemd/system/mariadbfix.service && chown root:root /usr/lib/systemd/system/mariadbfix.service && chmod 644 /usr/lib/systemd/system/mariadbfix.service && systemctl enable mariadbfix.service

It basically starts a script which creates /run/mariadb before mariadb is started.

Remove the workaround with:

systemctl disable mariadbfix.service && rm /usr/local/bin/mariadbfix.sh && rm /usr/lib/systemd/system/mariadbfix.service

Comment 5 Michal Schorm 2018-01-25 01:10:04 UTC
I took a deep look at what might be the cause.

I found out, it is most likely caused by incorrect configuration of tmpfiles. (/usr/lib/tmpfiles.d/mariadb.conf)

The upstream has recently started to ship their own, however it wasn't compatible with our downstream configuration.



I fixed the isue in the SPECfile and prepared the new build. The issue *should be* solved by update to the new version. I didn't need any manual work in my test environment.


Please test it and let me know, if the issues persists.

Comment 6 Fedora Update System 2018-01-25 01:10:50 UTC
mariadb-10.1.30-2.fc26 has been submitted as an update to Fedora 26. https://bodhi.fedoraproject.org/updates/FEDORA-2018-e2bc1c0aab

Comment 7 Fedora Update System 2018-01-25 07:55:47 UTC
mariadb-10.1.30-2.fc26 has been pushed to the Fedora 26 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2018-e2bc1c0aab

Comment 8 linux4desktop 2018-01-25 11:55:20 UTC
Hi,

i tried testing the fix with 
dnf config-manager --set-enabled updates-testing
dnf update --refresh

it offers to 
Install   1 Package
Upgrade  58 Packages

But mariadb-  is not in the list.
Do i need to take another repo for that package or just wait a bit?

Comment 9 Michal Schorm 2018-01-25 12:10:51 UTC
Dunno, but on my side, both your method and my
 # dnf update mariadb --enablerepo=updates-testing
found the correct version.

Maybe, the update did not yet reached your mirror, but I'd call it unlikely.

You still can download it manually from the KOJI:
https://koji.fedoraproject.org/koji/taskinfo?taskID=24423405
(yeah, a little pain but should always work)

Comment 10 linux4desktop 2018-01-26 08:20:46 UTC
seemed to be the mirror: 3hrs later i got the packages and today i got one more mariadb-configuration.

But now there is another error:

[root@oc ~]# systemctl status mariadb 
● mariadb.service - MariaDB 10.1 database server
   Loaded: loaded (/usr/lib/systemd/system/mariadb.service; enabled; vendor preset: disabled)
   Active: deactivating (stop-sigterm) (Result: timeout)
  Process: 794 ExecStartPre=/usr/libexec/mysql-prepare-db-dir mariadb.service (code=exited, status=0/SUCCESS)
  Process: 718 ExecStartPre=/usr/libexec/mysql-check-socket (code=exited, status=0/SUCCESS)
 Main PID: 844 (mysqld)
   Status: "InnoDB: Error: Could not find a valid tablespace file for 'owncloud/oc_authtoken'. See http://dev.mysql.com/doc/refman/5.6/en/innodb-troubleshooting-datadict.html for how to resolve the issue."
    Tasks: 16 (limit: 4915)
   CGroup: /system.slice/mariadb.service
           └─844 /usr/libexec/mysqld --basedir=/usr

Jan 26 09:06:33 oc.server.com systemd[1]: Starting MariaDB 10.1 database server...
Jan 26 09:06:33 oc.server.com mysql-check-socket[718]: Socket file /var/lib/mysql/mysql.sock exists.
Jan 26 09:06:33 oc.server.com mysql-prepare-db-dir[794]: Database MariaDB is probably initialized in /var/lib/mysql already, nothing is done.
Jan 26 09:06:33 oc.server.com mysql-prepare-db-dir[794]: If this is not the case, make sure the /var/lib/mysql is empty before running mysql-prepare-db-dir.
Jan 26 09:06:34 oc.server.com mysqld[844]: 2018-01-26  9:06:34 139838212134080 [Note] /usr/libexec/mysqld (mysqld 10.1.30-MariaDB) starting as process 844 ...
Jan 26 09:11:34 oc.server.com systemd[1]: mariadb.service: Start operation timed out. Terminating.
[root@oc ~]# ps aux | grep -i mysq
mysql      844  0.1  5.1 744852 104496 ?       Ssl  09:06   0:00 /usr/libexec/mysqld --basedir=/usr
root      1203  0.0  0.0 119468   964 pts/0    S+   09:14   0:00 grep --color=auto -i mysq

I don't know if it is still from the broken update, or from my tries to fix it.
Now it's missing a tablespace, although the file is lying in /var/lib/mysql/owncloud/oc_authtoken.
This looks like restoring the backup :-(

Comment 11 linux4desktop 2018-01-26 08:46:51 UTC
Fixed the issue. owncloud databse folder had root:root. Chowning that to mysql:mysql fixed it. Probably this was me trying to fix it.

8 drwx------.  2 root  root      4096 Jan 24 17:05 owncloud
chown -R mysql:mysql /var/lib/mysql/owncloud/

Now the service is starting fine also after reboot.
Thanks!

Comment 12 Michal Schorm 2018-01-26 09:27:35 UTC
Glad to hear it fully works again :)

Comment 13 Fedora Update System 2018-01-28 21:15:26 UTC
mariadb-10.1.30-2.fc26 has been pushed to the Fedora 26 stable repository. If problems still persist, please make note of it in this bug report.

Comment 14 Sunil P 2018-02-05 02:52:03 UTC
I had similar issue with mariadb startup on Fedora 27. Manually creating directory /run/mariadb and setting it's ownership to mysql:mysql works but then again reboot wipe out /run directory and mariadb startup fail again. Debug messages are not very helpful.

My mariadb version is mariadb-10.2.10-2.fc27.x86_64.

Here is my custom mariadb startup file which survive reboot.

$cat /etc/systemd/system/mariadb.service
#Custom service to fix mariadb startup issue with system reboot.
.include /usr/lib/systemd/system/mariadb.service
[Unit]
Description=MariaDB 10.2 database server
After=network.target

[Service]
Type=Idle
User=mysql
Group=mysql
PermissionsStartOnly=true
ExecStartPre=/bin/bash -c "/bin/mkdir -p /run/mariadb && /bin/chown mysql:mysql /run/mariadb"

$systemctl daemon-reload

Comment 15 Tim Elliott 2018-02-13 18:12:04 UTC
(In reply to Daniel Laczi from comment #3)
> (In reply to Tim Elliott from comment #2)
> > I'm seeing a similar problem in F27 after upgrading to mariadb.x86_64,
> > version 3:10.2.10-2.fc27.
> > 
> > In my case, /var/log/mariadb/mariadb.log shows:
> > 
> >   [ERROR] mysqld: Can't create/write to file '/run/mysqld/mariadb.pid'
> > (Errcode: 2 "No such file or directory")
> > 
> > And I had to manually create /run/mysqld and chown to mysql.mysql for
> > mariadb to start up again.
> 
> This might help you: https://bugzilla.redhat.com/show_bug.cgi?id=1483331

Just wanted to give you a belated thanks. I do have a change in /etc/mariadb-server.cnf (to change the datadir setting out of the root filesystem) and I had to merge my config with the .rpmnew version.

Everything works now.

Comment 16 Sunil P 2018-02-13 18:46:33 UTC
(In reply to Sunil P from comment #14)
> I had similar issue with mariadb startup on Fedora 27. Manually creating
> directory /run/mariadb and setting it's ownership to mysql:mysql works but
> then again reboot wipe out /run directory and mariadb startup fail again.
> Debug messages are not very helpful.
> 
> My mariadb version is mariadb-10.2.10-2.fc27.x86_64.
> 
> Here is my custom mariadb startup file which survive reboot.
> 
> $cat /etc/systemd/system/mariadb.service
> #Custom service to fix mariadb startup issue with system reboot.
> .include /usr/lib/systemd/system/mariadb.service
> [Unit]
> Description=MariaDB 10.2 database server
> After=network.target
> 
> [Service]
> Type=Idle
> User=mysql
> Group=mysql
> PermissionsStartOnly=true
> ExecStartPre=/bin/bash -c "/bin/mkdir -p /run/mariadb && /bin/chown
> mysql:mysql /run/mariadb"
> 
> $systemctl daemon-reload

It was working but fedora update broke this setup as it doesn't understand Type parameter and doesn't start mariadb.

/etc/systemd/system/mariadb.service:8: Failed to parse service type, ignoring: Idle

Here is corrected one which work with latest Fedora and survive reboot as of now.

$cat /etc/systemd/system/mariadb.service
#Custom service to fix mariadb startup issue with system reboot.
.include /usr/lib/systemd/system/mariadb.service
[Unit]
Description=MariaDB 10.2 database server
After=network.target
 
[Service]
User=mysql
Group=mysql
PermissionsStartOnly=true
ExecStartPre=/bin/bash -c "/bin/mkdir -p /run/mariadb && /bin/chown mysql:mysql /run/mariadb"
 
$systemctl daemon-reload


Note You need to log in before you can comment on or make changes to this bug.