Bug 835131 - PrivateTmp breaks if /var/tmp is a symlink
Summary: PrivateTmp breaks if /var/tmp is a symlink
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Fedora
Classification: Fedora
Component: systemd
Version: 20
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: systemd-maint
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
: 901926 (view as bug list)
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2012-06-25 15:40 UTC by Pavel Alexeev
Modified: 2016-09-23 07:57 UTC (History)
20 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2015-04-03 11:09:19 UTC
Type: Bug
Embargoed:


Attachments (Terms of Use)
Patch for mysqld-prepare-db-dir that clean up $datadir if not initialized properly (864 bytes, patch)
2012-08-23 14:50 UTC, Honza Horak
no flags Details | Diff
my.cnf (479 bytes, text/plain)
2012-08-28 12:54 UTC, Pavel Alexeev
no flags Details

Description Pavel Alexeev 2012-06-25 15:40:54 UTC
Description of problem:
Installation of MySQL which was early deleted.
Delete /var/log/mysql directory.
Try start:
systemctl start mysqld.service
Job failed. See system logs and 'systemctl status' for details.

In system log:
Jun 25 19:37:28 linux systemd[27615]: Failed at step NAMESPACE spawning /usr/libexec/mysqld-prepare-db-dir: No such file or directory
Jun 25 19:37:28 linux systemd[1]: mysqld.service: control process exited, code=exited status=226
Jun 25 19:37:28 linux systemd[1]: mysqld.service holdoff time over, scheduling restart.
Jun 25 19:37:28 linux systemd[1]: Job pending for unit, delaying automatic restart.
Jun 25 19:37:28 linux systemd[1]: Unit mysqld.service entered failed state.
Jun 25 19:37:28 linux systemd[27617]: Failed at step NAMESPACE spawning /usr/libexec/mysqld-prepare-db-dir: No such file or directory
Jun 25 19:37:28 linux systemd[1]: mysqld.service: control process exited, code=exited status=226
Jun 25 19:37:29 linux systemd[1]: mysqld.service holdoff time over, scheduling restart.
Jun 25 19:37:29 linux systemd[1]: Job pending for unit, delaying automatic restart.
Jun 25 19:37:29 linux systemd[1]: Unit mysqld.service entered failed state.
Jun 25 19:37:29 linux systemd[27619]: Failed at step NAMESPACE spawning /usr/libexec/mysqld-prepare-db-dir: No such file or directory
Jun 25 19:37:29 linux systemd[1]: mysqld.service: control process exited, code=exited status=226
Jun 25 19:37:29 linux systemd[1]: mysqld.service holdoff time over, scheduling restart.
Jun 25 19:37:29 linux systemd[1]: Job pending for unit, delaying automatic restart.
Jun 25 19:37:29 linux systemd[1]: Unit mysqld.service entered failed state.
Jun 25 19:37:29 linux systemd[27621]: Failed at step NAMESPACE spawning /usr/libexec/mysqld-prepare-db-dir: No such file or directory
Jun 25 19:37:29 linux systemd[1]: mysqld.service: control process exited, code=exited status=226
Jun 25 19:37:29 linux systemd[1]: mysqld.service holdoff time over, scheduling restart.
Jun 25 19:37:29 linux systemd[1]: Job pending for unit, delaying automatic restart.
Jun 25 19:37:29 linux systemd[1]: Unit mysqld.service entered failed state.
Jun 25 19:37:29 linux systemd[27623]: Failed at step NAMESPACE spawning /usr/libexec/mysqld-prepare-db-dir: No such file or directory
Jun 25 19:37:29 linux systemd[1]: mysqld.service: control process exited, code=exited status=226
Jun 25 19:37:29 linux systemd[1]: mysqld.service holdoff time over, scheduling restart.
Jun 25 19:37:29 linux systemd[1]: Job pending for unit, delaying automatic restart.
Jun 25 19:37:29 linux systemd[1]: Unit mysqld.service entered failed state.
Jun 25 19:37:29 linux systemd[27625]: Failed at step NAMESPACE spawning /usr/libexec/mysqld-prepare-db-dir: No such file or directory
Jun 25 19:37:29 linux systemd[1]: mysqld.service: control process exited, code=exited status=226
Jun 25 19:37:29 linux systemd[1]: mysqld.service holdoff time over, scheduling restart.
Jun 25 19:37:29 linux systemd[1]: Job pending for unit, delaying automatic restart.
Jun 25 19:37:29 linux systemd[1]: Unit mysqld.service entered failed state.
Jun 25 19:37:29 linux systemd[1]: mysqld.service start request repeated too quickly, refusing to start.

SELinux disabled:
# getenforce 
Disabled

Manual setup directory successful but does nothing:
# /usr/libexec/mysqld-prepare-db-dir
Initializing MySQL database
Installing MySQL system tables...
OK
Filling help tables...
OK

To start mysqld at boot time you have to copy
support-files/mysql.server to the right place for your system

PLEASE REMEMBER TO SET A PASSWORD FOR THE MySQL root USER !
To do so, start the server, then issue the following commands:

/usr/bin/mysqladmin -u root password 'new-password'
/usr/bin/mysqladmin -u root -h linux password 'new-password'

Alternatively you can run:
/usr/bin/mysql_secure_installation

which will also give you the option of removing the test
databases and anonymous user created by default.  This is
strongly recommended for production servers.

See the manual for more instructions.

You can start the MySQL daemon with:
cd /usr ; /usr/bin/mysqld_safe &

You can test the MySQL daemon with mysql-test-run.pl
cd /usr/mysql-test ; perl mysql-test-run.pl

Please report any problems with the /usr/bin/mysqlbug script!

# systemctl start mysqld.service
Job failed. See system logs and 'systemctl status' for details.

In logs the same information only.

Version-Release number of selected component (if applicable):
# rpm -qa 'mysql*'
mysql-server-5.5.23-1.fc16.x86_64
mysql-libs-5.5.23-1.fc16.x86_64
mysql-5.5.23-1.fc16.x86_64
mysql-utilities-1.0.5-2.fc16.noarch
mysql-connector-python-0.3.2-2.fc16.noarch

How reproducible:
Always

Comment 1 Tom Lane 2012-06-25 16:22:01 UTC
I don't think this is so much a bug as a "don't do that".  The log file is created on package install, and you really shouldn't be removing it by hand, any more than any other file installed by the package.

The reason mysql was able to create the log file for itself in previous versions was that it was started as root.  We don't do that anymore, and I think the gain in security is worth this small annoyance.

There is a discussion going on in bug #832035, which unfortunately some overprotective soul has marked Red Hat internal, about whether there is any way to recreate the log file at service start without adding any security risk.  So far no workable ideas though :-(

Comment 2 Pavel Alexeev 2012-06-26 16:05:54 UTC
(In reply to comment #1)
> I don't think this is so much a bug as a "don't do that".  The log file is
> created on package install, and you really shouldn't be removing it by hand,
> any more than any other file installed by the package.
About what file are you speaking? I had not delete any MySQL log files. /var/log/mysql was removed (renamed) only after MySQL wasn't be able start with it. And it does not helps also.

Comment 3 Tom Lane 2012-06-26 16:49:01 UTC
Well, I assumed that

> Delete /var/log/mysql directory.

was a typo for /var/log/mysqld.log, because the mysql package doesn't create any directory named that and wouldn't care if someone else did.  Maybe you meant something else?  Or did you change some configuration settings that you haven't mentioned?

Comment 4 Pavel Alexeev 2012-06-26 19:38:15 UTC
I'm very sorry. I mean /var/lib/mysql directory.

Comment 5 Tom Lane 2012-06-26 20:26:43 UTC
Oh, I should have guessed ...

In that case, this is more or less the same issue as bug #832757: mysql cannot recreate the /var/lib/mysql directory, because that would require root privileges (since /var/lib is root-owned).

I know that you used to be able to get away with this back in the initscript days, because that script did run as root and could recreate the directory.  But doing things that way opens assorted security risks, so now there's no root-privileged component in the package.

Recommendation if you want to blow away the database and start fresh is to remove the contents of /var/lib/mysql, but not that directory itself.  Or recreate (and chown) it while you're still root.

Comment 6 Pavel Alexeev 2012-07-02 18:01:34 UTC
# chown mysql:mysql -R /var/lib/mysql

# systemctl start mysqld.service
Job failed. See system logs and 'systemctl status' for details.

# systemctl status mysqld.service
mysqld.service - MySQL database server
          Loaded: loaded (/lib/systemd/system/mysqld.service; enabled)
          Active: activating (start-pre) since Mon, 02 Jul 2012 22:00:22 +0400; 222ms ago
         Control: 6021 (sd(EXEC))
          CGroup: name=systemd:/system/mysqld.service

Log:

Jul  2 22:00:22 linux systemd[6021]: Failed at step NAMESPACE spawning /usr/libexec/mysqld-prepare-db-dir: No such file or directory
Jul  2 22:00:23 linux systemd[1]: mysqld.service: control process exited, code=exited status=226
Jul  2 22:00:23 linux systemd[1]: mysqld.service holdoff time over, scheduling restart.
Jul  2 22:00:23 linux systemd[1]: Job pending for unit, delaying automatic restart.
Jul  2 22:00:23 linux systemd[1]: Unit mysqld.service entered failed state.
Jul  2 22:00:24 linux systemd[6030]: Failed at step NAMESPACE spawning /usr/libexec/mysqld-prepare-db-dir: No such file or directory
Jul  2 22:00:26 linux systemd[1]: mysqld.service: control process exited, code=exited status=226
Jul  2 22:00:26 linux systemd[1]: mysqld.service holdoff time over, scheduling restart.
Jul  2 22:00:26 linux systemd[1]: Job pending for unit, delaying automatic restart.
Jul  2 22:00:26 linux systemd[1]: Unit mysqld.service entered failed state.
Jul  2 22:00:27 linux systemd[6036]: Failed at step NAMESPACE spawning /usr/libexec/mysqld-prepare-db-dir: No such file or directory
Jul  2 22:00:27 linux systemd[1]: mysqld.service: control process exited, code=exited status=226
Jul  2 22:00:27 linux systemd[1]: mysqld.service holdoff time over, scheduling restart.
Jul  2 22:00:27 linux systemd[1]: Job pending for unit, delaying automatic restart.
Jul  2 22:00:27 linux systemd[1]: Unit mysqld.service entered failed state.
Jul  2 22:00:28 linux systemd[6041]: Failed at step NAMESPACE spawning /usr/libexec/mysqld-prepare-db-dir: No such file or directory
Jul  2 22:00:43 linux systemd[1]: mysqld.service: control process exited, code=exited status=226
Jul  2 22:00:44 linux systemd[1]: mysqld.service holdoff time over, scheduling restart.
Jul  2 22:00:44 linux systemd[1]: Job pending for unit, delaying automatic restart.
Jul  2 22:00:44 linux systemd[1]: Unit mysqld.service entered failed state.
Jul  2 22:00:54 linux systemd[6046]: Failed at step NAMESPACE spawning /usr/libexec/mysqld-prepare-db-dir: No such file or directory
Jul  2 22:00:56 linux systemd[1]: mysqld.service: control process exited, code=exited status=226
Jul  2 22:00:56 linux systemd[1]: mysqld.service holdoff time over, scheduling restart.
Jul  2 22:00:56 linux systemd[1]: Job pending for unit, delaying automatic restart.
Jul  2 22:00:56 linux systemd[1]: Unit mysqld.service entered failed state.
Jul  2 22:00:58 linux systemd[6050]: Failed at step NAMESPACE spawning /usr/libexec/mysqld-prepare-db-dir: No such file or directory
Jul  2 22:01:02 linux systemd[1]: mysqld.service: control process exited, code=exited status=226
Jul  2 22:01:02 linux systemd[1]: mysqld.service holdoff time over, scheduling restart.
Jul  2 22:01:02 linux systemd[1]: Job pending for unit, delaying automatic restart.
Jul  2 22:01:02 linux systemd[1]: Unit mysqld.service entered failed state.
Jul  2 22:01:16 linux systemd[6053]: Failed at step NAMESPACE spawning /usr/libexec/mysqld-prepare-db-dir: No such file or directory

Comment 7 Steve Traylen 2012-07-02 18:47:41 UTC
I agree not a bug, just a don't do that.

Anyway its not myproxy :-)

Comment 8 Pavel Alexeev 2012-07-02 19:43:21 UTC
Strange. But 100% I had not change component... Off course it should be mysql.

Comment 9 Tom Lane 2012-07-02 20:07:26 UTC
(In reply to comment #6)
> Jul  2 22:00:22 linux systemd[6021]: Failed at step NAMESPACE spawning
> /usr/libexec/mysqld-prepare-db-dir: No such file or directory

This looks more like a systemd bug than a mysql bug.  Do you have an up-to-date version of systemd?

Comment 10 Pavel Alexeev 2012-07-13 21:41:29 UTC
I have updates systemd today, now it emits in log such messages very fast:


Jul 14 01:39:55 linux systemd[1]: mysqld.service holdoff time over, scheduling restart.
Jul 14 01:39:55 linux systemd[1]: Job pending for unit, delaying automatic restart.
Jul 14 01:39:55 linux systemd[1]: mysqld.service holdoff time over, scheduling restart.
Jul 14 01:39:55 linux systemd[1]: Job pending for unit, delaying automatic restart.
Jul 14 01:39:55 linux systemd[1]: mysqld.service holdoff time over, scheduling restart.
Jul 14 01:39:55 linux systemd[1]: Job pending for unit, delaying automatic restart.
Jul 14 01:39:55 linux systemd[1]: mysqld.service holdoff time over, scheduling restart.
Jul 14 01:39:55 linux systemd[1]: Job pending for unit, delaying automatic restart.
Jul 14 01:39:55 linux systemd[1]: mysqld.service holdoff time over, scheduling restart.
Jul 14 01:39:55 linux systemd[1]: Job pending for unit, delaying automatic restart.
Jul 14 01:39:55 linux systemd[1]: mysqld.service holdoff time over, scheduling restart.
Jul 14 01:39:55 linux systemd[1]: Job pending for unit, delaying automatic restart.
Jul 14 01:39:55 linux systemd[1]: mysqld.service holdoff time over, scheduling restart.
Jul 14 01:39:55 linux systemd[1]: Job pending for unit, delaying automatic restart.
Jul 14 01:39:56 linux systemd[1]: mysqld.service holdoff time over, scheduling restart.
Jul 14 01:39:56 linux systemd[1]: Job pending for unit, delaying automatic restart.
Jul 14 01:39:56 linux systemd[1]: mysqld.service holdoff time over, scheduling restart.
Jul 14 01:39:56 linux systemd[1]: Job pending for unit, delaying automatic restart.
Jul 14 01:39:56 linux systemd[1]: mysqld.service holdoff time over, scheduling restart.
Jul 14 01:39:56 linux systemd[1]: Job pending for unit, delaying automatic restart.
Jul 14 01:39:56 linux systemd[1]: mysqld.service holdoff time over, scheduling restart.
Jul 14 01:39:56 linux systemd[1]: Job pending for unit, delaying automatic restart.
Jul 14 01:39:56 linux systemd[1]: mysqld.service holdoff time over, scheduling restart.
Jul 14 01:39:56 linux systemd[1]: Job pending for unit, delaying automatic restart.
Jul 14 01:39:56 linux systemd[1]: mysqld.service holdoff time over, scheduling restart.
Jul 14 01:39:56 linux systemd[1]: Job pending for unit, delaying automatic restart.
Jul 14 01:39:56 linux systemd[1]: mysqld.service holdoff time over, scheduling restart.
Jul 14 01:39:56 linux systemd[1]: Job pending for unit, delaying automatic restart.
Jul 14 01:39:56 linux systemd[1]: mysqld.service holdoff time over, scheduling restart.
Jul 14 01:39:56 linux systemd[1]: Job pending for unit, delaying automatic restart.
Jul 14 01:39:56 linux systemd[1]: mysqld.service holdoff time over, scheduling restart.
Jul 14 01:39:56 linux systemd[1]: Job pending for unit, delaying automatic restart.
Jul 14 01:39:56 linux systemd[1]: mysqld.service holdoff time over, scheduling restart.
Jul 14 01:39:56 linux systemd[1]: Job pending for unit, delaying automatic restart.
Jul 14 01:39:57 linux systemd[1]: mysqld.service holdoff time over, scheduling restart.
Jul 14 01:39:57 linux systemd[1]: Job pending for unit, delaying automatic restart.
Jul 14 01:39:57 linux systemd[1]: mysqld.service holdoff time over, scheduling restart.
Jul 14 01:39:57 linux systemd[1]: Job pending for unit, delaying automatic restart.
Jul 14 01:39:57 linux systemd[1]: mysqld.service holdoff time over, scheduling restart.
Jul 14 01:39:57 linux systemd[1]: Job pending for unit, delaying automatic restart.
Jul 14 01:39:57 linux systemd[1]: mysqld.service holdoff time over, scheduling restart.
Jul 14 01:39:57 linux systemd[1]: Job pending for unit, delaying automatic restart.
Jul 14 01:39:57 linux systemd[1]: mysqld.service holdoff time over, scheduling restart.
Jul 14 01:39:57 linux systemd[1]: Job pending for unit, delaying automatic restart.


# rpm -q systemd
systemd-37-25.fc16.x86_64

Comment 11 Tom Lane 2012-07-13 22:03:48 UTC
There's a pending update for mysql that should prevent the respawn loop, but the underlying cause is presumably a misconfiguration of some sort, and you've not shown anything that would indicate what that is.  Look into mysqld.log, or if there's nothing obvious there, maybe the system log from just before the respawn loop starts.

Comment 12 Pavel Alexeev 2012-08-11 12:21:26 UTC
(In reply to comment #11)
> There's a pending update for mysql that should prevent the respawn loop, but
> the underlying cause is presumably a misconfiguration of some sort, and
> you've not shown anything that would indicate what that is.  Look into
> mysqld.log, or if there's nothing obvious there, maybe the system log from
> just before the respawn loop starts.
mysqld.log indicate nothing helpful for that. What info I can provide to resolve it?

Comment 13 Honza Horak 2012-08-14 15:25:58 UTC
Pavel, can you, please, provide output of mysql.log and content of my.cnf file?

Comment 14 Honza Horak 2012-08-21 11:33:56 UTC
With SELinux disabled I did:
# rm -rf /var/lib/mysql/
# mkdir /var/lib/mysql/
# chown mysql:mysql /var/lib/mysql/
# systemctl start mysqld.service
..and it worked fine.

Then I played with /etc/my.cnf a bit and this is what could happen:
# rm -rf /var/lib/mysql/
# mkdir /var/lib/mysql/
# chown mysql:mysql /var/lib/mysql/
..damage /etc/my.cnf somehow
# systemctl start mysqld.service 
It fails, because mysqld-prepare-db-dir fails. But empty directories /var/lib/mysql/mysql and /var/lib/mysql/test have been still created.

Then we fix /etc/my.cnf, and run mysqld-prepare-db-dir again (doesn't matter if directly or using systemctl start). mysqld-prepare-db-dir checks if /var/lib/mysql/mysql exists and finishes successfully even if /var/lib/mysql/mysql is empty, so no cluster is actually prepared any more.

Pavel, could this be your case?

Then we have to either run
/usr/bin/mysql_install_db --datadir="$datadir" --user=mysql
manually or remove /var/lib/mysql/* and run mysqld-prepare-db-dir again.

To prevent such issue we can add extra check to mysqld-prepare-db-dir and print error when initialization is not ready. The following patch doesn't change behaviour, but gives at least some tips about what's happening:

diff -up /usr/libexec/mysqld-prepare-db-dir.orig  /usr/libexec/mysqld-prepare-db-dir
--- /usr/libexec/mysqld-prepare-db-dir.orig	2012-08-21 13:04:36.640001342 +0200
+++ /usr/libexec/mysqld-prepare-db-dir	2012-08-21 13:22:41.876000060 +0200
@@ -47,8 +47,18 @@ if [ ! -d "$datadir/mysql" ] ; then
     ret=$?
     chown -R mysql:mysql "$datadir"
     if [ $ret -ne 0 ] ; then
+        echo "Initialization failed and won't be run automatically again. "\
+             "It could be caused by a misconfigured my.cnf, run "\
+             "/usr/bin/mysql_install_db --datadir='$datadir' --user=mysql "\
+             "manually in that case." >&2
         exit $ret
     fi
 fi
 
+if [ -d "$datadir/mysql" -a ! -e "$datadir/mysql/user.frm" ] ; then
+    echo "DB cluster is probably damaged. If you wish to initialize it again, "\
+         "backup the existing directory '$datadir', empty that directory "\
+         "and run $0 again." >&2
+fi
+
 exit 0

Comment 15 Tom Lane 2012-08-21 15:11:20 UTC
I wonder if we should actually clean out the datadir again after a failure.

Comment 16 Honza Horak 2012-08-23 14:50:53 UTC
Created attachment 606627 [details]
Patch for mysqld-prepare-db-dir that clean up $datadir if not initialized properly

Currently, if initialization fails (e.g. because of misconfigured my.cnf) $datadir/mysql has been already created. Even after the problem is fixed, initialization won't process again, because it is only proceeded when $datadir/mysql is missing.

This patch removes already created content of $datadir/*, so initialization can be run again after my.cnf is fixed (or whatever problem is fixed).

Comment 17 Pavel Alexeev 2012-08-28 12:54:37 UTC
Created attachment 607501 [details]
my.cnf

(In reply to comment #13)
> Pavel, can you, please, provide output of mysql.log and content of my.cnf
> file?

Off course. my.cnf is default (attached). /var/log/mysqld.log is empty if start via systemd.

Comment 18 Pavel Alexeev 2012-08-28 13:10:36 UTC
Thanks, at least mysql not respawning now infinity.

(In reply to comment #14)
> Pavel, could this be your case?
I'm unsure but it seems to be true.

> Then we have to either run
> /usr/bin/mysql_install_db --datadir="$datadir" --user=mysql
> manually

# /usr/bin/mysql_install_db --datadir="$datadir" --user=mysql
Installing MySQL system tables...
ERROR: 1049  Unknown database 'mysql'
120828 17:08:51 [ERROR] Aborting

120828 17:08:51 [Note] /usr/libexec/mysqld: Shutdown complete


Installation of system tables failed!  Examine the logs in
 for more information.

You can try to start the mysqld daemon with:

    shell> /usr/libexec/mysqld --skip-grant &

and use the command line tool /usr/bin/mysql
to connect to the mysql database and look at the grant tables:

    shell> /usr/bin/mysql -u root mysql
    mysql> show tables

Try 'mysqld --help' if you have problems with paths.  Using --log
gives you a log in  that may be helpful.

Please consult the MySQL manual section
'Problems running mysql_install_db', and the manual section that
describes problems on your OS.  Another information source are the
MySQL email archives available at http://lists.mysql.com/.

Please check all of the above before mailing us!  And remember, if
you do mail us, you MUST use the /usr/bin/mysqlbug script!

> or remove /var/lib/mysql/* and run mysqld-prepare-db-dir again.

# rm -rf /var/lib/mysql/
# mkdir /var/lib/mysql/
# chown mysql:mysql /var/lib/mysql/
# systemctl start mysqld.service

Job failed. See system logs and 'systemctl status' for details.

Comment 19 Pavel Alexeev 2012-08-28 13:13:29 UTC
Sorry, run from my previous post was wrong without variable initiated:

# /usr/bin/mysql_install_db --datadir="/var/lib/mysql" --user=mysql
Installing MySQL system tables...
OK
Filling help tables...
OK

To start mysqld at boot time you have to copy
support-files/mysql.server to the right place for your system

PLEASE REMEMBER TO SET A PASSWORD FOR THE MySQL root USER !
To do so, start the server, then issue the following commands:

/usr/bin/mysqladmin -u root password 'new-password'
/usr/bin/mysqladmin -u root -h linux password 'new-password'

Alternatively you can run:
/usr/bin/mysql_secure_installation

which will also give you the option of removing the test
databases and anonymous user created by default.  This is
strongly recommended for production servers.

See the manual for more instructions.

You can start the MySQL daemon with:
cd /usr ; /usr/bin/mysqld_safe &

You can test the MySQL daemon with mysql-test-run.pl
cd /usr/mysql-test ; perl mysql-test-run.pl

Please report any problems with the /usr/bin/mysqlbug script!




# systemctl start mysqld.service 
Job failed. See system logs and 'systemctl status' for details.

# systemctl status mysqld.service 
mysqld.service - MySQL database server
          Loaded: loaded (/lib/systemd/system/mysqld.service; enabled)
          Active: failed since Tue, 28 Aug 2012 17:11:43 +0400; 53s ago
         Process: 12886 ExecStartPre=/usr/libexec/mysqld-prepare-db-dir (code=exited, status=226/NAMESPACE)
          CGroup: name=systemd:/system/mysqld.service


/var/log/mysqld.log still empty.

In log:

Aug 28 17:11:43 linux systemd[12886]: Failed at step NAMESPACE spawning /usr/libexec/mysqld-prepare-db-dir: No such file or directory
Aug 28 17:11:43 linux systemd[1]: mysqld.service: control process exited, code=exited status=226
Aug 28 17:11:43 linux systemd[1]: Unit mysqld.service entered failed state.

Comment 20 Honza Horak 2012-08-28 13:36:41 UTC
Thanks Pavel, it really seems like a systemd bug. Can you, please, try the following two things?

1) check if /usr/libexec/mysqld-prepare-db-dir exists and user "mysql" can read/execute it

2) remove or comment "PrivateTmp=true" line in /lib/systemd/system/mysqld.service for a moment and see, if the problem vanishes (setting NAMESPACE can be somehow broken during spawning a new process)

Comment 21 Pavel Alexeev 2012-08-28 13:41:08 UTC
(In reply to comment #20)
> Thanks Pavel, it really seems like a systemd bug. Can you, please, try the
> following two things?
> 
> 1) check if /usr/libexec/mysqld-prepare-db-dir exists and user "mysql" can
> read/execute it

# ll /usr/libexec/mysqld-prepare-db-dir
-rwxr-xr-x 1 root root 1679 июля   7 04:50 /usr/libexec/mysqld-prepare-db-dir

# su mysql
bash-4.2$ /usr/libexec/mysqld-prepare-db-dir
bash-4.2$ 

> 2) remove or comment "PrivateTmp=true" line in
> /lib/systemd/system/mysqld.service for a moment and see, if the problem
> vanishes (setting NAMESPACE can be somehow broken during spawning a new
> process)
YES! That helps!

Comment 22 Honza Horak 2012-08-28 13:53:27 UTC
So it seems the problem is in systemd, re-assigning.

Comment 23 Pavel Alexeev 2012-08-28 13:55:37 UTC
Honza, Tom, thank you for the help.

Comment 24 Lukáš Nykrýn 2012-08-29 10:56:01 UTC
I am not sure what can couse this, but for start can you check privileges of /tmp directory?

Comment 25 Jóhann B. Guðmundsson 2012-08-29 20:42:45 UTC
(In reply to comment #24)
> I am not sure what can couse this, but for start can you check privileges of
> /tmp directory?

You should be able to recreate this with any type simple unit that runs as a different user ( and probably without as well ) with ExecStartPre and PrivateTmp set

As in I'm pretty sure something like this recreates this bug 

Description=Test Unit

[Service]
User=test
Group=test
# Script that creates directories and or just sleeps for few seconds 
ExecStartPre=/path/to/script 
ExecStart=/usr/bin/foo
PrivateTmp=true

Comment 26 Tom Lane 2012-08-30 04:57:23 UTC
(In reply to comment #25)
> You should be able to recreate this with any type simple unit that runs as a
> different user ( and probably without as well ) with ExecStartPre and
> PrivateTmp set

[ shrug... ]  No.  On a perfectly stock, up-to-date F17 setup, I do "yum install mysql-server" then "service mysqld start", and it starts just fine.  So there is something else required to show the problem.

Comment 27 Jóhann B. Guðmundsson 2012-08-30 08:42:41 UTC
(In reply to comment #26)
> (In reply to comment #25)
> > You should be able to recreate this with any type simple unit that runs as a
> > different user ( and probably without as well ) with ExecStartPre and
> > PrivateTmp set
> 
> [ shrug... ]  No.  On a perfectly stock, up-to-date F17 setup, I do "yum
> install mysql-server" then "service mysqld start", and it starts just fine. 
> So there is something else required to show the problem.

"Failed at step NAMESPACE spawning" is just one of those unhelpful messages in systemd since this is usually happens if /tmp or /var/tmp is a symlink wrong permission, non existing the usual stuff like that. 

First and foremost this was a test case ( for us systemd ) to start working with and try duplicating this with totally irrelevant to mysql or all the other daemon/service that can trigger this after /tmp or /var/tmp has been messed with or has broken in some weird way, 

Most likely the solution to this bug will be fixing that stupid unhelpful error msg for users to something they understand like "/tmp or /var/tmp is broken or setup in some unsupported way fix it!"

Comment 28 Jonathan Hoser 2012-09-18 14:29:43 UTC
 (In reply to comment #27)
> "Failed at step NAMESPACE spawning" is just one of those unhelpful messages
> in systemd since this is usually happens if /tmp or /var/tmp is a symlink
> wrong permission, non existing the usual stuff like that. 
> 
> First and foremost this was a test case ( for us systemd ) to start working
> with and try duplicating this with totally irrelevant to mysql or all the
> other daemon/service that can trigger this after /tmp or /var/tmp has been
> messed with or has broken in some weird way, 

Aha.

And *how* should /tmp and /var/tmp look like?

Some of us out there *would like* to keep a temporary directory off the / partition.

In my setup e.g.
I have /tmp as an own partition,
drwxrwxrwt. 110 root root 40960 Sep 18 16:22 tmp

with /var/tmp being
lrwxrwxrwx. 1 root root 4 Aug 31 11:08 /var/tmp -> /tmp

now should the rights for the symlink look like? And why shouldn't it be allowed to be a symlink?
I have been puzzling on that bug on and off for a bunch of weeks now....

Comment 29 Jonathan Hoser 2012-09-18 14:35:26 UTC
(In reply to comment #28)
>  (In reply to comment #27)
> > "Failed at step NAMESPACE spawning" is just one of those unhelpful messages
> > in systemd since this is usually happens if /tmp or /var/tmp is a symlink
> > wrong permission, non existing the usual stuff like that. 
> > 
> > First and foremost this was a test case ( for us systemd ) to start working
> > with and try duplicating this with totally irrelevant to mysql or all the
> > other daemon/service that can trigger this after /tmp or /var/tmp has been
> > messed with or has broken in some weird way, 
> 
> Aha.
> 
> And *how* should /tmp and /var/tmp look like?
> 
> Some of us out there *would like* to keep a temporary directory off the /
> partition.
> 
> In my setup e.g.
> I have /tmp as an own partition,
> drwxrwxrwt. 110 root root 40960 Sep 18 16:22 tmp
> 
> with /var/tmp being
> lrwxrwxrwx. 1 root root 4 Aug 31 11:08 /var/tmp -> /tmp
> 
> now should the rights for the symlink look like? And why shouldn't it be
> allowed to be a symlink?
> I have been puzzling on that bug on and off for a bunch of weeks now....

Ah, and yes, revamping /var/tmp to be a 'regular' directory solves the issue.

Comment 30 Lennart Poettering 2013-01-14 20:35:36 UTC
So, basically this but is about PrivateTmp= not working correctly if /var/tmp is a symlink? Did I get this right?

Comment 31 Sam Varshavchik 2013-01-15 05:04:54 UTC
Yes, but not always. My /var/tmp was a symlink since April 2011, and systemd was happy until the systemd-44-23.fc17.x86_64, when it broke.

In the strace, I can see the mount of /var/tmp, after unshare(CLONE_NEWNS); failing.

Comment 32 Jonathan Hoser 2013-01-15 07:05:59 UTC
+1 to the "Yes, but not always".
We had the /var/tmp symlink in operation for quite some time, up to the moment where the mysqld-stuff broke. Thought I never had the time to investigate (or even link it) the upgrade of systemd.

Comment 33 Lennart Poettering 2013-03-08 00:11:40 UTC
*** Bug 901926 has been marked as a duplicate of this bug. ***

Comment 34 Curtis Doty 2013-09-22 21:34:12 UTC
Reproducible on F19 with ntpd.service where /tmp -> /var/tmp.

From stracing systemd(1) it first sets up both PrivateTmp dirs:

umask(077)                              = 0
mkdir("/tmp/systemd-private-Lsfcny", 0700) = 0
umask(0)                                = 077
umask(0)                                = 0
mkdir("/tmp/systemd-private-Lsfcny/tmp", 0777) = 0
umask(0)                                = 0
chmod("/tmp/systemd-private-Lsfcny/tmp", 01777) = 0
umask(077)                              = 0
mkdir("/var/tmp/systemd-private-mhECrJ", 0700) = 0
umask(0)                                = 077
umask(0)                                = 0
mkdir("/var/tmp/systemd-private-mhECrJ/tmp", 0777) = 0
umask(0)                                = 0
chmod("/var/tmp/systemd-private-mhECrJ/tmp", 01777) = 0
clone(Process 1468 attached

Then things appear to go haywire when it tries to mount(MS_BIND) the second time:

[pid  1468] mount(NULL, "/", NULL, MS_REC|MS_SLAVE, NULL) = 0
[pid  1468] mount("/tmp/systemd-private-Lsfcny/tmp", "/tmp", NULL, MS_BIND|MS_REC, NULL) = 0
[pid  1468] mount("/var/tmp/systemd-private-mhECrJ/tmp", "/var/tmp", NULL, MS_BIND|MS_REC, NULL) = -1 ENOENT (No such file or directory)
[pid  1468] umount("/var/tmp", MNT_DETACH) = 0

In this example, the system is running btrfs. And / and /var are different mountpoints.

Comment 35 Curtis Doty 2013-09-22 21:36:34 UTC
Can't systemd be made smart enough to know when /tmp and /var/tmp are the same? And then, of course, only use a single systemd-private tempdir?

Comment 36 Curtis Doty 2013-09-22 22:40:29 UTC
(In reply to Curtis Doty from comment #34)
> In this example, the system is running btrfs. And / and /var are different
> mountpoints.

Whoops! I _meant_ otherwise. But they are actually both on / in this example reproduction.

Comment 37 Fedora End Of Life 2013-12-21 15:03:54 UTC
This message is a reminder that Fedora 18 is nearing its end of life.
Approximately 4 (four) weeks from now Fedora will stop maintaining
and issuing updates for Fedora 18. It is Fedora's policy to close all
bug reports from releases that are no longer maintained. At that time
this bug will be closed as WONTFIX if it remains open with a Fedora 
'version' of '18'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version prior to Fedora 18's end of life.

Thank you for reporting this issue and we are sorry that we may not be 
able to fix it before Fedora 18 is end of life. If you would still like 
to see this bug fixed and are able to reproduce it against a later version 
of Fedora, you are encouraged  change the 'version' to a later Fedora 
version prior to Fedora 18's end of life.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events. Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.

Comment 38 Jan Synacek 2015-03-03 10:09:04 UTC
I can't reproduce this issue from F20 onward. Can anyone confirm?


Note You need to log in before you can comment on or make changes to this bug.