Description of problem: Installation of MySQL which was early deleted. Delete /var/log/mysql directory. Try start: systemctl start mysqld.service Job failed. See system logs and 'systemctl status' for details. In system log: Jun 25 19:37:28 linux systemd[27615]: Failed at step NAMESPACE spawning /usr/libexec/mysqld-prepare-db-dir: No such file or directory Jun 25 19:37:28 linux systemd[1]: mysqld.service: control process exited, code=exited status=226 Jun 25 19:37:28 linux systemd[1]: mysqld.service holdoff time over, scheduling restart. Jun 25 19:37:28 linux systemd[1]: Job pending for unit, delaying automatic restart. Jun 25 19:37:28 linux systemd[1]: Unit mysqld.service entered failed state. Jun 25 19:37:28 linux systemd[27617]: Failed at step NAMESPACE spawning /usr/libexec/mysqld-prepare-db-dir: No such file or directory Jun 25 19:37:28 linux systemd[1]: mysqld.service: control process exited, code=exited status=226 Jun 25 19:37:29 linux systemd[1]: mysqld.service holdoff time over, scheduling restart. Jun 25 19:37:29 linux systemd[1]: Job pending for unit, delaying automatic restart. Jun 25 19:37:29 linux systemd[1]: Unit mysqld.service entered failed state. Jun 25 19:37:29 linux systemd[27619]: Failed at step NAMESPACE spawning /usr/libexec/mysqld-prepare-db-dir: No such file or directory Jun 25 19:37:29 linux systemd[1]: mysqld.service: control process exited, code=exited status=226 Jun 25 19:37:29 linux systemd[1]: mysqld.service holdoff time over, scheduling restart. Jun 25 19:37:29 linux systemd[1]: Job pending for unit, delaying automatic restart. Jun 25 19:37:29 linux systemd[1]: Unit mysqld.service entered failed state. Jun 25 19:37:29 linux systemd[27621]: Failed at step NAMESPACE spawning /usr/libexec/mysqld-prepare-db-dir: No such file or directory Jun 25 19:37:29 linux systemd[1]: mysqld.service: control process exited, code=exited status=226 Jun 25 19:37:29 linux systemd[1]: mysqld.service holdoff time over, scheduling restart. Jun 25 19:37:29 linux systemd[1]: Job pending for unit, delaying automatic restart. Jun 25 19:37:29 linux systemd[1]: Unit mysqld.service entered failed state. Jun 25 19:37:29 linux systemd[27623]: Failed at step NAMESPACE spawning /usr/libexec/mysqld-prepare-db-dir: No such file or directory Jun 25 19:37:29 linux systemd[1]: mysqld.service: control process exited, code=exited status=226 Jun 25 19:37:29 linux systemd[1]: mysqld.service holdoff time over, scheduling restart. Jun 25 19:37:29 linux systemd[1]: Job pending for unit, delaying automatic restart. Jun 25 19:37:29 linux systemd[1]: Unit mysqld.service entered failed state. Jun 25 19:37:29 linux systemd[27625]: Failed at step NAMESPACE spawning /usr/libexec/mysqld-prepare-db-dir: No such file or directory Jun 25 19:37:29 linux systemd[1]: mysqld.service: control process exited, code=exited status=226 Jun 25 19:37:29 linux systemd[1]: mysqld.service holdoff time over, scheduling restart. Jun 25 19:37:29 linux systemd[1]: Job pending for unit, delaying automatic restart. Jun 25 19:37:29 linux systemd[1]: Unit mysqld.service entered failed state. Jun 25 19:37:29 linux systemd[1]: mysqld.service start request repeated too quickly, refusing to start. SELinux disabled: # getenforce Disabled Manual setup directory successful but does nothing: # /usr/libexec/mysqld-prepare-db-dir Initializing MySQL database Installing MySQL system tables... OK Filling help tables... OK To start mysqld at boot time you have to copy support-files/mysql.server to the right place for your system PLEASE REMEMBER TO SET A PASSWORD FOR THE MySQL root USER ! To do so, start the server, then issue the following commands: /usr/bin/mysqladmin -u root password 'new-password' /usr/bin/mysqladmin -u root -h linux password 'new-password' Alternatively you can run: /usr/bin/mysql_secure_installation which will also give you the option of removing the test databases and anonymous user created by default. This is strongly recommended for production servers. See the manual for more instructions. You can start the MySQL daemon with: cd /usr ; /usr/bin/mysqld_safe & You can test the MySQL daemon with mysql-test-run.pl cd /usr/mysql-test ; perl mysql-test-run.pl Please report any problems with the /usr/bin/mysqlbug script! # systemctl start mysqld.service Job failed. See system logs and 'systemctl status' for details. In logs the same information only. Version-Release number of selected component (if applicable): # rpm -qa 'mysql*' mysql-server-5.5.23-1.fc16.x86_64 mysql-libs-5.5.23-1.fc16.x86_64 mysql-5.5.23-1.fc16.x86_64 mysql-utilities-1.0.5-2.fc16.noarch mysql-connector-python-0.3.2-2.fc16.noarch How reproducible: Always
I don't think this is so much a bug as a "don't do that". The log file is created on package install, and you really shouldn't be removing it by hand, any more than any other file installed by the package. The reason mysql was able to create the log file for itself in previous versions was that it was started as root. We don't do that anymore, and I think the gain in security is worth this small annoyance. There is a discussion going on in bug #832035, which unfortunately some overprotective soul has marked Red Hat internal, about whether there is any way to recreate the log file at service start without adding any security risk. So far no workable ideas though :-(
(In reply to comment #1) > I don't think this is so much a bug as a "don't do that". The log file is > created on package install, and you really shouldn't be removing it by hand, > any more than any other file installed by the package. About what file are you speaking? I had not delete any MySQL log files. /var/log/mysql was removed (renamed) only after MySQL wasn't be able start with it. And it does not helps also.
Well, I assumed that > Delete /var/log/mysql directory. was a typo for /var/log/mysqld.log, because the mysql package doesn't create any directory named that and wouldn't care if someone else did. Maybe you meant something else? Or did you change some configuration settings that you haven't mentioned?
I'm very sorry. I mean /var/lib/mysql directory.
Oh, I should have guessed ... In that case, this is more or less the same issue as bug #832757: mysql cannot recreate the /var/lib/mysql directory, because that would require root privileges (since /var/lib is root-owned). I know that you used to be able to get away with this back in the initscript days, because that script did run as root and could recreate the directory. But doing things that way opens assorted security risks, so now there's no root-privileged component in the package. Recommendation if you want to blow away the database and start fresh is to remove the contents of /var/lib/mysql, but not that directory itself. Or recreate (and chown) it while you're still root.
# chown mysql:mysql -R /var/lib/mysql # systemctl start mysqld.service Job failed. See system logs and 'systemctl status' for details. # systemctl status mysqld.service mysqld.service - MySQL database server Loaded: loaded (/lib/systemd/system/mysqld.service; enabled) Active: activating (start-pre) since Mon, 02 Jul 2012 22:00:22 +0400; 222ms ago Control: 6021 (sd(EXEC)) CGroup: name=systemd:/system/mysqld.service Log: Jul 2 22:00:22 linux systemd[6021]: Failed at step NAMESPACE spawning /usr/libexec/mysqld-prepare-db-dir: No such file or directory Jul 2 22:00:23 linux systemd[1]: mysqld.service: control process exited, code=exited status=226 Jul 2 22:00:23 linux systemd[1]: mysqld.service holdoff time over, scheduling restart. Jul 2 22:00:23 linux systemd[1]: Job pending for unit, delaying automatic restart. Jul 2 22:00:23 linux systemd[1]: Unit mysqld.service entered failed state. Jul 2 22:00:24 linux systemd[6030]: Failed at step NAMESPACE spawning /usr/libexec/mysqld-prepare-db-dir: No such file or directory Jul 2 22:00:26 linux systemd[1]: mysqld.service: control process exited, code=exited status=226 Jul 2 22:00:26 linux systemd[1]: mysqld.service holdoff time over, scheduling restart. Jul 2 22:00:26 linux systemd[1]: Job pending for unit, delaying automatic restart. Jul 2 22:00:26 linux systemd[1]: Unit mysqld.service entered failed state. Jul 2 22:00:27 linux systemd[6036]: Failed at step NAMESPACE spawning /usr/libexec/mysqld-prepare-db-dir: No such file or directory Jul 2 22:00:27 linux systemd[1]: mysqld.service: control process exited, code=exited status=226 Jul 2 22:00:27 linux systemd[1]: mysqld.service holdoff time over, scheduling restart. Jul 2 22:00:27 linux systemd[1]: Job pending for unit, delaying automatic restart. Jul 2 22:00:27 linux systemd[1]: Unit mysqld.service entered failed state. Jul 2 22:00:28 linux systemd[6041]: Failed at step NAMESPACE spawning /usr/libexec/mysqld-prepare-db-dir: No such file or directory Jul 2 22:00:43 linux systemd[1]: mysqld.service: control process exited, code=exited status=226 Jul 2 22:00:44 linux systemd[1]: mysqld.service holdoff time over, scheduling restart. Jul 2 22:00:44 linux systemd[1]: Job pending for unit, delaying automatic restart. Jul 2 22:00:44 linux systemd[1]: Unit mysqld.service entered failed state. Jul 2 22:00:54 linux systemd[6046]: Failed at step NAMESPACE spawning /usr/libexec/mysqld-prepare-db-dir: No such file or directory Jul 2 22:00:56 linux systemd[1]: mysqld.service: control process exited, code=exited status=226 Jul 2 22:00:56 linux systemd[1]: mysqld.service holdoff time over, scheduling restart. Jul 2 22:00:56 linux systemd[1]: Job pending for unit, delaying automatic restart. Jul 2 22:00:56 linux systemd[1]: Unit mysqld.service entered failed state. Jul 2 22:00:58 linux systemd[6050]: Failed at step NAMESPACE spawning /usr/libexec/mysqld-prepare-db-dir: No such file or directory Jul 2 22:01:02 linux systemd[1]: mysqld.service: control process exited, code=exited status=226 Jul 2 22:01:02 linux systemd[1]: mysqld.service holdoff time over, scheduling restart. Jul 2 22:01:02 linux systemd[1]: Job pending for unit, delaying automatic restart. Jul 2 22:01:02 linux systemd[1]: Unit mysqld.service entered failed state. Jul 2 22:01:16 linux systemd[6053]: Failed at step NAMESPACE spawning /usr/libexec/mysqld-prepare-db-dir: No such file or directory
I agree not a bug, just a don't do that. Anyway its not myproxy :-)
Strange. But 100% I had not change component... Off course it should be mysql.
(In reply to comment #6) > Jul 2 22:00:22 linux systemd[6021]: Failed at step NAMESPACE spawning > /usr/libexec/mysqld-prepare-db-dir: No such file or directory This looks more like a systemd bug than a mysql bug. Do you have an up-to-date version of systemd?
I have updates systemd today, now it emits in log such messages very fast: Jul 14 01:39:55 linux systemd[1]: mysqld.service holdoff time over, scheduling restart. Jul 14 01:39:55 linux systemd[1]: Job pending for unit, delaying automatic restart. Jul 14 01:39:55 linux systemd[1]: mysqld.service holdoff time over, scheduling restart. Jul 14 01:39:55 linux systemd[1]: Job pending for unit, delaying automatic restart. Jul 14 01:39:55 linux systemd[1]: mysqld.service holdoff time over, scheduling restart. Jul 14 01:39:55 linux systemd[1]: Job pending for unit, delaying automatic restart. Jul 14 01:39:55 linux systemd[1]: mysqld.service holdoff time over, scheduling restart. Jul 14 01:39:55 linux systemd[1]: Job pending for unit, delaying automatic restart. Jul 14 01:39:55 linux systemd[1]: mysqld.service holdoff time over, scheduling restart. Jul 14 01:39:55 linux systemd[1]: Job pending for unit, delaying automatic restart. Jul 14 01:39:55 linux systemd[1]: mysqld.service holdoff time over, scheduling restart. Jul 14 01:39:55 linux systemd[1]: Job pending for unit, delaying automatic restart. Jul 14 01:39:55 linux systemd[1]: mysqld.service holdoff time over, scheduling restart. Jul 14 01:39:55 linux systemd[1]: Job pending for unit, delaying automatic restart. Jul 14 01:39:56 linux systemd[1]: mysqld.service holdoff time over, scheduling restart. Jul 14 01:39:56 linux systemd[1]: Job pending for unit, delaying automatic restart. Jul 14 01:39:56 linux systemd[1]: mysqld.service holdoff time over, scheduling restart. Jul 14 01:39:56 linux systemd[1]: Job pending for unit, delaying automatic restart. Jul 14 01:39:56 linux systemd[1]: mysqld.service holdoff time over, scheduling restart. Jul 14 01:39:56 linux systemd[1]: Job pending for unit, delaying automatic restart. Jul 14 01:39:56 linux systemd[1]: mysqld.service holdoff time over, scheduling restart. Jul 14 01:39:56 linux systemd[1]: Job pending for unit, delaying automatic restart. Jul 14 01:39:56 linux systemd[1]: mysqld.service holdoff time over, scheduling restart. Jul 14 01:39:56 linux systemd[1]: Job pending for unit, delaying automatic restart. Jul 14 01:39:56 linux systemd[1]: mysqld.service holdoff time over, scheduling restart. Jul 14 01:39:56 linux systemd[1]: Job pending for unit, delaying automatic restart. Jul 14 01:39:56 linux systemd[1]: mysqld.service holdoff time over, scheduling restart. Jul 14 01:39:56 linux systemd[1]: Job pending for unit, delaying automatic restart. Jul 14 01:39:56 linux systemd[1]: mysqld.service holdoff time over, scheduling restart. Jul 14 01:39:56 linux systemd[1]: Job pending for unit, delaying automatic restart. Jul 14 01:39:56 linux systemd[1]: mysqld.service holdoff time over, scheduling restart. Jul 14 01:39:56 linux systemd[1]: Job pending for unit, delaying automatic restart. Jul 14 01:39:56 linux systemd[1]: mysqld.service holdoff time over, scheduling restart. Jul 14 01:39:56 linux systemd[1]: Job pending for unit, delaying automatic restart. Jul 14 01:39:57 linux systemd[1]: mysqld.service holdoff time over, scheduling restart. Jul 14 01:39:57 linux systemd[1]: Job pending for unit, delaying automatic restart. Jul 14 01:39:57 linux systemd[1]: mysqld.service holdoff time over, scheduling restart. Jul 14 01:39:57 linux systemd[1]: Job pending for unit, delaying automatic restart. Jul 14 01:39:57 linux systemd[1]: mysqld.service holdoff time over, scheduling restart. Jul 14 01:39:57 linux systemd[1]: Job pending for unit, delaying automatic restart. Jul 14 01:39:57 linux systemd[1]: mysqld.service holdoff time over, scheduling restart. Jul 14 01:39:57 linux systemd[1]: Job pending for unit, delaying automatic restart. Jul 14 01:39:57 linux systemd[1]: mysqld.service holdoff time over, scheduling restart. Jul 14 01:39:57 linux systemd[1]: Job pending for unit, delaying automatic restart. # rpm -q systemd systemd-37-25.fc16.x86_64
There's a pending update for mysql that should prevent the respawn loop, but the underlying cause is presumably a misconfiguration of some sort, and you've not shown anything that would indicate what that is. Look into mysqld.log, or if there's nothing obvious there, maybe the system log from just before the respawn loop starts.
(In reply to comment #11) > There's a pending update for mysql that should prevent the respawn loop, but > the underlying cause is presumably a misconfiguration of some sort, and > you've not shown anything that would indicate what that is. Look into > mysqld.log, or if there's nothing obvious there, maybe the system log from > just before the respawn loop starts. mysqld.log indicate nothing helpful for that. What info I can provide to resolve it?
Pavel, can you, please, provide output of mysql.log and content of my.cnf file?
With SELinux disabled I did: # rm -rf /var/lib/mysql/ # mkdir /var/lib/mysql/ # chown mysql:mysql /var/lib/mysql/ # systemctl start mysqld.service ..and it worked fine. Then I played with /etc/my.cnf a bit and this is what could happen: # rm -rf /var/lib/mysql/ # mkdir /var/lib/mysql/ # chown mysql:mysql /var/lib/mysql/ ..damage /etc/my.cnf somehow # systemctl start mysqld.service It fails, because mysqld-prepare-db-dir fails. But empty directories /var/lib/mysql/mysql and /var/lib/mysql/test have been still created. Then we fix /etc/my.cnf, and run mysqld-prepare-db-dir again (doesn't matter if directly or using systemctl start). mysqld-prepare-db-dir checks if /var/lib/mysql/mysql exists and finishes successfully even if /var/lib/mysql/mysql is empty, so no cluster is actually prepared any more. Pavel, could this be your case? Then we have to either run /usr/bin/mysql_install_db --datadir="$datadir" --user=mysql manually or remove /var/lib/mysql/* and run mysqld-prepare-db-dir again. To prevent such issue we can add extra check to mysqld-prepare-db-dir and print error when initialization is not ready. The following patch doesn't change behaviour, but gives at least some tips about what's happening: diff -up /usr/libexec/mysqld-prepare-db-dir.orig /usr/libexec/mysqld-prepare-db-dir --- /usr/libexec/mysqld-prepare-db-dir.orig 2012-08-21 13:04:36.640001342 +0200 +++ /usr/libexec/mysqld-prepare-db-dir 2012-08-21 13:22:41.876000060 +0200 @@ -47,8 +47,18 @@ if [ ! -d "$datadir/mysql" ] ; then ret=$? chown -R mysql:mysql "$datadir" if [ $ret -ne 0 ] ; then + echo "Initialization failed and won't be run automatically again. "\ + "It could be caused by a misconfigured my.cnf, run "\ + "/usr/bin/mysql_install_db --datadir='$datadir' --user=mysql "\ + "manually in that case." >&2 exit $ret fi fi +if [ -d "$datadir/mysql" -a ! -e "$datadir/mysql/user.frm" ] ; then + echo "DB cluster is probably damaged. If you wish to initialize it again, "\ + "backup the existing directory '$datadir', empty that directory "\ + "and run $0 again." >&2 +fi + exit 0
I wonder if we should actually clean out the datadir again after a failure.
Created attachment 606627 [details] Patch for mysqld-prepare-db-dir that clean up $datadir if not initialized properly Currently, if initialization fails (e.g. because of misconfigured my.cnf) $datadir/mysql has been already created. Even after the problem is fixed, initialization won't process again, because it is only proceeded when $datadir/mysql is missing. This patch removes already created content of $datadir/*, so initialization can be run again after my.cnf is fixed (or whatever problem is fixed).
Created attachment 607501 [details] my.cnf (In reply to comment #13) > Pavel, can you, please, provide output of mysql.log and content of my.cnf > file? Off course. my.cnf is default (attached). /var/log/mysqld.log is empty if start via systemd.
Thanks, at least mysql not respawning now infinity. (In reply to comment #14) > Pavel, could this be your case? I'm unsure but it seems to be true. > Then we have to either run > /usr/bin/mysql_install_db --datadir="$datadir" --user=mysql > manually # /usr/bin/mysql_install_db --datadir="$datadir" --user=mysql Installing MySQL system tables... ERROR: 1049 Unknown database 'mysql' 120828 17:08:51 [ERROR] Aborting 120828 17:08:51 [Note] /usr/libexec/mysqld: Shutdown complete Installation of system tables failed! Examine the logs in for more information. You can try to start the mysqld daemon with: shell> /usr/libexec/mysqld --skip-grant & and use the command line tool /usr/bin/mysql to connect to the mysql database and look at the grant tables: shell> /usr/bin/mysql -u root mysql mysql> show tables Try 'mysqld --help' if you have problems with paths. Using --log gives you a log in that may be helpful. Please consult the MySQL manual section 'Problems running mysql_install_db', and the manual section that describes problems on your OS. Another information source are the MySQL email archives available at http://lists.mysql.com/. Please check all of the above before mailing us! And remember, if you do mail us, you MUST use the /usr/bin/mysqlbug script! > or remove /var/lib/mysql/* and run mysqld-prepare-db-dir again. # rm -rf /var/lib/mysql/ # mkdir /var/lib/mysql/ # chown mysql:mysql /var/lib/mysql/ # systemctl start mysqld.service Job failed. See system logs and 'systemctl status' for details.
Sorry, run from my previous post was wrong without variable initiated: # /usr/bin/mysql_install_db --datadir="/var/lib/mysql" --user=mysql Installing MySQL system tables... OK Filling help tables... OK To start mysqld at boot time you have to copy support-files/mysql.server to the right place for your system PLEASE REMEMBER TO SET A PASSWORD FOR THE MySQL root USER ! To do so, start the server, then issue the following commands: /usr/bin/mysqladmin -u root password 'new-password' /usr/bin/mysqladmin -u root -h linux password 'new-password' Alternatively you can run: /usr/bin/mysql_secure_installation which will also give you the option of removing the test databases and anonymous user created by default. This is strongly recommended for production servers. See the manual for more instructions. You can start the MySQL daemon with: cd /usr ; /usr/bin/mysqld_safe & You can test the MySQL daemon with mysql-test-run.pl cd /usr/mysql-test ; perl mysql-test-run.pl Please report any problems with the /usr/bin/mysqlbug script! # systemctl start mysqld.service Job failed. See system logs and 'systemctl status' for details. # systemctl status mysqld.service mysqld.service - MySQL database server Loaded: loaded (/lib/systemd/system/mysqld.service; enabled) Active: failed since Tue, 28 Aug 2012 17:11:43 +0400; 53s ago Process: 12886 ExecStartPre=/usr/libexec/mysqld-prepare-db-dir (code=exited, status=226/NAMESPACE) CGroup: name=systemd:/system/mysqld.service /var/log/mysqld.log still empty. In log: Aug 28 17:11:43 linux systemd[12886]: Failed at step NAMESPACE spawning /usr/libexec/mysqld-prepare-db-dir: No such file or directory Aug 28 17:11:43 linux systemd[1]: mysqld.service: control process exited, code=exited status=226 Aug 28 17:11:43 linux systemd[1]: Unit mysqld.service entered failed state.
Thanks Pavel, it really seems like a systemd bug. Can you, please, try the following two things? 1) check if /usr/libexec/mysqld-prepare-db-dir exists and user "mysql" can read/execute it 2) remove or comment "PrivateTmp=true" line in /lib/systemd/system/mysqld.service for a moment and see, if the problem vanishes (setting NAMESPACE can be somehow broken during spawning a new process)
(In reply to comment #20) > Thanks Pavel, it really seems like a systemd bug. Can you, please, try the > following two things? > > 1) check if /usr/libexec/mysqld-prepare-db-dir exists and user "mysql" can > read/execute it # ll /usr/libexec/mysqld-prepare-db-dir -rwxr-xr-x 1 root root 1679 июля 7 04:50 /usr/libexec/mysqld-prepare-db-dir # su mysql bash-4.2$ /usr/libexec/mysqld-prepare-db-dir bash-4.2$ > 2) remove or comment "PrivateTmp=true" line in > /lib/systemd/system/mysqld.service for a moment and see, if the problem > vanishes (setting NAMESPACE can be somehow broken during spawning a new > process) YES! That helps!
So it seems the problem is in systemd, re-assigning.
Honza, Tom, thank you for the help.
I am not sure what can couse this, but for start can you check privileges of /tmp directory?
(In reply to comment #24) > I am not sure what can couse this, but for start can you check privileges of > /tmp directory? You should be able to recreate this with any type simple unit that runs as a different user ( and probably without as well ) with ExecStartPre and PrivateTmp set As in I'm pretty sure something like this recreates this bug Description=Test Unit [Service] User=test Group=test # Script that creates directories and or just sleeps for few seconds ExecStartPre=/path/to/script ExecStart=/usr/bin/foo PrivateTmp=true
(In reply to comment #25) > You should be able to recreate this with any type simple unit that runs as a > different user ( and probably without as well ) with ExecStartPre and > PrivateTmp set [ shrug... ] No. On a perfectly stock, up-to-date F17 setup, I do "yum install mysql-server" then "service mysqld start", and it starts just fine. So there is something else required to show the problem.
(In reply to comment #26) > (In reply to comment #25) > > You should be able to recreate this with any type simple unit that runs as a > > different user ( and probably without as well ) with ExecStartPre and > > PrivateTmp set > > [ shrug... ] No. On a perfectly stock, up-to-date F17 setup, I do "yum > install mysql-server" then "service mysqld start", and it starts just fine. > So there is something else required to show the problem. "Failed at step NAMESPACE spawning" is just one of those unhelpful messages in systemd since this is usually happens if /tmp or /var/tmp is a symlink wrong permission, non existing the usual stuff like that. First and foremost this was a test case ( for us systemd ) to start working with and try duplicating this with totally irrelevant to mysql or all the other daemon/service that can trigger this after /tmp or /var/tmp has been messed with or has broken in some weird way, Most likely the solution to this bug will be fixing that stupid unhelpful error msg for users to something they understand like "/tmp or /var/tmp is broken or setup in some unsupported way fix it!"
(In reply to comment #27) > "Failed at step NAMESPACE spawning" is just one of those unhelpful messages > in systemd since this is usually happens if /tmp or /var/tmp is a symlink > wrong permission, non existing the usual stuff like that. > > First and foremost this was a test case ( for us systemd ) to start working > with and try duplicating this with totally irrelevant to mysql or all the > other daemon/service that can trigger this after /tmp or /var/tmp has been > messed with or has broken in some weird way, Aha. And *how* should /tmp and /var/tmp look like? Some of us out there *would like* to keep a temporary directory off the / partition. In my setup e.g. I have /tmp as an own partition, drwxrwxrwt. 110 root root 40960 Sep 18 16:22 tmp with /var/tmp being lrwxrwxrwx. 1 root root 4 Aug 31 11:08 /var/tmp -> /tmp now should the rights for the symlink look like? And why shouldn't it be allowed to be a symlink? I have been puzzling on that bug on and off for a bunch of weeks now....
(In reply to comment #28) > (In reply to comment #27) > > "Failed at step NAMESPACE spawning" is just one of those unhelpful messages > > in systemd since this is usually happens if /tmp or /var/tmp is a symlink > > wrong permission, non existing the usual stuff like that. > > > > First and foremost this was a test case ( for us systemd ) to start working > > with and try duplicating this with totally irrelevant to mysql or all the > > other daemon/service that can trigger this after /tmp or /var/tmp has been > > messed with or has broken in some weird way, > > Aha. > > And *how* should /tmp and /var/tmp look like? > > Some of us out there *would like* to keep a temporary directory off the / > partition. > > In my setup e.g. > I have /tmp as an own partition, > drwxrwxrwt. 110 root root 40960 Sep 18 16:22 tmp > > with /var/tmp being > lrwxrwxrwx. 1 root root 4 Aug 31 11:08 /var/tmp -> /tmp > > now should the rights for the symlink look like? And why shouldn't it be > allowed to be a symlink? > I have been puzzling on that bug on and off for a bunch of weeks now.... Ah, and yes, revamping /var/tmp to be a 'regular' directory solves the issue.
So, basically this but is about PrivateTmp= not working correctly if /var/tmp is a symlink? Did I get this right?
Yes, but not always. My /var/tmp was a symlink since April 2011, and systemd was happy until the systemd-44-23.fc17.x86_64, when it broke. In the strace, I can see the mount of /var/tmp, after unshare(CLONE_NEWNS); failing.
+1 to the "Yes, but not always". We had the /var/tmp symlink in operation for quite some time, up to the moment where the mysqld-stuff broke. Thought I never had the time to investigate (or even link it) the upgrade of systemd.
*** Bug 901926 has been marked as a duplicate of this bug. ***
Reproducible on F19 with ntpd.service where /tmp -> /var/tmp. From stracing systemd(1) it first sets up both PrivateTmp dirs: umask(077) = 0 mkdir("/tmp/systemd-private-Lsfcny", 0700) = 0 umask(0) = 077 umask(0) = 0 mkdir("/tmp/systemd-private-Lsfcny/tmp", 0777) = 0 umask(0) = 0 chmod("/tmp/systemd-private-Lsfcny/tmp", 01777) = 0 umask(077) = 0 mkdir("/var/tmp/systemd-private-mhECrJ", 0700) = 0 umask(0) = 077 umask(0) = 0 mkdir("/var/tmp/systemd-private-mhECrJ/tmp", 0777) = 0 umask(0) = 0 chmod("/var/tmp/systemd-private-mhECrJ/tmp", 01777) = 0 clone(Process 1468 attached Then things appear to go haywire when it tries to mount(MS_BIND) the second time: [pid 1468] mount(NULL, "/", NULL, MS_REC|MS_SLAVE, NULL) = 0 [pid 1468] mount("/tmp/systemd-private-Lsfcny/tmp", "/tmp", NULL, MS_BIND|MS_REC, NULL) = 0 [pid 1468] mount("/var/tmp/systemd-private-mhECrJ/tmp", "/var/tmp", NULL, MS_BIND|MS_REC, NULL) = -1 ENOENT (No such file or directory) [pid 1468] umount("/var/tmp", MNT_DETACH) = 0 In this example, the system is running btrfs. And / and /var are different mountpoints.
Can't systemd be made smart enough to know when /tmp and /var/tmp are the same? And then, of course, only use a single systemd-private tempdir?
(In reply to Curtis Doty from comment #34) > In this example, the system is running btrfs. And / and /var are different > mountpoints. Whoops! I _meant_ otherwise. But they are actually both on / in this example reproduction.
This message is a reminder that Fedora 18 is nearing its end of life. Approximately 4 (four) weeks from now Fedora will stop maintaining and issuing updates for Fedora 18. It is Fedora's policy to close all bug reports from releases that are no longer maintained. At that time this bug will be closed as WONTFIX if it remains open with a Fedora 'version' of '18'. Package Maintainer: If you wish for this bug to remain open because you plan to fix it in a currently maintained version, simply change the 'version' to a later Fedora version prior to Fedora 18's end of life. Thank you for reporting this issue and we are sorry that we may not be able to fix it before Fedora 18 is end of life. If you would still like to see this bug fixed and are able to reproduce it against a later version of Fedora, you are encouraged change the 'version' to a later Fedora version prior to Fedora 18's end of life. Although we aim to fix as many bugs as possible during every release's lifetime, sometimes those efforts are overtaken by events. Often a more recent Fedora release includes newer upstream software that fixes bugs or makes them obsolete.
I can't reproduce this issue from F20 onward. Can anyone confirm?