Bug 730209

I have not seen this happen myself, but if it does happen it's a bug in the tmpfiles support, which makes it systemd's problem.  mysql-server contains an
/etc/tmpfiles.d/mysql.conf file containing

d /var/run/mysqld 0755 mysql mysql -

so if that directory isn't already there when the initscript is run, it's not mysql's fault.  I wonder whether there is a race condition between tmpfiles setup and everything else (or at least sysv-compatible-initscript startups)

Yes, well, my configuration is running with NIS enabled, a static ip address, and for a time I had to restart cron (because it didn't know the users in NIS yet) and sendmail (I don't know why it wasn't started) on a reboot. These problems seem to have been solved by updates. The mysql user is also in NIS, btw, but the problem with the /var/run/mysqld is also present when the system is fully up and running.

(In reply to comment #2)
> Yes, well, my configuration is running with NIS enabled.
> The mysql user is also in NIS,

All system uids/gids should be resolvable locally. It's too hard to get
things like that working, and we do not really support that, not in
systemd, not in udev. If services need specific uid/gid mappings, they
better do not rely on network/server connectivity to get started.

NIS should be used for 'real' users only, the ones that log into the
machine, not for uids that just exist to separate services from each
other for security reasons, but which never log into the machine.

Nss can be configured to consult the local user database before NIS is
queried. The system users should be in the machine's local database.

So you're thinking that the tmpfiles code fails to resolve the uid/gid values for "mysql" (because it runs long before NIS will work), and then doesn't execute the tmpfiles item at all?  That would match the symptoms, apparently, but would there be a log entry or any other trace to confirm that?

I see this in /var/log/messages:
Aug  9 09:01:38 pc67340132 systemd[1]: mysqld.service: control process exited, code=exited status=1
Aug  9 09:01:38 pc67340132 systemd[1]: Unit mysqld.service entered failed state.
Aug  9 09:01:41 pc67340132 systemd[1]: Startup finished in 3s 984ms 94us (kernel) + 21s 29ms 488us (initrd) + 30s 913ms 637us (userspace) = 55s 927ms 219us.

No other mention of mysqld in the messages.

About the resolving of system uid/gids; I'll try monday to put the mysql entry in the local password file and see if that solves the problem.

In my opinion, what is and what isn't a system uid/gid and what we put in NIS is mostly for us to decide, because we have to glue an heterogeneous environment together. Certainly, for the system to boot and start basic services like network and NIS, everything should be local. But `higher level' services that in our environment depend on network, NIS, NFS and so on anyway should be waiting on the, in this case, nss we configured. In a way, mysqld is just another application that just happens to be started by the system automatically.
If systemd cannot create a consistent dependency graph of what services depend on each other, including tmpfiles, it's a shortcoming in my opinion. In this example, that systemd `knows' that tmpfiles will only depend on the local user database could be improved upon.
By the way, in the old sysvinit system, mysqld was by default started _after_ network, nis, etc were started so we never had this problem.

(In reply to comment #5)
> If systemd cannot create a consistent dependency graph of what services depend
> on each other, including tmpfiles, it's a shortcoming in my opinion. In this
> example, that systemd `knows' that tmpfiles will only depend on the local user
> database could be improved upon.
> By the way, in the old sysvinit system, mysqld was by default started _after_
> network, nis, etc were started so we never had this problem.

Yeah, it's up the service to decide that. If mysql needs to install a tmpfiles
file instead of taking care of its directories on its own at startup, mysql
puts the limitations on the system.

Systemd needs to run the tmpfiles early in the boot process, before services
are started to work around their expectations. Ideally services would just
fulfil such requirements on their own at service startup, instead of needing
a tmpfiles workaround. There is not much systemd can do about this.

For robustness reasons, it's in general not a good idea to have any textual
uids/gids configured for local system services that are started at system
bootup. These uids/gids should be always resolvable on the local system, they
can not be safely changed anyway, because many times, the files on the
filessystem carry them.

(In reply to comment #6)
> Systemd needs to run the tmpfiles early in the boot process, before services
> are started to work around their expectations. Ideally services would just
> fulfil such requirements on their own at service startup, instead of needing
> a tmpfiles workaround. There is not much systemd can do about this.

If that were the consensus, nobody would have bothered to invent the tmpfiles mechanism at all: packagers would just have been told "it's your responsibility to reconstruct /var/run/ after reboot".  I'm already unhappy that mysql had to do anything at all to work around that ill-considered hack, and I'm certainly not going to hold still for being told that I shouldn't use tmpfiles.

> For robustness reasons, it's in general not a good idea to have any textual
> uids/gids configured for local system services that are started at system
> bootup. These uids/gids should be always resolvable on the local system, they
> can not be safely changed anyway, because many times, the files on the
> filessystem carry them.

Yes, this is quite a good point: you put the whole mysql database at some risk if the uid/gid that owns the database files can't be resolved reliably (which means locally).

(In reply to comment #7)
> If that were the consensus, nobody would have bothered to invent the tmpfiles
> mechanism at all: packagers would just have been told "it's your responsibility
> to reconstruct /var/run/ after reboot".  I'm already unhappy that mysql had to
> do anything at all to work around that ill-considered hack, and I'm certainly
> not going to hold still for being told that I shouldn't use tmpfiles.

Ideally, the services would do that on their own, instead of expecting
pre-created content in a volatile directory. Services which are started
with root privileges should just create the needed stuff on their own,
from within the daemon,  with every service startup, and not rely on
any pre-existing content in runtime directories.

Tmpfiles is a very simple mechanism offered by systemd, to work around
services which need help here, meeting their historic expectations.

Tmpfiles main use-case is services which are not started as a daemon,
but by an unprivileged user. Some of them need directories with specific
permissions pre-created, to get access to them, which they wouldn't have
otherwise, and which they are not allowed to setup from the untrusted user
context.

All that works fine, also for mysql, I think, as long as system uids/gids
are resolvable at bootup, which they should be in all setups.

> > For robustness reasons, it's in general not a good idea to have any textual
> > uids/gids configured for local system services that are started at system
> > bootup. These uids/gids should be always resolvable on the local system, they
> > can not be safely changed anyway, because many times, the files on the
> > filessystem carry them.
> 
> Yes, this is quite a good point: you put the whole mysql database at some risk
> if the uid/gid that owns the database files can't be resolved reliably (which
> means locally).

Yeah, it's a very fragile setup otherwise. We don't recommend anybody doing
that.

(In reply to comment #8)
> (In reply to comment #7)
> > If that were the consensus, nobody would have bothered to invent the tmpfiles
> > mechanism at all: packagers would just have been told "it's your responsibility
> > to reconstruct /var/run/ after reboot".  I'm already unhappy that mysql had to
> > do anything at all to work around that ill-considered hack, and I'm certainly
> > not going to hold still for being told that I shouldn't use tmpfiles.
> 
> Ideally, the services would do that on their own, instead of expecting
> pre-created content in a volatile directory. Services which are started
> with root privileges should just create the needed stuff on their own,
> from within the daemon,  with every service startup, and not rely on
> any pre-existing content in runtime directories.
> 
> Tmpfiles is a very simple mechanism offered by systemd, to work around
> services which need help here, meeting their historic expectations.

We're getting a bit off-topic here, but I categorically reject that line of reasoning.  It presumes that every daemon starts as root, which is neither true nor a good idea.  If you're not root then you can't make a file in /var/run/.  The *entire point* of a mysql-owned /var/run/mysql/ subdirectory is so that mysqld can create pidfiles in it without being root, and from a security standpoint it is necessary that something other than mysqld be responsible for providing that subdirectory.  Historically, installing the RPM took care of that.  Now we need a crutch (tmpfiles) because somebody thought that it was OK to blow off the expectation that objects created by an RPM install would stay around after reboot.  But at least there's still just one trust point involved in recreating those must-be-root-to-create objects.  Saying that it's better to redistribute this responsibility into individual daemons is flat out BS.  I do hope our system design is not being controlled by people who think that.

Most services are started as root and drop privileges. All SYSV scripts have
been started as root. That's what the explanation was about. Such services
are better off to create the stuff on their own at bootup.

Services which are started right away as a specific uid/gid *can* use tmpfiles,
to fulfil their requirements just by dropping a text file. That's why it is
provided by systemd.

Service who don't like to use tmpfiles for whatever reason, and do directly
start-up an non-root, can use User=, PermissionsStartOnly=true, ExecStartPre=,
and do their needed preparation with root privileges in ExecStartPre=.

To use your own words: relying on RPM filling volatile directories is flat
out BS done by packagers, today and in the past. And I hope people who did
such things in the past understand today why it was always a broken design.

I've tested it, and when the uid/gid are locally resolvable mysqld indeed starts correctly.

Would it be possible to change the tmpfiles stuff from eg
d /var/run/mysqld 0755 mysql mysql -
to
d /var/run/mysqld 0755 mysql mysql - mysqld
where the last (optional) item is the service that depends on this. systemd
could then execute this entry just before the service is to be started.
This would make tmpfiles more general and could be used for other stuff that's
locally added to init.d and to scheduled to run after all needed resources are
fulfilled.

If the network is not available local user names should still be resolvable. tmpfiles is started very early at boot, before MySQL is started, and hence creation of the directory should finish successfully before MySQL gets started.

If I read you bug report correctly then the directory is not created by tmpfiles at all, even though a tmpfiles fragment has been dropped in /etc/tmpfiles.d?

I see some confusion here about /var/run/mysqld vs. /var/run/mysql? Maybe that's the problem?

If tmpfiles fails to create a directory then it will log about this. Do you find any tmpfiles-related log output in syslog?

Hi,
As my test (https://bugzilla.redhat.com/show_bug.cgi?id=730209#c11) showed this problem doesn't show up when the mysql uid/gid are locally resolvable.

The /var/run/mysql thing is my typo in the bug report; it's actually /var/run/mysqld, of course. I've changed the title of this report.

Indeed I find in the messages file:
Aug  1 13:30:44 pc67340132 kernel: [   32.552464] systemd-tmpfiles[1007]: [/etc/tmpfiles.d/mysql.conf:1] Unknown user 'mysql'.
Aug  1 13:30:44 pc67340132 kernel: [   34.828012] systemd[1]: systemd-tmpfiles-setup.service: main process exited, code=exited, status=1
Aug  1 13:30:44 pc67340132 kernel: [   34.834106] systemd[1]: Unit systemd-tmpfiles-setup.service entered failed state.


I understand why this happens.
What I don't understand is why creating the /var/run/<servicename> directory is done with tmpfiles when the service startupscript runs as root anyway and easily can do that itself. When the goal is to eventually let everything be done by systemd, I would like to submit that tmpfiles should be parametrized so that entries in tmpfile for a service would run when that service starts. That would give us the freedom to make services dependent on other things than the designers of systemd (can) foresee and allows us for example to add our own services that should be started when everything else is already up and running.
Otherwise, we would need to implement our own init.d system run off of rc.local, which at least I wouldn't want to do. (An example of this would be starting license servers).

(In reply to comment #13)
> What I don't understand is why creating the /var/run/<servicename> directory is
> done with tmpfiles when the service startupscript runs as root anyway and
> easily can do that itself.

Just to clarify that: the mysqld service *doesn't* launch as root, and I don't think it would be a step in the right direction to make it do so.

> When the goal is to eventually let everything be
> done by systemd, I would like to submit that tmpfiles should be parametrized so
> that entries in tmpfile for a service would run when that service starts.

Yeah, the current design of tmpfiles seems a bit inadequate for this use-case.  I'm not sure that the specific use-case is important enough to justify improving tmpfiles, but perhaps the systemd designers will keep this in mind in case they come across other cases that could use better support here.

Well, the service startupscript *does* run as root. In this case mysqld_safe is started as root but this drops privileges to the user mysql because of the --user=mysql parameter.

To clarify, I mean (service startupscript) == /etc/init.d/mysqld

I am sorry, but system users (and "mysql" is one) need to be resolvable locally at any time. I don't think it is worth trying to make a system work where system users can be stored on remote user databases only. If you really think it is, then please bring this up with FESCO.

Since I see nothing to fix here, I'll close this bug now.