Hide Forgot
Description of problem: Upon each reboot, a ~/.local/share/systemd/user symlink (broken it appears) is created for every user on the system. This breaks applications such as mariadb/mysql that relies on a clean filesystem to operate. In fact, only users and/or user invoked applications should touch ~. systemd should really stay out of it. Version-Release number of selected component (if applicable): systemd-207-2.el7.x86_64 How reproducible: always Step to reproduce: [root@rhel7-ha-node2 mysql]# pwd /var/lib/mysql [root@rhel7-ha-node2 mysql]# rm .local/ -rf [root@rhel7-ha-node2 mysql]# reboot [login again] [root@rhel7-ha-node2 ~]# cd /var/lib/mysql/.local/ [root@rhel7-ha-node2 .local]# ls share Actual results: .local is recreated Expected results: systemd should never touch a home dirs. Additional info: mysqlshow will recognize .local as invalid DB: | #mysql50#.local | and all mysql tools will start to fail.
Hmm, this might be caused by a systemd --user instance. Does something create a login through PAM for the user with that home directory? It sounds all a bit weird, does "loginctl" show something on the running system?
[root@rhel7-ha-node2 log]# loginctl SESSION UID USER SEAT 1 0 root all mariadb/mysql operations are generally executed as mysql user. I don't have the details on how mysql operates. The problem is easily reproducible, even if you don't run mariadb or mysql at startup. In fact, mine is disabled: [root@rhel7-ha-node2 log]# systemctl status mariadb mariadb.service - MariaDB database server Loaded: loaded (/usr/lib/systemd/system/mariadb.service; disabled) Active: inactive (dead) and I still get the .local created. The point is that beside whatever mariadb or mysql do, systemd should NOT create random stuff in home dirs. home dirs are for users and applications run by user. This is a serious violation of user home dirs. I didn't check if it's possible to exploit this symlink creation to override system or user data yet.
(In reply to Fabio Massimo Di Nitto from comment #2) > The point is that beside whatever mariadb or mysql do, systemd should NOT > create random stuff in home dirs. > > home dirs are for users and applications run by user. This is a serious > violation of user home dirs. > > I didn't check if it's possible to exploit this symlink creation to override > system or user data yet. Again, this stuff seems to run as the user that logs in, and if something in mysql uses PAM to login to become this user, this behavior is just expected functionality. This does not look like the privileged systemd, not PID1, not init; there should be no violation and no exploitable behavior.
(In reply to Kay Sievers from comment #3) > (In reply to Fabio Massimo Di Nitto from comment #2) > > The point is that beside whatever mariadb or mysql do, systemd should NOT > > create random stuff in home dirs. > > > > home dirs are for users and applications run by user. This is a serious > > violation of user home dirs. > > > > I didn't check if it's possible to exploit this symlink creation to override > > system or user data yet. > > Again, this stuff seems to run as the user that logs in, and if something in > mysql uses PAM to login to become this user, this behavior is just expected > functionality. No, this is unacceptable behaviour. A PAM login or not, as user or not, systemd or whatever part of systemd should NOT write in a user home directory. What kind of functionality do you provide by creating some kind of broken symlink?
(In reply to Fabio Massimo Di Nitto from comment #4) > No, this is unacceptable behaviour. It is. > A PAM login or not, as user or not, > systemd or whatever part of systemd should NOT write in a user home > directory. It still is. Users who log in will get a systemd --user, if that is not intended there should be no such login called by tools. > What kind of functionality do you provide by creating some kind of broken > symlink? Logged-in users get their environment set up. Please find the part on your system that logs-in as that user through PAM, I cannot reproduce your issue here with an installed and running mysql.
(In reply to Kay Sievers from comment #5) > (In reply to Fabio Massimo Di Nitto from comment #4) > > > No, this is unacceptable behaviour. > > It is. > > > A PAM login or not, as user or not, > > systemd or whatever part of systemd should NOT write in a user home > > directory. > > It still is. Users who log in will get a systemd --user, if that is not > intended > there should be no such login called by tools. > > > What kind of functionality do you provide by creating some kind of broken > > symlink? > > Logged-in users get their environment set up. > > Please find the part on your system that logs-in as that user through PAM, > I cannot reproduce your issue here with an installed and running mysql. Provide the systemd debugging tool to figure it out.
Reassigning, this is not a systemd issue. Please find out who logs in as the mysql user, and stop it from doing that.
Moving back. This is a systemd issue that should NOT write in a user home dir. Every user should be able to login to perform unprivileged operations and systemd has NO RIGHTS to alter the home dir to track anything. Write that file somewhere else in the system instead of constantly blaming other applications for your design that clearly breaks stuff.
Sure, this makes no sense at all, just login and now what do you do with the file .bash_history? Or any other file that will be created when someone logs in. A home dir is a home dir, and and a login uses the home dir, and that has nothing really to do with systemd. Just find the thing that does that, and fix it; or fix the assumption that a home dir must be empty.
(In reply to Kay Sievers from comment #9) > Sure, this makes no sense at all, > just login and now what do you do with the > file .bash_history? Or any other file that will be created when someone > logs in. [root@rhel7-ha-node2 mysql]# su - mysql Last login: Fri Sep 27 14:53:02 CEST 2013 This account is currently not available. So no .* of any kind that you can create. > A home dir is a home dir, and and a login uses the home dir, and that > has nothing really to do with systemd. Exactly, the user uses it, not systemd or PAM to add some tracking files. > > Just find the thing that does that, and fix it; or fix the assumption > that a home dir must be empty. There is no obvious "su - " or login. The only possibility is mysql de-rooting itself and that's not uncommon operation either from other applications. There is nothing to fix in mysql. It's doing what it has been doing forever.
This is getting annoying, but here, if you really need that: [root@lon]# ls -la /var/lib/mysql/.bash_history ls: cannot access /var/lib/mysql/.bash_history: No such file or directory [root@lon]# su - mysql -bash-4.2$ exit logout [root@lon]# ls -la /var/lib/mysql/.bash_history -rw------- 1 mysql mysql 5 Sep 27 15:27 /var/lib/mysql/.bash_history Dang, there goes your nice theory. And now all is broken, without any systemd involved. If stuff cannot handle a LOGIN, stuff should not use a LOGIN! It is a easy as that, and has nothing really to do with systemd, it was always that way, long before systemd existed.
Sep 27 15:19:17 localhost su: (to mysql) root on none Sep 27 15:19:17 localhost systemd: Starting user-27.slice. Sep 27 15:19:17 localhost systemd: Created slice user-27.slice. Sep 27 15:19:17 localhost systemd: Starting User Manager for 27... Sep 27 15:19:17 localhost systemd: Starting Session c1 of user mysql. Sep 27 15:19:17 localhost systemd: Started Session c1 of user mysql. Sep 27 15:19:18 localhost systemd: Failed to open private bus connection: Failed to connect to socket /run/user/27/dbus/user_bus_socket: No such file or directory Sep 27 15:19:18 localhost systemd: Mounted /sys/kernel/config. Sep 27 15:19:18 localhost systemd: Startup finished in 6ms. Sep 27 15:19:18 localhost systemd: Starting Default. Sep 27 15:19:18 localhost systemd: Reached target Default. Sep 27 15:19:18 localhost systemd: Started User Manager for 27. Sep 27 15:19:18 localhost systemd: Starting Session c2 of user mysql. Sep 27 15:19:18 localhost systemd: Started Session c2 of user mysql. Sep 27 15:19:18 localhost su: (to mysql) root on none does this log look familiar to you? There is one call to su as mysql user, but executed manually from pty doesn't generate .local. Executed from a script with no pty it does.. is that expected behaviour?
OK, closing then. mariadb shouldn't do the full login stuff for its users user, including PAM. If it does that it will get a user setup, btmp/utmp/wtmp stuff, written to the home directory, audit records, blablabla. CLosing this since there's nothing to fix in systemd, and mariadb folks don't consider this a bug.
(In reply to Lennart Poettering from comment #16) > OK, closing then. mariadb shouldn't do the full login stuff for its users > user, including PAM. If it does that it will get a user setup, > btmp/utmp/wtmp stuff, written to the home directory, audit records, > blablabla. > > CLosing this since there's nothing to fix in systemd, and mariadb folks > don't consider this a bug. Which Mariadb guys? can we please have a record of the mariadb guys recorded here?
BTW, if this is caused by some script or so doing "su mysql ..." or so to execute some command as the mysql user, then this is really a bug. "su" is mostly a user command to create real user login sessions. There's "runuser" for the more light-weight change-uid cases, which only runs a very reduced PAM session precisely to avoid issues like this.
There are two PAM config files for runuser: "runuser" and "runuser-l" The former does not call pam_systemd.so, the latter does. Didn't we want to make a similar distinction with "su" and "su-l" at some point?
(In reply to Michal Schmidt from comment #23) > There are two PAM config files for runuser: "runuser" and "runuser-l" > The former does not call pam_systemd.so, the latter does. > Didn't we want to make a similar distinction with "su" and "su-l" at some > point? Yeah, I guess that makes sense at a general level. At least "su" instead of "su -" would probably not fail then. (The real problem in the case above is still that the assumption that a directory needs to be empty for a service to function AND the same directory is the home dir of the user of the service AND the user wants, to be able to login ... it seems like a pretty broken and fragile setup.)
(In reply to Lennart Poettering from comment #22) > BTW, if this is caused by some script or so doing "su mysql ..." or so to > execute some command as the mysql user, then this is really a bug. "su" is > mostly a user command to create real user login sessions. There's "runuser" > for the more light-weight change-uid cases, which only runs a very reduced > PAM session precisely to avoid issues like this. The problem is, again, that the user dir is violated by either pam or systemd for a technical reason that still remains unanswered. If, as a user, i don't want .local/ and I remove it, either pam or systemd recreates it. this is a violation of my user dir. Is this more clear as explanation? and again: why do you need that dir/symlink there? Anything other speculation on why a script needs to su or not is out of the point.
On my system: $ ls -ad ~/.* | wc -l 63 Two of them are "." and "..". Three others are files that I created knowingly. So I have 58 dotfiles in my home directory automatically created by programs. I do not consider them to be violations of my home directory. (Although if all the programs followed the XDG specification, I'd have fewer dotfiles directly under my home directory. Instead I'd have more directories under ~/.local, ~/.config or ~/.cache.)
(In reply to Michal Schmidt from comment #26) > On my system: > $ ls -ad ~/.* | wc -l > 63 > > Two of them are "." and "..". Three others are files that I created > knowingly. So I have 58 dotfiles in my home directory automatically created > by programs. You executed those programs and you accept the fact that they created files of which you have control over. If you decide to remove them and stop executing this or that program, they are gone. You have control over your home. In this case, I can't get rid of it. I have no control without systemwide changes and it is recreated against my will (beside not logging in the system that's not really useful solution to which i guess we can both agree). I do expect to find my home as I left it basically. How would you feel if you left for work and drive back in your home to find out that somebody painted your walls of another color? :) > I do not consider them to be violations of my home directory. I don't consider a violation of my homedir either, if i willingly executed a program that could create files (or dotfiles). > (Although if all the programs followed the XDG specification, I'd have fewer > dotfiles directly under my home directory. Instead I'd have more directories > under ~/.local, ~/.config or ~/.cache.) This is a valid argument, but I think it's for another forum. Let's discuss that somewhere else ;)
All kinds of programs you knowingly or unknowingly run will create directories and files in your home directory and your runtime directory. The shell does that, some PAM modules do, GNOME does, libraries do. Everything does.
Don't get disturb in creating-stuff-in-home discussion, I just feel the need to comment on how MySQL/MariaDB handles directories in /var/lib/mysql, which is the home directory for mysql user at the same time. I asked upstream to be sure but it seems like every directory (not files; that's why all .bash_history and such stuff doesn't make problems) under /var/lib/mysql is really handled like a *database* and even empty directory is already a database. I haven't reproduced (not even tried hard) the case when .local directory is created automatically by something, but I simply tried to create such directory by hand and MariaDB was not very happy about that. I haven't studied all the consequences but it could probably make troubles during upgrade, while reporting weird messages is the least what users could expect. Worth saying, that I can't say if current behaviour of MariaDB/MySQL is correct (by behaviour I mean that it generally makes assumptions about directories in home directory of some user) but it definitely has been there for ages and I haven't noticed any issue with such approach until now. What's unfortunate -- I even can't realize what the fix in MariaDB could be if we need to fix it. Currently I see two general ways: A) change home directory of mysql user B) blacklist some directories that fulfil some requirements (e.g. ".local" with no table files for example). Just for the record, since MySQL 5.1, database directories can't include symbols like "-", "." and their ASCII value is used instead, e.g. "@002e" for ".". If a directory including pure "." is found, the name of the table is prefixed by #mysql50# and upgrade should be proceeded. That said, simply filtering ".local" directory should influence only databases called ".local" and created in MySQL 5.0 or earlier. Quite simple solution, which should work fine for 99.99%. However, there is still the 0.01%...
Aha, there is quite simple work-around in MariaDB and will be in MySQL 5.6 as well (ignore-db-dir option): https://lists.launchpad.net/maria-developers/msg06372.html https://mariadb.com/kb/en/issues-with-ignore-db-dir/
(In reply to Honza Horak from comment #30) > Don't get disturb in creating-stuff-in-home discussion, I just feel the need > to comment on how MySQL/MariaDB handles directories in /var/lib/mysql, which > is the home directory for mysql user at the same time. > > I asked upstream to be sure but it seems like every directory (not files; > that's why all .bash_history and such stuff doesn't make problems) under > /var/lib/mysql is really handled like a *database* and even empty directory > is already a database. I haven't reproduced (not even tried hard) the case > when .local directory is created automatically by something, but I simply > tried to create such directory by hand and MariaDB was not very happy about > that. I haven't studied all the consequences but it could probably make > troubles during upgrade, while reporting weird messages is the least what > users could expect. > > Worth saying, that I can't say if current behaviour of MariaDB/MySQL is > correct (by behaviour I mean that it generally makes assumptions about > directories in home directory of some user) but it definitely has been there > for ages and I haven't noticed any issue with such approach until now. > > What's unfortunate -- I even can't realize what the fix in MariaDB could be > if we need to fix it. Currently I see two general ways: A) change home > directory of mysql user B) blacklist some directories that fulfil some > requirements (e.g. ".local" with no table files for example). > > Just for the record, since MySQL 5.1, database directories can't include > symbols like "-", "." and their ASCII value is used instead, e.g. "@002e" > for ".". If a directory including pure "." is found, the name of the table > is prefixed by #mysql50# and upgrade should be proceeded. That said, simply > filtering ".local" directory should influence only databases called ".local" > and created in MySQL 5.0 or earlier. Quite simple solution, which should > work fine for 99.99%. However, there is still the 0.01%... In my tests i noticed several mysql_* tools breaking because they can't deal with the invalid db. mysqldump is an easy one to test. While we can certainly fix "in house" bits, such as mysql/mariadb, there are other projects that we know of that will fail for the same problem (galera, percona as an example). And more worrying is that we can't fix applications that we know nothing about, that are not shipped in RHEL.
(In reply to Fabio Massimo Di Nitto from comment #32) > And more worrying is that we can't fix applications that we know nothing > about, that are not shipped in RHEL. Well, we can't fix it, but with work-around from comment#31 users should be able to avoid issues, right? Or are there use-cases where such work-around won't help?
(In reply to Honza Horak from comment #33) > (In reply to Fabio Massimo Di Nitto from comment #32) > > And more worrying is that we can't fix applications that we know nothing > > about, that are not shipped in RHEL. > > Well, we can't fix it, but with work-around from comment#31 users should be > able to avoid issues, right? Or are there use-cases where such work-around > won't help? There are applications that will stop working where workaround might not exists. That leads us straight back to square 0. And today is .local, tomorrow is .something_else.. do we really expect users to set workarounds when things should be working out of the box?
(In reply to Fabio Massimo Di Nitto from comment #34) > And today is .local, tomorrow is .something_else.. Exactly. There will be more, any time, as long as the user can log in and uses the database dir as its home. > do we really expect users > to set workarounds when things should be working out of the box? No, but either the database has these strict rules about this directory _OR_ that direectory is used as the home dir for the database user. Combining both is just madness, and I-hope-nobody-will-login, or I-hope-there-will-be-no-files is broken by design. What happens here is not a bug, it is just what such a fragile setup asked for. So either the database ignores all "."-files in the database dir, or the user gets a new home directory if it wants to be able to log in.
(In reply to Kay Sievers from comment #35) > (In reply to Fabio Massimo Di Nitto from comment #34) > > > And today is .local, tomorrow is .something_else.. > > Exactly. There will be more, any time, as long as the user can log in and > uses the database dir as its home. We are cycling back again over the same issue. User has control of the home dir. I can wipe it clean and it stays clean. This new integration between systemd+PAM will restore stuff that i don't want in my homedir.. Again, we can loop here forever. The root problem is that I want control over my home dir, you are taking that away by enforcing files/dirs I don't care about. > > > do we really expect users > > to set workarounds when things should be working out of the box? > > No, but either the database has these strict rules about this directory _OR_ > that direectory is used as the home dir for the database user. Combining > both is just madness, and I-hope-nobody-will-login, or > I-hope-there-will-be-no-files is broken by design. The design is a lot simpler than you want to make it sound. I am in control of my home (and again we repeat itself) and the new approach breaks that rule. > > What happens here is not a bug, it is just what such a fragile setup asked > for. What happened here is that an assumption was made when systemd+PAM integration was designed. This new "feature" how you call it, breaks a very well established behaviour. > > So either the database ignores all "."-files in the database dir, or the > user gets a new home directory if it wants to be able to log in. Or you move that file away, because again, we fix our stuff, we have no control over all applications out there, restore a known behaviour and the problem solves itself.
Would mysql/mariadb get confused by the presence of the "lost+found" directory that would be there if the administrator chooses to mount a dedicated ext4 filesystem for /var/lib/mysql? Have we found what exactly runs the "su" command on Fabio's system? I could not find anything on my system. Is it some custom cronjob or similar?
(In reply to Michal Schmidt from comment #37) > Would mysql/mariadb get confused by the presence of the "lost+found" > directory that would be there if the administrator chooses to mount a > dedicated ext4 filesystem for /var/lib/mysql? OK, ignore this question. From the documentation it seems "lost+found" was in fact the original reason for introducing the --ignore-db-dir option in the first place.
(In reply to Michal Schmidt from comment #37) > Would mysql/mariadb get confused by the presence of the "lost+found" > directory that would be there if the administrator chooses to mount a > dedicated ext4 filesystem for /var/lib/mysql? Apparently no, it doesn't get confused, but I can retest it. > > Have we found what exactly runs the "su" command on Fabio's system? I could > not find anything on my system. Is it some custom cronjob or similar? I haven't been able to isolate it exactly. It's not trivial to reproduce as I originally described (it did look trivial/reboot because I had cluster services starting at boot). I asked for some help in the previous comments but we were too busy flaming each other ;) RHEL-HA uses a bunch of scripts to drive resources, including a mysql agent that can drive mariadb. When driving mysql that way there are 2 or 3 calls to "su - mysql ..." that executed via the resource manager, trigger the creation of .local/. When I executed all manually, one at a time, I could not recreate .local/. The only difference I could spot is the presence of a pty (none via resource manager, ptySOME_NUMBER when executed as root manually) but is that real culprit? Tho I am unable to pinpoint the exact call, without adding tons of debug lines. Even if we fix the resource agent, the problem remains for other 3rd party vendors that ship similar solutions or applications.
(In reply to Michal Schmidt from comment #38) > (In reply to Michal Schmidt from comment #37) > > Would mysql/mariadb get confused by the presence of the "lost+found" > > directory that would be there if the administrator chooses to mount a > > dedicated ext4 filesystem for /var/lib/mysql? > > OK, ignore this question. From the documentation it seems "lost+found" was > in fact the original reason for introducing the --ignore-db-dir option in > the first place. But also, a ".dir" has a special meaning for mysql/mariadb as Honza described a few comments above. a plain dir such as lost+found would only indicate an empty DB vs a DB that requires processing (and it's "a broken DB" from mysql/mariadb standpoint).
I haven't mentioned yet that now we set /sbin/nologin as a login shell for user mysql, so that user is not able to log in by default any more. Again, not a solution, but worth mentioning.
(In reply to Michal Schmidt from comment #23) > There are two PAM config files for runuser: "runuser" and "runuser-l" > The former does not call pam_systemd.so, the latter does. > Didn't we want to make a similar distinction with "su" and "su-l" at some > point? Well, this depends. I think pam_systemd should be in the stack if the tool drops env vars.. If it doesn't reset env vars it doesn't need to be in the stack. The reason is simply that I think env vars such as XDG_SESSION_ID and XDG_RUNTIME_DIR and so on should be set for all processes, always.