Fedora Account System
Red Hat Associate
Red Hat Customer
Since updating my network of machines to Fedora 26, I find that loginctl terminate-user does not work reliably. It does work rarely but most of the time it will tell me "No user XXX known or logged in" where XXX is the numeric ID of the user I provided: [root@ld74 ~]# who shobeiri :0 2017-08-23 08:08 shobeiri pts/0 2017-08-23 08:08 (:0) root pts/1 2017-08-30 10:24 (172.21.86.85) [root@ld74 ~]# loginctl terminate-user shobeiri Could not terminate user: No user 8535 known or logged in [root@ld74 ~]# id shobeiri uid=8535(shobeiri) gid=8535(shobeiri) groups=8535(shobeiri) [root@ld74 ~]# loginctl terminate-user 8535 Could not terminate user: No user 8535 known or logged in This doesn't seem to happen for every machine for every user I want to log out, but one I see it for a user/machine pair it will continue to happen. I have a script which identifies idle users on my network and logs them out, and this is causing that to fail in the majority of cases. In case it matters here, my systems are using SSSD and user information comes from LDAP. As far as I can tell, the mapping from UID to user name and back are working fine and no other component of the system seems to have any issues mapping users.
What do "loginctl list-sessions" and "loginctl list-users" show?
Surprisingly to me, they only show the root login I've made to check. So I'm starting to think that terminate-user worked once on each problem machine, but somehow it failed to actually terminate the entire session such that they still have multiple active logins. For the record, all of my machines have KillUserProcesses=yes, which in F26 has somehow stopped actually killing all of the user processes. For most of the time that results in pulseaudio sticking around after everything else has gone away, but maybe it's worse than I thought and the whole session actually lives on.
Created attachment 1322831 [details] Systemd-related logs when running systemctl terminate-user OK, so I confirmed that it was as I suspected. I ran my script which finds idle users and runs loginctl terminate-user. This runs but doesn't actually kill the user processes, so they stay logged in even though loginctl list-sessions (and list-users) shows that nobody is logged in. So basically, even with KillUserProcesses=yes, I have significant problems with user proceses persisting after logout or after loginctl terminate-user. I updated the ticket summary to match what I'm seeing. Attached is everything systemd-related that was logged when I ran my idle-killing script, which itself just greps idle time and runs loginctl terminate-user. There might be a bit of other stuff in there but I think it should give you the relevant bits. One thing that stands out is: Sep 06 19:09:07 ld92.e.math.uh.edu audit[21482]: AVC avc: denied { read } for pid=21482 comm="systemd-logind" name="dbus-1" dev="tmpfs" ino=5548194 scontext=system_u:system_r:systemd_logind_t:s0 tcontext=unconfined_u:object_r:session_dbusd_tmp_t:s0 tclass=dir permissive=0 Sep 06 19:09:07 ld92.e.math.uh.edu systemd-logind[21482]: Failed to remove runtime directory /run/user/8664: Permission denied I've no idea what that's about, but when I look afterwards, that directory doesn't exist. And to reiterate, even without using loginctl terminate-session, I still see processes persisting after user logout. Specifically, pulseaudio seems to never exit at all, so on my network are hundreds of pulseaudio processes, each keeping open a reference to the user home directory and causing all sorts of problems. At this point I have no idea why; I set KillUserProcesses=yes on purpose to ensure that things die when they're supposed to.
That error by itself doesn't mean much: user_remove_runtime_path() is called by user_finalize() which is called by user_finalize() without checking the return value. But I can reproduce the bug. On F26, not on F27 so far.
https://github.com/systemd/systemd/pull/7096
I'm vacationing at the moment so I can't easily check, but getting this implemented and tested on my F26 machines is going to be at the top of my list when I get back next week. Is there any chance this can easily be massaged to apply to the version of systemd in F26?
Yes, the patch is trivial, so no trouble to apply it in F26. We're still discussing and testing the patch upstream though.
So I'm back in the office today; I went ahead and applied your PR against the F26 systemd, built it and pushed it out to 100+ desktops. So far it does look like it helps some of my situation in that I haven't seen a session persist wholesale after loginctl terminate-user. However, I am still seeing pulseaudio survive. I don't know if it's catching whatever signal systemd sends to it (which I guess must not be SIGKILL), or if something is restarting it and somehow causing it to escape the cgroup. I do note that in the couple of examples I was able to watch tonight, the pulseaudio process did appear to exit later. I'm not sure if that's pulseaudio's natural behavior or if systemd is sending it a SIGKILL after a while, though. In any case, I just want to make sure that no bits of the user session (and particularly, no bits which keep their home directories counted) persist after the session has been closed.
It should send SIGKILL after sending SIGTERM. Hmmm. I'll try to reproduce this. Are you using the defaults for cgroups layout? I.e. hybrid, with /sys/fs/cgroup/systemd/cgroup.subtree_control present. What kernel version?
I'm using whatever Fedora 26 does by default. /sys/fs/cgroup/systemd/cgroup.sane_behavior is present. Most kernels are 4.13.5-200.fc26.x86_64 but a couple are 4.12.12-300.fc26.x86_64. And for the record, all machines have systemd-233-6.fc26.uh.1.x86_64, which is just 233-6 with https://github.com/systemd/systemd/pull/7096.patch applied. I have no systemd.kill file. I did a bit of additional monitoring this morning and I also saw gam_server processes surviving the initial session teardown, but they appear to go away after about 20 seconds. What's the default systemd timeout before sending KILL? The systemd.kill manpage just says "after a timeout". I do know that before I went on vacation I was definitely seeing pulseaudio persist basically forever after a session termination but it does sort of look like it's getting cleaned up properly now. I'm not sure if that's due to your work in pull 7096 or some other update that happened, but I'll keep watching and see if I can reproduce those problems.
I note that https://github.com/systemd/systemd/pull/7096 has been merged to systemd master. I also note that after updating a few machines to stock F27 (from F26 + a locally built systemd with that PR applied on top), my issue came back. So I've rebuilt F27's systemd package (systemd-234-9.fc27.x86_64) with that patch on top and applied it to my F27 machines. I don't expect any issues but we'll know in a few days.
It seems that we've pushed a git snapshot into F27 (though I don't see any branches upstream so I'm not sure exactly where it's coming from). Unfortunately the patch to fix this issue didn't seem to make it in. Local builds to the rescue.
This message is a reminder that Fedora 26 is nearing its end of life. Approximately 4 (four) weeks from now Fedora will stop maintaining and issuing updates for Fedora 26. It is Fedora's policy to close all bug reports from releases that are no longer maintained. At that time this bug will be closed as EOL if it remains open with a Fedora 'version' of '26'. Package Maintainer: If you wish for this bug to remain open because you plan to fix it in a currently maintained version, simply change the 'version' to a later Fedora version. Thank you for reporting this issue and we are sorry that we were not able to fix it before Fedora 26 is end of life. If you would still like to see this bug fixed and are able to reproduce it against a later version of Fedora, you are encouraged change the 'version' to a later Fedora version prior this bug is closed as described in the policy above. Although we aim to fix as many bugs as possible during every release's lifetime, sometimes those efforts are overtaken by events. Often a more recent Fedora release includes newer upstream software that fixes bugs or makes them obsolete.
Fedora 26 changed to end-of-life (EOL) status on 2018-05-29. Fedora 26 is no longer maintained, which means that it will not receive any further security or bug fix updates. As a result we are closing this bug. If you can reproduce this bug against a currently maintained version of Fedora please feel free to reopen this bug against that version. If you are unable to reopen this bug, please file a new report against the current release. If you experience problems, please add a comment to this bug. Thank you for reporting this bug and we are sorry it could not be fixed.