Bug 1185277
Summary: | Failed at step CGROUP spawning /usr/lib/systemd/systemd | ||||||||
---|---|---|---|---|---|---|---|---|---|
Product: | [Fedora] Fedora | Reporter: | Harald Reindl <h.reindl> | ||||||
Component: | systemd | Assignee: | systemd-maint | ||||||
Status: | CLOSED EOL | QA Contact: | Fedora Extras Quality Assurance <extras-qa> | ||||||
Severity: | unspecified | Docs Contact: | |||||||
Priority: | unspecified | ||||||||
Version: | 24 | CC: | aschorr, daherrma, damnedyankee, dominik, igeorgex, jik, johannbg, jsynacek, lnykryn, loganjerry, mschmidt, msekleta, nerijus, ngaywood, pspacek, redhat-bugzilla, s, systemd-maint, thomas, tim, twaugh, vedran, vpavlin, watanabe.yu, zbyszek | ||||||
Target Milestone: | --- | Keywords: | Reopened | ||||||
Target Release: | --- | ||||||||
Hardware: | Unspecified | ||||||||
OS: | Unspecified | ||||||||
Whiteboard: | |||||||||
Fixed In Version: | Doc Type: | Bug Fix | |||||||
Doc Text: | Story Points: | --- | |||||||
Clone Of: | Environment: | ||||||||
Last Closed: | 2017-08-08 11:54:15 UTC | Type: | Bug | ||||||
Regression: | --- | Mount Type: | --- | ||||||
Documentation: | --- | CRM: | |||||||
Verified Versions: | Category: | --- | |||||||
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||||
Cloudforms Team: | --- | Target Upstream Version: | |||||||
Embargoed: | |||||||||
Attachments: |
|
Description
Harald Reindl
2015-01-23 11:14:39 UTC
Is this with the latest systemd, or the one from F21 installation media? Can you attach the full boot log? systemd-216-17.fc21.x86_64 was also there with the latest build(s) and i upgraded from munal koji download before wrote the bugreport, dracut -f also called after update Created attachment 983407 [details] dmesg and /var/log/messages after reboot keep in mind that is the same environment as https://bugzilla.redhat.com/show_bug.cgi?id=1184016 if you look for something in that case OK, this is an issue I can reproduce, not bug 1184016. Sorry for bugspam. Please attache the output from journalctl -b and confirm that your are not running an custom kernel Created attachment 984842 [details]
output of journalctl
output attached, happens again around the *useless* log-flood of *useless* usermanager for ordinary cronjobs which worked decades before systemd and "user-manager"
hopefully somebody re-considers all that bload for simple tasks....
(In reply to Vedran Miletić from comment #5) > OK, this is an issue I can reproduce, not bug 1184016. Sorry for bugspam. Please attache the output from journalctl -b and confirm that your are not running an custom kernel (In reply to Harald Reindl from comment #7) > Created attachment 984842 [details] > output of journalctl > > output attached, happens again around the *useless* log-flood of *useless* > usermanager for ordinary cronjobs which worked decades before systemd and > "user-manager" > > hopefully somebody re-considers all that bload for simple tasks.... I asked you for the output of journalctl -b and there is not a single kernel entry in that journal. Please provide the exact output of the journal which contains the complete bootlog as Zbigniew had already asked for in comment two and leave this rant of yours at home. > I asked you for the output of journalctl -b and there > is not a single kernel entry in that journal so what - a full bootlog is already attached for days now and i don't use journald for persistent storage - that won't change > Please provide the exact output of the journal which contains > the complete bootlog as Zbigniew had already asked for what about look at the attachment i made 2 hours after that request days ago? https://bugzilla.redhat.com/attachment.cgi?id=983407 > and leave this rant of yours at home given the ongoing troubles with log-flooding, still persistent bugs from F20 and the complete failing of systemd in F21 if someone is so crazy using systemd features my rant is more than moderate I also asked you to confirm you are not running custom kernel and you did not answer that. no - otherwise i would not write bugreports for fedora core components [root@testserver:~]$ uname -r 3.18.3-201.fc21.x86_64 [root@testserver:~]$ cat yum.log | grep kernel Jan 22 01:18:36 Installed: kernel-core-3.18.3-201.fc21.x86_64 Jan 22 01:18:49 Updated: kernel-headers-3.18.3-201.fc21.x86_64 Jan 22 01:40:38 Erased: kernel-core-3.18.2-200.fc21.x86_64 Enable persistent journal reboot and provide the full output from the journal and confirm you are not running this in containers or as virtual image Based on the hostname of the machine in the journal log your provide that should not be a problem since it's labeled as a test machine this is a VMNware guest running from Fedora 9 to Fedora 20, recently upgraded to F21, since our whole production is running on top of VMware vSphere and before systemd upgrade of F21 it *must not* matter if this is a virtual machine - until F21 is not running smooth and without namespace troubles (see other bugreport) i even don't consider touch any of my physical hosts with it persistent journal - somwhere around tomorrow, there is currently running a long DBMail stress test against MariaDB 10 It should not matter if it's VMWare but is anyone experiencing this running Fedora in VMware or running custom kernel which has not cgroups enable ( which usally triggers this error ) ? The other issue might be that your cron jobs are doing something weird or you have a wierd setup which locks them out from doing what they are supposed to do You need to attach your cron script that is being run as apache here... Jan 27 08:00:01 testserver.rhsoft.net systemd[1]: Starting User Manager for UID 48... Jan 27 08:00:01 testserver.rhsoft.net systemd[4881]: Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory Jan 27 08:00:01 testserver.rhsoft.net systemd[1]: Started User Manager for UID 48. Jan 27 08:00:01 testserver.rhsoft.net systemd[1]: Starting Session 133 of user apache. Jan 27 08:00:01 testserver.rhsoft.net systemd[1]: Started Session 133 of user apache. Jan 27 08:00:01 testserver.rhsoft.net systemd[1]: Starting Session 134 of user root. Jan 27 08:00:01 testserver.rhsoft.net systemd[1]: Started Session 134 of user root. Jan 27 08:00:01 testserver.rhsoft.net CROND[4894]: (root) CMD ( nice php /scripts/dbmail-snapshot/build.php) Jan 27 08:00:01 testserver.rhsoft.net CROND[4901]: (root) CMD ( nice -n 19 ionice -c 3 php /scripts/check-uploadtemp.php) Jan 27 08:00:01 testserver.rhsoft.net CROND[4905]: (root) CMD ( nice php /usr/local/scripts/rh_watchdog.php) Jan 27 08:00:01 testserver.rhsoft.net CROND[4902]: (apache) CMD ( nice -n 19 ionice -c 3 dash /scripts/cleanup-uploadtemp.cron) Jan 27 08:00:01 testserver.rhsoft.net CROND[4906]: (root) CMD ( /usr/bin/rpm -q dbmail > /www/mailadmin/modules/dbmail-snapshot.txt) Jan 27 08:00:01 testserver.rhsoft.net systemd[1]: Stopped User Manager for UID 48. Jan 27 08:00:01 testserver.rhsoft.net systemd[1]: Stopping user-48.slice. Same thing about this one Jan 27 08:03:01 testserver.rhsoft.net systemd[5009]: Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory Jan 27 08:03:01 testserver.rhsoft.net systemd[1]: Started User Manager for UID 0. Jan 27 08:03:01 testserver.rhsoft.net systemd[1]: Starting Session 136 of user root. Jan 27 08:03:01 testserver.rhsoft.net systemd[1]: Started Session 136 of user root. Jan 27 08:03:01 testserver.rhsoft.net CROND[5014]: (root) CMD ( nice -n 19 ionice -c 3 php /scripts/check-uploadtemp.php) Jan 27 08:03:01 testserver.rhsoft.net systemd[1]: Stopped User Manager for UID 0. This one runs fine Jan 27 08:05:01 testserver.rhsoft.net systemd[1]: Started User Manager for UID 0. Jan 27 08:05:01 testserver.rhsoft.net CROND[5058]: (root) CMD ( nice php /scripts/dbmail-snapshot/build.php) Jan 27 08:05:01 testserver.rhsoft.net CROND[5059]: (root) CMD ( /usr/bin/rpm -q dbmail > /www/mailadmin/modules/dbmail-snapshot.txt) Jan 27 08:05:01 testserver.rhsoft.net systemd[1]: Stopping User Manager for UID 0... So what's the difference between those cron jobs $PATH issues maybe? the cronjob running as "apache" is as simple as a cronjob can be, that's just a replacement for manage expired session-files in random webserver-workers and also remove remaining old temp-uploads in the previuos shared folder */30 * * * * apache nice -n 19 ionice -c 3 dash /scripts/cleanup-uploadtemp.cron #!/usr/bin/dash find /var/www/uploadtemp -type f -mmin +60 -delete find /var/www/sessiondata -type f -mmin +45 -delete __________________________________________________________________ /scripts/check-uploadtemp.php does nothing else than read the apache-vhost-configs, parse out the document root and make sure under each docroot a subfolder "uploadtemp" exists with the correct permissions the reason behind is that our cms-installer refuses to rsync the template system if the document root is not empty and the script create vhosts may also create ones not became a cms-setup (in both cases the vhost and it's config is created with the same script-call) __________________________________________________________________ /scripts/dbmail-snapshot/build.php just looks if there is a new upstream-coomit ID is stored in a database, fetch the tarball in that case, fires up the rpm-build and restarts dbmail services __________________________________________________________________ $PATH issues are unlikely, that below is on top of /etc/crontab and all cronjobs are inside /etc/crontab SHELL=/usr/bin/bash PATH=/usr/bin:/usr/sbin LANG=en_GB.UTF-8 MAILTO=root HOME=/ One is set to bash while the cron job runs as dash ( #!/usr/bin/dash ) try changing that to "bash" and see if that fixes the issue the dash one is the simple posted above don't get me wrong but with that logic since bash is the default shell, it would not be possible to run crojobs with /usr/bin/perl, /usr/bin/php, /usr/bin/python or whatever interpreter in a clean way, others are running and it's happening randomly if that would be the reason it would happen every 30 minutes, the machine has a uptime of 18 hours (23:11:17 up 18:56, 1 user, load average: 0,11, 0,19, 0,21) and the journal don't have entries for each call [root@testserver:~]$ journalctl | grep "spawning /usr/lib/systemd/systemd" Jan 27 12:45:01 testserver.rhsoft.net systemd[9346]: Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory Jan 27 13:00:01 testserver.rhsoft.net systemd[9535]: Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory Jan 27 13:05:01 testserver.rhsoft.net systemd[9718]: Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory Jan 27 13:30:01 testserver.rhsoft.net systemd[10058]: Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory Jan 27 14:01:01 testserver.rhsoft.net systemd[10585]: Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory Jan 27 14:20:01 testserver.rhsoft.net systemd[10871]: Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory Jan 27 14:40:01 testserver.rhsoft.net systemd[11162]: Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory Jan 27 15:00:01 testserver.rhsoft.net systemd[11733]: Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory Jan 27 17:00:01 testserver.rhsoft.net systemd[13269]: Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory Jan 27 18:03:01 testserver.rhsoft.net systemd[14196]: Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory Jan 27 18:03:46 testserver.rhsoft.net systemd[14338]: Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory Jan 27 18:03:47 testserver.rhsoft.net systemd[14523]: Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory Jan 27 19:00:02 testserver.rhsoft.net systemd[31202]: Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory Jan 27 19:15:01 testserver.rhsoft.net systemd[5079]: Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory Jan 27 21:30:01 testserver.rhsoft.net systemd[8480]: Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory Jan 27 22:30:01 testserver.rhsoft.net systemd[15933]: Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory I did not ask you to spew nonsense into the bug report I asked you to run the cron jobs that triggered this as bash( the apache job you refered to ) and see if it triggers it. Your Dash setup could be broken, your php setup could be broken etc there is a difference in your cron jobs as I mentioned in comment 16 Some run without problems some do not and trigger this so surely you can use those administrative skills of yours change the cron job that triggered this to bash and reduce that particular cron job to a minute to see if it happens. besides i did now change the one to bash and run every minute, don't call others to spew nonsense until you re-think about about your assumptions if my PHP setup would be broken the other cronjobs running every 5 minutes would trigger the same error and if my bash setup would be broken where where the problem at 15:30, 16:00, and 17:30 Jan 27 15:00:01 testserver.rhsoft.net systemd[11733]: Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory Jan 27 17:00:01 testserver.rhsoft.net systemd[13269]: Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory Jan 27 18:03:01 testserver.rhsoft.net systemd[14196]: Failed at step CGROUP and EVEN IF my php and dash setup would be broken (broken in what way) systemd would have a massive problem if it's cgroups-controller got shoot in the foot by a userland script that's the result of running the small shell-script with bash instead das every minute, as you can see the error is triggered only randomly Jan 27 23:35:01 testserver systemd: Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory Jan 27 23:38:01 testserver systemd: Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory Jan 27 23:41:01 testserver systemd: Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory Jan 27 23:40:01 testserver CROND[18543]: (apache) CMD ( nice -n 19 ionice -c 3 bash /scripts/cleanup-uploadtemp.cron) Jan 27 23:41:01 testserver CROND[18607]: (apache) CMD ( nice -n 19 ionice -c 3 bash /scripts/cleanup-uploadtemp.cron) Jan 27 23:42:01 testserver CROND[18649]: (apache) CMD ( nice -n 19 ionice -c 3 bash /scripts/cleanup-uploadtemp.cron) Jan 27 23:43:01 testserver CROND[18704]: (apache) CMD ( nice -n 19 ionice -c 3 bash /scripts/cleanup-uploadtemp.cron) Jan 27 23:44:01 testserver CROND[18758]: (apache) CMD ( nice -n 19 ionice -c 3 bash /scripts/cleanup-uploadtemp.cron) Jan 27 23:41:01 testserver.rhsoft.net systemd[1]: Starting user-48.slice. Jan 27 23:41:01 testserver.rhsoft.net systemd[1]: Created slice user-48.slice. Jan 27 23:41:01 testserver.rhsoft.net systemd[1]: Starting User Manager for UID 48... Jan 27 23:41:01 testserver.rhsoft.net systemd[18602]: Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory Jan 27 23:41:01 testserver.rhsoft.net systemd[1]: Started User Manager for UID 48. Jan 27 23:41:01 testserver.rhsoft.net systemd[1]: Starting Session 712 of user apache. Jan 27 23:41:01 testserver.rhsoft.net systemd[1]: Started Session 712 of user apache. Jan 27 23:41:01 testserver.rhsoft.net CROND[18607]: (apache) CMD ( nice -n 19 ionice -c 3 bash /scripts/cleanup-uploadtemp.cron) Jan 27 23:41:01 testserver.rhsoft.net systemd[1]: Stopped User Manager for UID 48. Jan 27 23:41:01 testserver.rhsoft.net systemd[1]: Stopping user-48.slice. Jan 27 23:41:01 testserver.rhsoft.net systemd[1]: Removed slice user-48.slice. Then we can rule out dash and it's $PATH/Environment as the underlying problem which just leaves potential PAM problems. Can you paste all the cron jobs running comment 16. They seem to be similar in nature /scripts/check-uploadtemp.php from #16 depends on a lot of libraries and /scripts/dbmail-snapshot/build.php too - i doubt they are similar in nature given that a simple dash or bash 2-liner has the same behavior since we ruled out one script and it's environment the scripts itself are completly innocent and keep in mind that it don't happen at every run if you look at these two links you see SYSTEMD has a major problem and not single scripts independent in which language they are written https://bugzilla.redhat.com/show_bug.cgi?id=1184016 https://bugzilla.redhat.com/show_bug.cgi?id=1184016#c7 Apparently it's to hard for you to do what you are asked. For example this could be because a spesific user is running the cron job ( or not ) that triggers this and relates to PAM. In anycase I got better things to do with my time let's see if Zbigniew can better deal with you and proceed further in trying to find the root cause for this. it's not a matter of "hard for you to" - it just makes no sense to paste thousands of line wired PHP code not simple runable on a different environment and containing sensible code since it also affects the *default apache user* running a 2-liner shell script with not more than two find-calls which i already posted so why do you need *thousands of lines php code* instead focus on that 2 liner? so keep your polemic "I got better things to do" for yourself when it is pretty clear that you are asking for nonsense meaning a lot fo work to review it before publish while knowing it don't help at all Jóhann, the error happens with stock cron jobs as well. So it looks like a systemd issue indeed. And it's random (i.e. doesn't happen _every_ time), like Harald said. I also have good news. I installed systemd 218-3 from F22 on F21 ~16 hours ago and I have not had "Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory" error since. I'm running a machine on DigitalOcean. Update: I see it on workstation machine as well: Sij 24 09:01:01 clover CROND[11661]: (root) CMD (run-parts /etc/cron.hourly) Sij 24 09:01:01 clover run-parts[11665]: (/etc/cron.hourly) starting 0anacron Sij 24 09:01:01 clover run-parts[11671]: (/etc/cron.hourly) finished 0anacron Sij 24 09:01:01 clover run-parts[11673]: (/etc/cron.hourly) starting mcelog.cron Sij 24 09:01:01 clover run-parts[11678]: (/etc/cron.hourly) finished mcelog.cron Sij 24 09:01:01 clover systemd[11659]: Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory Sij 25 04:01:01 clover kernel: SELinux: initialized (dev tmpfs, type tmpfs), uses transition SIDs Sij 25 04:01:01 clover CROND[14080]: (root) CMD (run-parts /etc/cron.hourly) Sij 25 04:01:01 clover systemd[14078]: Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory Sij 25 04:01:01 clover run-parts[14084]: (/etc/cron.hourly) starting 0anacron Sij 25 04:01:01 clover run-parts[14090]: (/etc/cron.hourly) finished 0anacron Sij 25 04:01:01 clover run-parts[14093]: (/etc/cron.hourly) starting mcelog.cron Sij 25 04:01:01 clover run-parts[14097]: (/etc/cron.hourly) finished mcelog.cron Before Jan 24, I was running 216-14 and I didn't get that particular error. OK, downgrade from 218-3.fc22 to 216-5.fc21 went nicely (I'm impressed, btw) so let's see if the change happened between -14 and -15 or -15 and -16. Just got this on 216-15: Sij 28 12:01:01 gedora kernel: SELinux: initialized (dev tmpfs, type tmpfs), use s transition SIDs Sij 28 12:01:01 gedora systemd[1]: Starting user-0.slice. Sij 28 12:01:01 gedora systemd[1]: Created slice user-0.slice. Sij 28 12:01:01 gedora systemd[1]: Starting Session 25 of user root. Sij 28 12:01:01 gedora systemd[1]: Started Session 25 of user root. Sij 28 12:01:01 gedora systemd[1]: Starting User Manager for UID 0... Sij 28 12:01:01 gedora systemd[3630]: Failed at step CGROUP spawning /usr/lib/sy stemd/systemd: No such file or directory Sij 28 12:01:01 gedora systemd[1]: Started User Manager for UID 0. Sij 28 12:01:01 gedora CROND[3631]: (root) CMD (run-parts /etc/cron.hourly) Sij 28 12:01:01 gedora run-parts[3634]: (/etc/cron.hourly) starting 0anacron Sij 28 12:01:01 gedora run-parts[3640]: (/etc/cron.hourly) finished 0anacron Sij 28 12:01:01 gedora systemd[1]: Stopped User Manager for UID 0. Sij 28 12:01:01 gedora systemd[1]: Stopping user-0.slice. Sij 28 12:01:01 gedora systemd[1]: Removed slice user-0.slice. So the regression happened between -14 and -15. I see this too, systemd-216-17.fc21.x86_64. It's pretty intermittent: $ journalctl MESSAGE_ID=641257651c1b4ec9a8624d7a40a9e1e7 -- Logs begin at Sat 2015-01-10 16:37:07 GMT, end at Mon 2015-02-09 17:45:40 GMT. -- Jan 25 00:43:45 snoopy systemd[4779]: Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory -- Reboot -- Jan 28 11:01:01 snoopy systemd[1786]: Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory -- Reboot -- Jan 28 17:22:11 snoopy systemd[2209]: Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory -- Reboot -- Jan 29 22:01:01 snoopy systemd[2560]: Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory -- Reboot -- Jan 31 16:01:01 snoopy systemd[3751]: Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory -- Reboot -- Feb 01 18:01:01 snoopy systemd[2384]: Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory -- Reboot -- Feb 09 14:01:01 snoopy systemd[2573]: Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory ...but most often seems to happen after crond runs cron.hourly jobs. Tim, yes, cron jobs trigger it most often. But not necessarily. no journal entries for 7 minutes previous to this -- snip -- Feb 17 09:01:01 builder2 kernel: SELinux: initialized (dev tmpfs, type tmpfs), uses transition SIDs Feb 17 09:01:01 builder2 systemd[31058]: Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory Feb 17 09:01:01 builder2 CROND[31059]: (root) CMD (run-parts /etc/cron.hourly) Feb 17 09:01:01 builder2 run-parts[31062]: (/etc/cron.hourly) starting 0anacron Feb 17 09:01:01 builder2 run-parts[31068]: (/etc/cron.hourly) finished 0anacron Feb 17 09:01:01 builder2 run-parts[31070]: (/etc/cron.hourly) starting mcelog.cron Feb 17 09:01:01 builder2 run-parts[31074]: (/etc/cron.hourly) finished mcelog.cron -- snip -- no activity for 13 minutes after that no SELinux here - system boots with selinux=0 I got it too: Feb 18 23:02:01 nerijus systemd: Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory # uname -a Linux nerijus 3.18.7-200.fc21.x86_64 #1 SMP Wed Feb 11 21:53:17 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux Not a VM. systemd-216-20.fc21.x86_64 I'm getting it also when a remote user ssh's into the system to run a simple command (but this is not always repeatable): Feb 23 08:41:01 ti139 systemd[1]: Starting user-360.slice. Feb 23 08:41:01 ti139 systemd[1]: Created slice user-360.slice. Feb 23 08:41:01 ti139 systemd[1]: Starting User Manager for UID 360... Feb 23 08:41:01 ti139 systemd[13080]: Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory Feb 23 08:41:01 ti139 systemd[1]: Started User Manager for UID 360. Feb 23 08:41:01 ti139 systemd[1]: Starting Session 113 of user zero. Feb 23 08:41:01 ti139 systemd[1]: Started Session 113 of user zero. Feb 23 08:41:01 ti139 systemd[1]: Stopped User Manager for UID 360. Feb 23 08:41:01 ti139 systemd[1]: Stopping user-360.slice. Feb 23 08:41:01 ti139 systemd[1]: Removed slice user-360.slice. This is vanilla F21 just installed this weekend on a physical server. [root@ti139 ~]# uname -a Linux ti139 3.18.7-200.fc21.x86_64 #1 SMP Wed Feb 11 21:53:17 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux [root@ti139 ~]# rpm -q systemd systemd-216-20.fc21.x86_64 SElinux is disabled. ____________________________________________________________ that's new or at least the first time due reboot after regular updates Feb 25 20:49:26 testserver systemd: Unit systemd-exit.service entered failed state. Feb 25 20:49:26 testserver systemd: systemd-exit.service failed Feb 25 20:43:56 Updated: libcom_err-1.42.12-3.fc21.x86_64 Feb 25 20:43:58 Updated: 2:samba-libs-4.1.17-1.fc21.x86_64 Feb 25 20:43:59 Updated: 2:libwbclient-4.1.17-1.fc21.x86_64 Feb 25 20:43:59 Updated: libcurl-7.37.0-13.fc21.x86_64 Feb 25 20:43:59 Updated: 32:bind-license-9.9.6-8.P1.fc21.noarch Feb 25 20:44:00 Updated: 32:bind-libs-lite-9.9.6-8.P1.fc21.x86_64 Feb 25 20:44:00 Updated: 32:bind-libs-9.9.6-8.P1.fc21.x86_64 Feb 25 20:44:01 Updated: 2:samba-common-4.1.17-1.fc21.x86_64 Feb 25 20:44:01 Updated: libss-1.42.12-3.fc21.x86_64 Feb 25 20:44:01 Updated: e2fsprogs-libs-1.42.12-3.fc21.x86_64 Feb 25 20:44:02 Updated: freetype-2.5.3-16.fc21.x86_64 Feb 25 20:44:03 Updated: freetype-devel-2.5.3-16.fc21.x86_64 Feb 25 20:44:04 Updated: e2fsprogs-1.42.12-3.fc21.x86_64 Feb 25 20:44:04 Updated: 2:libsmbclient-4.1.17-1.fc21.x86_64 Feb 25 20:44:05 Updated: 32:bind-utils-9.9.6-8.P1.fc21.x86_64 Feb 25 20:44:05 Updated: curl-7.37.0-13.fc21.x86_64 Feb 25 20:44:05 Updated: libcurl-devel-7.37.0-13.fc21.x86_64 Feb 25 20:44:06 Updated: libcom_err-devel-1.42.12-3.fc21.x86_64 Feb 25 20:44:09 Updated: initscripts-9.56.1-7.fc21.x86_64 Feb 25 20:44:09 Updated: perl-Getopt-Long-2.45-1.fc21.noarch Feb 25 20:44:09 Updated: setup-2.9.0-4.fc21.noarch Feb 25 20:44:10 Updated: xdg-utils-1.1.0-0.39.rc3.fc21.noarch ____________________________________________________________ Feb 25 18:45:01 testserver systemd: Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory Feb 25 18:55:01 testserver systemd: Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory Feb 25 19:00:01 testserver systemd: Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory Feb 25 19:03:01 testserver systemd: Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory Feb 25 19:15:01 testserver systemd: Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory Feb 25 19:30:01 testserver systemd: Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory Feb 25 19:40:01 testserver systemd: Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory Feb 25 19:50:01 testserver systemd: Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory Feb 25 20:01:01 testserver systemd: Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory Feb 25 20:10:01 testserver systemd: Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory Feb 25 20:20:01 testserver systemd: Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory Feb 25 20:30:01 testserver systemd: Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory Feb 25 20:49:26 testserver systemd: Failed at step CHDIR spawning /usr/bin/kill: Permission denied Feb 25 20:49:26 testserver systemd: Failed to start Exit the Session. Feb 25 20:49:26 testserver systemd: Dependency failed for Exit the Session. Feb 25 20:49:26 testserver systemd: Unit systemd-exit.service entered failed state. Feb 25 20:49:26 testserver systemd: systemd-exit.service failed. Feb 25 21:00:27 testserver systemd: Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory Same here with out-of-the-box stock cron jobs... Mär 03 04:01:01 nuc systemd[18183]: pam_unix(systemd-user:session): session closed for user root Mär 03 05:01:01 nuc kernel: SELinux: initialized (dev tmpfs, type tmpfs), uses transition SIDs Mär 03 05:01:01 nuc systemd[18295]: Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory Mär 03 05:01:01 nuc CROND[18296]: (root) CMD (run-parts /etc/cron.hourly) Mär 03 05:01:01 nuc run-parts[18299]: (/etc/cron.hourly) starting 0anacron Mär 03 05:01:01 nuc run-parts[18305]: (/etc/cron.hourly) finished 0anacron Mär 03 05:01:01 nuc run-parts[18307]: (/etc/cron.hourly) starting mcelog.cron Mär 03 05:01:01 nuc run-parts[18311]: (/etc/cron.hourly) finished mcelog.cron Mär 03 06:01:01 nuc kernel: SELinux: initialized (dev tmpfs, type tmpfs), uses transition SIDs Mär 03 06:01:01 nuc systemd[18404]: pam_unix(systemd-user:session): session opened for user root by (uid=0) Clean install of F21 on a physical system. [root@nuc ~]# uname -a Linux nuc 3.18.7-200.fc21.x86_64 #1 SMP Wed Feb 11 21:53:17 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux [root@nuc ~]# rpm -q systemd systemd-216-20.fc21.x86_64 I confirm that this happens on F21 with systemd-216-21.fc21.x86_64. This is clean installation (no upgrades) on bare metal. that happens on *any* Fedora 21 setup and nobody of the developers can tell me they won't see if they just read their logs, especially given that 6 different people are confirming this over 2 months now On my F21 system "journalctl MESSAGE_ID=641257651c1b4ec9a8624d7a40a9e1e7" shows me only three log entries indicating this bug (on Feb 04, 10, and 25). Do not assume this is easy to reproduce for the developers. try to setup a few cronjobs per minute calling a simple bash script doing a "sleep 1" and it will get visible, until now i hvae not faced a single Fedora 21 setup (virtual and physical) without that bug all day long and to be honest "Failed at step CGROUP spawning" sounds terrible Mar 26 12:40:01 srv-rhsoft systemd: Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory Mar 26 13:00:01 srv-rhsoft systemd: Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory Mar 26 13:00:01 srv-rhsoft systemd: Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory Mar 26 13:02:01 srv-rhsoft systemd: Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory Mar 26 13:20:01 srv-rhsoft systemd: Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory Mar 26 13:40:01 srv-rhsoft systemd: Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory Mar 26 14:00:01 srv-rhsoft systemd: Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory Mar 26 14:02:01 srv-rhsoft systemd: Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory Mar 26 14:20:01 srv-rhsoft systemd: Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory Mar 26 14:40:01 srv-rhsoft systemd: Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory Mar 26 15:00:01 srv-rhsoft systemd: Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory Mar 26 15:02:01 srv-rhsoft systemd: Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory Mar 26 15:20:01 srv-rhsoft systemd: Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory Mar 26 15:40:01 srv-rhsoft systemd: Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory Mar 26 16:01:01 srv-rhsoft systemd: Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory Mar 26 16:20:01 srv-rhsoft systemd: Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory (In reply to Michal Schmidt from comment #41) > Do not assume this is easy to reproduce for the developers. It is not about this being easy to reproduce, but that there appears to be no indication that the maintainer even accept this as a valid problem. (In reply to Jóhann B. Guðmundsson from comment #25) > Apparently it's to hard for you to do what you are asked. > > For example this could be because a spesific user is running the cron job ( > or not ) that triggers this and relates to PAM. > > In anycase I got better things to do with my time let's see if Zbigniew can > better deal with you and proceed further in trying to find the root cause > for this. It is nice that Jóhann apparently works for an employer that has no problem with internal code being posted in public. Expecting this to be the case for everyone is pretty arrogant. In any case, several users (including myself) have commented that this happens even for the most basic cron jobs on a new and clean installation, basically ruling out any local misconfiguration or weird interaction with exotic tools or scripts. Still, there is no indication whatsoever that indicates that someone is actually looking into this. (Probably better things to do) (In reply to Thomas Müller from comment #43) > It is not about this being easy to reproduce, but that there appears to be > no indication that the maintainer even accept this as a valid problem. I accept this is a valid problem. (Otherwise I would have already closed it as NOTABUG.) We do not tend to post comments that merely acknowledge the validity of bugs, unless we already have some insight to share. > Still, there is no indication whatsoever that indicates that someone is > actually looking into this. (Probably better things to do) I have been looking into this. I don't know what makes the occurrence of the bug so common for some users, while it's so rare for others. With a cron job scheduled to run every minute, it still took hours to reproduce for me. Thankfully I was able to find faster reproducer. I'm running a service like this: [Service] ExecStart=/bin/sh -c 'while :; do su - root -c "sleep 1"; done' This takes a few minutes to trigger the bug. Based on my findings so far, the problem seems to be that sometimes when recursively trimming cgroups of user-0.slice, the cgroup .../user-0.slice/user is still present and gets removed too, making the service's 'cgroup_realized' attribute out of sync with reality. Michal you might want to have a look at and share your findings with David since he had been trying to narrow down this ( race ) trigger in upstream bug 89145 any news here? looks like this also affcets F22 for a highly critical component like systemd bugs visible by just read regular syslog happen way too often and stay way too long Apr 19 22:25:37 rawhide systemd: Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory [root@rawhide ~]# rpm -q systemd systemd-219-11.fc22.x86_64 Indeed it does. But it's much rarer. true - on F22 it happens not that often while on F21 i see that messages all day long on any machine virtual as well as physical Can someone tell me how to reproduce this on -git? Running Michal's unit triggers a `systemd --user` restart every second (and I fixed several unrelated bugs now), but I could not trigger the CGROUP issue. I also played with some explicit preemption points, but it seems I can no longer trigger it after the cgroup-rewrite in systemd-225. This message is a reminder that Fedora 21 is nearing its end of life. Approximately 4 (four) weeks from now Fedora will stop maintaining and issuing updates for Fedora 21. It is Fedora's policy to close all bug reports from releases that are no longer maintained. At that time this bug will be closed as EOL if it remains open with a Fedora 'version' of '21'. Package Maintainer: If you wish for this bug to remain open because you plan to fix it in a currently maintained version, simply change the 'version' to a later Fedora version. Thank you for reporting this issue and we are sorry that we were not able to fix it before Fedora 21 is end of life. If you would still like to see this bug fixed and are able to reproduce it against a later version of Fedora, you are encouraged change the 'version' to a later Fedora version prior this bug is closed as described in the policy above. Although we aim to fix as many bugs as possible during every release's lifetime, sometimes those efforts are overtaken by events. Often a more recent Fedora release includes newer upstream software that fixes bugs or makes them obsolete. Fedora 21 changed to end-of-life (EOL) status on 2015-12-01. Fedora 21 is no longer maintained, which means that it will not receive any further security or bug fix updates. As a result we are closing this bug. If you can reproduce this bug against a currently maintained version of Fedora please feel free to reopen this bug against that version. If you are unable to reopen this bug, please file a new report against the current release. If you experience problems, please add a comment to this bug. Thank you for reporting this bug and we are sorry it could not be fixed. Still happening on Fedora 24 when running munin cron job intermittently: [...] Oct 23 15:55:01 amaterasu.greysector.net systemd[3033]: user: Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory Oct 23 17:40:01 amaterasu.greysector.net systemd[20066]: user: Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory Oct 23 19:45:01 amaterasu.greysector.net systemd[7980]: user: Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory $ id munin uid=383(munin) gid=381(munin) groups=381(munin) $ rpm -qf /etc/cron.d/munin munin-2.0.26-1.fc24.noarch $ rpm -q systemd systemd-229-16.fc24.x86_64 This message is a reminder that Fedora 24 is nearing its end of life. Approximately 2 (two) weeks from now Fedora will stop maintaining and issuing updates for Fedora 24. It is Fedora's policy to close all bug reports from releases that are no longer maintained. At that time this bug will be closed as EOL if it remains open with a Fedora 'version' of '24'. Package Maintainer: If you wish for this bug to remain open because you plan to fix it in a currently maintained version, simply change the 'version' to a later Fedora version. Thank you for reporting this issue and we are sorry that we were not able to fix it before Fedora 24 is end of life. If you would still like to see this bug fixed and are able to reproduce it against a later version of Fedora, you are encouraged change the 'version' to a later Fedora version prior this bug is closed as described in the policy above. Although we aim to fix as many bugs as possible during every release's lifetime, sometimes those efforts are overtaken by events. Often a more recent Fedora release includes newer upstream software that fixes bugs or makes them obsolete. Fedora 24 changed to end-of-life (EOL) status on 2017-08-08. Fedora 24 is no longer maintained, which means that it will not receive any further security or bug fix updates. As a result we are closing this bug. If you can reproduce this bug against a currently maintained version of Fedora please feel free to reopen this bug against that version. If you are unable to reopen this bug, please file a new report against the current release. If you experience problems, please add a comment to this bug. Thank you for reporting this bug and we are sorry it could not be fixed. |