Bug 1185277 - Failed at step CGROUP spawning /usr/lib/systemd/systemd
Summary: Failed at step CGROUP spawning /usr/lib/systemd/systemd
Keywords:
Status: CLOSED EOL
Alias: None
Product: Fedora
Classification: Fedora
Component: systemd
Version: 24
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: systemd-maint
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2015-01-23 11:14 UTC by Harald Reindl
Modified: 2017-08-08 11:54 UTC (History)
25 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2017-08-08 11:54:15 UTC
Type: Bug


Attachments (Terms of Use)
dmesg and /var/log/messages after reboot (27.85 KB, application/octet-stream)
2015-01-23 15:41 UTC, Harald Reindl
no flags Details
output of journalctl (764.63 KB, text/plain)
2015-01-27 19:42 UTC, Harald Reindl
no flags Details


Links
System ID Priority Status Summary Last Updated
FreeDesktop.org 89145 'medium' 'RESOLVED' 'Failed at step CGROUP spawning /usr/lib/systemd/systemd' 2019-11-20 10:11:27 UTC

Description Harald Reindl 2015-01-23 11:14:39 UTC
Jan 23 05:01:02 testserver systemd: Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory                                                                                             
Jan 23 05:10:01 testserver systemd: Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory                                                                                             
Jan 23 05:20:01 testserver systemd: Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory                                                                                             
Jan 23 06:00:01 testserver systemd: Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory                                                                                             
Jan 23 06:01:01 testserver systemd: Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory                                                                                             
Jan 23 07:15:01 testserver systemd: Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory
Jan 23 07:30:01 testserver systemd: Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory
Jan 23 07:55:01 testserver systemd: Failed at step CGROUP spawning /usr/lib
besides https://bugzilla.redhat.com/show_bug.cgi?id=1184016 and https://bugzilla.redhat.com/show_bug.cgi?id=1184016#c7 a reason hestitate to upgrade my physical boxes to Fedora 21

/systemd/systemd: No such file or directory
Jan 23 08:15:01 testserver systemd: Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory
Jan 23 08:55:01 testserver systemd: Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory
Jan 23 09:03:01 testserver systemd: Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory
Jan 23 09:40:01 testserver systemd: Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory
Jan 23 09:50:01 testserver systemd: Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory
Jan 23 10:00:01 testserver systemd: Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory
Jan 23 10:05:01 testserver systemd: Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory
Jan 23 10:25:01 testserver systemd: Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory
Jan 23 10:50:01 testserver systemd: Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory
Jan 23 11:00:01 testserver systemd: Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory
Jan 23 11:15:01 testserver systemd: Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory
Jan 23 11:40:01 testserver systemd: Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory

Comment 1 Zbigniew Jędrzejewski-Szmek 2015-01-23 13:32:03 UTC
Is this with the latest systemd, or the one from F21 installation media?

Comment 2 Zbigniew Jędrzejewski-Szmek 2015-01-23 13:32:46 UTC
Can you attach the full boot log?

Comment 3 Harald Reindl 2015-01-23 15:35:51 UTC
systemd-216-17.fc21.x86_64

was also there with the latest build(s) and i upgraded from munal koji download before wrote the bugreport, dracut -f also called after update

Comment 4 Harald Reindl 2015-01-23 15:41:19 UTC
Created attachment 983407 [details]
dmesg and /var/log/messages after reboot

keep in mind that is the same environment as https://bugzilla.redhat.com/show_bug.cgi?id=1184016 if you look for something in that case

Comment 5 Vedran Miletić 2015-01-27 18:51:44 UTC
OK, this is an issue I can reproduce, not bug 1184016. Sorry for bugspam.

Comment 6 Jóhann B. Guðmundsson 2015-01-27 19:34:25 UTC
Please attache the output from journalctl -b and confirm that your are not running an custom kernel

Comment 7 Harald Reindl 2015-01-27 19:42:53 UTC
Created attachment 984842 [details]
output of journalctl

output attached, happens again around the *useless* log-flood of *useless* usermanager for ordinary cronjobs which worked decades before systemd and "user-manager"

hopefully somebody re-considers all that bload for simple tasks....

Comment 8 Jóhann B. Guðmundsson 2015-01-27 20:06:58 UTC
(In reply to Vedran Miletić from comment #5)
> OK, this is an issue I can reproduce, not bug 1184016. Sorry for bugspam.

Please attache the output from journalctl -b and confirm that your are not running an custom kernel

Comment 9 Jóhann B. Guðmundsson 2015-01-27 20:10:26 UTC
(In reply to Harald Reindl from comment #7)
> Created attachment 984842 [details]
> output of journalctl
> 
> output attached, happens again around the *useless* log-flood of *useless*
> usermanager for ordinary cronjobs which worked decades before systemd and
> "user-manager"
> 
> hopefully somebody re-considers all that bload for simple tasks....

I asked you for the output of journalctl -b and there is not a single kernel entry in that journal. 

Please provide the exact output of the journal which contains the complete bootlog as Zbigniew had already asked for in comment two and leave this rant of yours at home.

Comment 10 Harald Reindl 2015-01-27 20:16:48 UTC
> I asked you for the output of journalctl -b and there 
> is not a single kernel entry in that journal

so what - a full bootlog is already attached for days now and i don't use journald for persistent storage - that won't change

> Please provide the exact output of the journal which contains 
> the complete bootlog as Zbigniew had already asked for

what about look at the attachment i made 2 hours after that request days ago?
https://bugzilla.redhat.com/attachment.cgi?id=983407

> and leave this rant of yours at home

given the ongoing troubles with log-flooding, still persistent bugs from F20 and the complete failing of systemd in F21 if someone is so crazy using systemd features my rant is more than moderate

Comment 11 Jóhann B. Guðmundsson 2015-01-27 20:22:09 UTC
I also asked you to confirm you are not running custom kernel and you did not answer that.

Comment 12 Harald Reindl 2015-01-27 20:33:22 UTC
no - otherwise i would not write bugreports for fedora core components

[root@testserver:~]$ uname -r
3.18.3-201.fc21.x86_64

[root@testserver:~]$ cat yum.log | grep kernel
Jan 22 01:18:36 Installed: kernel-core-3.18.3-201.fc21.x86_64
Jan 22 01:18:49 Updated: kernel-headers-3.18.3-201.fc21.x86_64
Jan 22 01:40:38 Erased: kernel-core-3.18.2-200.fc21.x86_64

Comment 13 Jóhann B. Guðmundsson 2015-01-27 20:43:14 UTC
Enable persistent journal reboot and provide the full output from the journal and confirm you are not running this in containers or as virtual image 

Based on the hostname of the machine in the journal log your provide that should not be a problem since it's labeled as a test machine

Comment 14 Harald Reindl 2015-01-27 20:49:19 UTC
this is a VMNware guest running from Fedora 9 to Fedora 20, recently upgraded to F21, since our whole production is running on top of VMware vSphere and before systemd upgrade of F21 it *must not* matter if this is a virtual machine - until F21 is not running smooth and without namespace troubles (see other bugreport) i even don't consider touch any of my physical hosts with it

persistent journal - somwhere around tomorrow, there is currently running a long DBMail stress test against MariaDB 10

Comment 15 Jóhann B. Guðmundsson 2015-01-27 21:06:45 UTC
It should not matter if it's VMWare but is anyone experiencing this running Fedora in VMware or running custom kernel which has not cgroups enable ( which usally triggers this error ) ? 

The other issue might be that your cron jobs are doing something weird or you have a wierd setup which locks them out from doing what they are supposed to do

You need to attach your cron script that is being run as apache here...

Jan 27 08:00:01 testserver.rhsoft.net systemd[1]: Starting User Manager for UID 48...
Jan 27 08:00:01 testserver.rhsoft.net systemd[4881]: Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory
Jan 27 08:00:01 testserver.rhsoft.net systemd[1]: Started User Manager for UID 48.
Jan 27 08:00:01 testserver.rhsoft.net systemd[1]: Starting Session 133 of user apache.
Jan 27 08:00:01 testserver.rhsoft.net systemd[1]: Started Session 133 of user apache.
Jan 27 08:00:01 testserver.rhsoft.net systemd[1]: Starting Session 134 of user root.
Jan 27 08:00:01 testserver.rhsoft.net systemd[1]: Started Session 134 of user root.
Jan 27 08:00:01 testserver.rhsoft.net CROND[4894]: (root) CMD (     nice php /scripts/dbmail-snapshot/build.php)
Jan 27 08:00:01 testserver.rhsoft.net CROND[4901]: (root) CMD (     nice -n 19 ionice -c 3 php /scripts/check-uploadtemp.php)
Jan 27 08:00:01 testserver.rhsoft.net CROND[4905]: (root) CMD (     nice php /usr/local/scripts/rh_watchdog.php)
Jan 27 08:00:01 testserver.rhsoft.net CROND[4902]: (apache) CMD (   nice -n 19 ionice -c 3 dash /scripts/cleanup-uploadtemp.cron)
Jan 27 08:00:01 testserver.rhsoft.net CROND[4906]: (root) CMD (     /usr/bin/rpm -q dbmail > /www/mailadmin/modules/dbmail-snapshot.txt)
Jan 27 08:00:01 testserver.rhsoft.net systemd[1]: Stopped User Manager for UID 48.
Jan 27 08:00:01 testserver.rhsoft.net systemd[1]: Stopping user-48.slice.

Comment 16 Jóhann B. Guðmundsson 2015-01-27 21:23:40 UTC
Same thing about this one

Jan 27 08:03:01 testserver.rhsoft.net systemd[5009]: Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory
Jan 27 08:03:01 testserver.rhsoft.net systemd[1]: Started User Manager for UID 0.
Jan 27 08:03:01 testserver.rhsoft.net systemd[1]: Starting Session 136 of user root.
Jan 27 08:03:01 testserver.rhsoft.net systemd[1]: Started Session 136 of user root.
Jan 27 08:03:01 testserver.rhsoft.net CROND[5014]: (root) CMD (     nice -n 19 ionice -c 3 php /scripts/check-uploadtemp.php)
Jan 27 08:03:01 testserver.rhsoft.net systemd[1]: Stopped User Manager for UID 0.

This one runs fine

Jan 27 08:05:01 testserver.rhsoft.net systemd[1]: Started User Manager for UID 0.
Jan 27 08:05:01 testserver.rhsoft.net CROND[5058]: (root) CMD (     nice php /scripts/dbmail-snapshot/build.php)
Jan 27 08:05:01 testserver.rhsoft.net CROND[5059]: (root) CMD (     /usr/bin/rpm -q dbmail > /www/mailadmin/modules/dbmail-snapshot.txt)
Jan 27 08:05:01 testserver.rhsoft.net systemd[1]: Stopping User Manager for UID 0...

So what's the difference between those cron jobs $PATH issues maybe?

Comment 17 Harald Reindl 2015-01-27 21:42:23 UTC
the cronjob running as "apache" is as simple as a cronjob can be, that's just a replacement for manage expired session-files in random webserver-workers and also remove remaining old temp-uploads in the previuos shared folder

*/30 * * * *  apache    nice -n 19 ionice -c 3 dash /scripts/cleanup-uploadtemp.cron

#!/usr/bin/dash
find /var/www/uploadtemp -type f -mmin +60 -delete
find /var/www/sessiondata -type f -mmin +45 -delete
__________________________________________________________________

/scripts/check-uploadtemp.php does nothing else than read the apache-vhost-configs, parse out the document root and make sure under each docroot a subfolder "uploadtemp" exists with the correct permissions

the reason behind is that our cms-installer refuses to rsync the template system if the document root is not empty and the script create vhosts may also create ones not became a cms-setup (in both cases the vhost and it's config is created with the same script-call)
__________________________________________________________________

/scripts/dbmail-snapshot/build.php just looks if there is a new upstream-coomit ID is stored in a database, fetch the tarball in that case, fires up the rpm-build and restarts dbmail services
__________________________________________________________________

$PATH issues are unlikely, that below is on top of /etc/crontab and all cronjobs are inside /etc/crontab

SHELL=/usr/bin/bash
PATH=/usr/bin:/usr/sbin
LANG=en_GB.UTF-8
MAILTO=root
HOME=/

Comment 18 Jóhann B. Guðmundsson 2015-01-27 21:48:14 UTC
One is set to bash while the cron job runs as dash ( #!/usr/bin/dash ) try changing that to "bash" and see if that fixes the issue

Comment 19 Harald Reindl 2015-01-27 22:15:07 UTC
the dash one is the simple posted above

don't get me wrong but with that logic since bash is the default shell, it would not be possible to run crojobs with /usr/bin/perl, /usr/bin/php, /usr/bin/python or whatever interpreter in a clean way, others are running and it's happening randomly

if that would be the reason it would happen every 30 minutes, the machine has a uptime of 18 hours (23:11:17 up 18:56,  1 user,  load average: 0,11, 0,19, 0,21) and the journal don't have entries for each call

[root@testserver:~]$ journalctl | grep "spawning /usr/lib/systemd/systemd"
Jan 27 12:45:01 testserver.rhsoft.net systemd[9346]: Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory
Jan 27 13:00:01 testserver.rhsoft.net systemd[9535]: Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory
Jan 27 13:05:01 testserver.rhsoft.net systemd[9718]: Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory
Jan 27 13:30:01 testserver.rhsoft.net systemd[10058]: Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory
Jan 27 14:01:01 testserver.rhsoft.net systemd[10585]: Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory
Jan 27 14:20:01 testserver.rhsoft.net systemd[10871]: Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory
Jan 27 14:40:01 testserver.rhsoft.net systemd[11162]: Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory
Jan 27 15:00:01 testserver.rhsoft.net systemd[11733]: Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory
Jan 27 17:00:01 testserver.rhsoft.net systemd[13269]: Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory
Jan 27 18:03:01 testserver.rhsoft.net systemd[14196]: Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory
Jan 27 18:03:46 testserver.rhsoft.net systemd[14338]: Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory
Jan 27 18:03:47 testserver.rhsoft.net systemd[14523]: Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory
Jan 27 19:00:02 testserver.rhsoft.net systemd[31202]: Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory
Jan 27 19:15:01 testserver.rhsoft.net systemd[5079]: Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory
Jan 27 21:30:01 testserver.rhsoft.net systemd[8480]: Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory
Jan 27 22:30:01 testserver.rhsoft.net systemd[15933]: Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory

Comment 20 Jóhann B. Guðmundsson 2015-01-27 22:27:16 UTC
I did not ask you to spew nonsense into the bug report I asked you to run the cron jobs that triggered this as bash( the apache job you refered to )  and see if it triggers it. 

Your Dash setup could be broken, your php setup could be broken etc there is a difference in your cron jobs as I mentioned in comment 16

Some run without problems some do not and trigger this so surely you can use those administrative skills of yours change the cron job that triggered this to bash and reduce that particular cron job to a minute to see if it happens.

Comment 21 Harald Reindl 2015-01-27 22:39:40 UTC
besides i did now change the one to bash and run every minute, don't call others to spew nonsense until you re-think about about your assumptions

if my PHP setup would be broken the other cronjobs running every 5 minutes would trigger the same error and if my bash setup would be broken where where the problem  at 15:30, 16:00, and 17:30

Jan 27 15:00:01 testserver.rhsoft.net systemd[11733]: Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory
Jan 27 17:00:01 testserver.rhsoft.net systemd[13269]: Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory
Jan 27 18:03:01 testserver.rhsoft.net systemd[14196]: Failed at step CGROUP

and EVEN IF my php and dash setup would be broken (broken in what way) systemd would have a massive problem if it's cgroups-controller got shoot in the foot by a userland script

Comment 22 Harald Reindl 2015-01-27 22:45:49 UTC
that's the result of running the small shell-script with bash instead das every minute, as you can see the error is triggered only randomly

Jan 27 23:35:01 testserver systemd: Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory
Jan 27 23:38:01 testserver systemd: Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory
Jan 27 23:41:01 testserver systemd: Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory

Jan 27 23:40:01 testserver CROND[18543]: (apache) CMD (   nice -n 19 ionice -c 3 bash /scripts/cleanup-uploadtemp.cron)
Jan 27 23:41:01 testserver CROND[18607]: (apache) CMD (   nice -n 19 ionice -c 3 bash /scripts/cleanup-uploadtemp.cron)
Jan 27 23:42:01 testserver CROND[18649]: (apache) CMD (   nice -n 19 ionice -c 3 bash /scripts/cleanup-uploadtemp.cron)
Jan 27 23:43:01 testserver CROND[18704]: (apache) CMD (   nice -n 19 ionice -c 3 bash /scripts/cleanup-uploadtemp.cron)
Jan 27 23:44:01 testserver CROND[18758]: (apache) CMD (   nice -n 19 ionice -c 3 bash /scripts/cleanup-uploadtemp.cron)

Jan 27 23:41:01 testserver.rhsoft.net systemd[1]: Starting user-48.slice.
Jan 27 23:41:01 testserver.rhsoft.net systemd[1]: Created slice user-48.slice.
Jan 27 23:41:01 testserver.rhsoft.net systemd[1]: Starting User Manager for UID 48...
Jan 27 23:41:01 testserver.rhsoft.net systemd[18602]: Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory
Jan 27 23:41:01 testserver.rhsoft.net systemd[1]: Started User Manager for UID 48.
Jan 27 23:41:01 testserver.rhsoft.net systemd[1]: Starting Session 712 of user apache.
Jan 27 23:41:01 testserver.rhsoft.net systemd[1]: Started Session 712 of user apache.
Jan 27 23:41:01 testserver.rhsoft.net CROND[18607]: (apache) CMD (   nice -n 19 ionice -c 3 bash /scripts/cleanup-uploadtemp.cron)
Jan 27 23:41:01 testserver.rhsoft.net systemd[1]: Stopped User Manager for UID 48.
Jan 27 23:41:01 testserver.rhsoft.net systemd[1]: Stopping user-48.slice.
Jan 27 23:41:01 testserver.rhsoft.net systemd[1]: Removed slice user-48.slice.

Comment 23 Jóhann B. Guðmundsson 2015-01-27 23:23:21 UTC
Then we can rule out dash and it's $PATH/Environment as the underlying problem which just leaves potential PAM problems. 

Can you paste all the cron jobs running comment 16. 

They seem to be similar in nature

Comment 24 Harald Reindl 2015-01-27 23:37:04 UTC
/scripts/check-uploadtemp.php from #16 depends on a lot of libraries and /scripts/dbmail-snapshot/build.php too - i doubt they are similar in nature given that a simple dash or bash 2-liner has the same behavior

since we ruled out one script and it's environment the scripts itself are completly innocent and keep in mind that it don't happen at every run

if you look at these two links you see SYSTEMD has a major problem and not single scripts independent in which language they are written

https://bugzilla.redhat.com/show_bug.cgi?id=1184016
https://bugzilla.redhat.com/show_bug.cgi?id=1184016#c7

Comment 25 Jóhann B. Guðmundsson 2015-01-27 23:48:31 UTC
Apparently it's to hard for you to do what you are asked. 

For example this could be because a spesific user is running the cron job ( or not ) that triggers this and relates to PAM.

In anycase I got better things to do with my time let's see if Zbigniew can better deal with you and proceed further in trying to find the root cause for this.

Comment 26 Harald Reindl 2015-01-27 23:56:21 UTC
it's not a matter of "hard for you to" - it just makes no sense to paste thousands of line wired PHP code not simple runable on a different environment and containing sensible code 

since it also affects the *default apache user* running a 2-liner shell script with not more than two find-calls which i already posted so why do you need *thousands of lines php code* instead focus on that 2 liner?

so keep your polemic "I got better things to do" for yourself when it is pretty clear that you are asking for nonsense meaning a lot fo work to review it before publish while knowing it don't help at all

Comment 27 Vedran Miletić 2015-01-28 08:34:33 UTC
Jóhann, the error happens with stock cron jobs as well. So it looks like a systemd issue indeed. And it's random (i.e. doesn't happen _every_ time), like Harald said.

I also have good news. I installed systemd 218-3 from F22 on F21 ~16 hours ago and I have not had "Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory" error since. I'm running a machine on DigitalOcean.

Comment 28 Vedran Miletić 2015-01-28 08:52:01 UTC
Update: I see it on workstation machine as well:

Sij 24 09:01:01 clover CROND[11661]: (root) CMD (run-parts /etc/cron.hourly)
Sij 24 09:01:01 clover run-parts[11665]: (/etc/cron.hourly) starting 0anacron
Sij 24 09:01:01 clover run-parts[11671]: (/etc/cron.hourly) finished 0anacron
Sij 24 09:01:01 clover run-parts[11673]: (/etc/cron.hourly) starting mcelog.cron
Sij 24 09:01:01 clover run-parts[11678]: (/etc/cron.hourly) finished mcelog.cron
Sij 24 09:01:01 clover systemd[11659]: Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory

Sij 25 04:01:01 clover kernel: SELinux: initialized (dev tmpfs, type tmpfs), uses transition SIDs
Sij 25 04:01:01 clover CROND[14080]: (root) CMD (run-parts /etc/cron.hourly)
Sij 25 04:01:01 clover systemd[14078]: Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory
Sij 25 04:01:01 clover run-parts[14084]: (/etc/cron.hourly) starting 0anacron
Sij 25 04:01:01 clover run-parts[14090]: (/etc/cron.hourly) finished 0anacron
Sij 25 04:01:01 clover run-parts[14093]: (/etc/cron.hourly) starting mcelog.cron
Sij 25 04:01:01 clover run-parts[14097]: (/etc/cron.hourly) finished mcelog.cron

Before Jan 24, I was running 216-14 and I didn't get that particular error.

Comment 29 Vedran Miletić 2015-01-28 09:00:38 UTC
OK, downgrade from 218-3.fc22 to 216-5.fc21 went nicely (I'm impressed, btw) so let's see if the change happened between -14 and -15 or -15 and -16.

Comment 30 Vedran Miletić 2015-01-28 12:33:52 UTC
Just got this on 216-15:

Sij 28 12:01:01 gedora kernel: SELinux: initialized (dev tmpfs, type tmpfs), use
s transition SIDs
Sij 28 12:01:01 gedora systemd[1]: Starting user-0.slice.
Sij 28 12:01:01 gedora systemd[1]: Created slice user-0.slice.
Sij 28 12:01:01 gedora systemd[1]: Starting Session 25 of user root.
Sij 28 12:01:01 gedora systemd[1]: Started Session 25 of user root.
Sij 28 12:01:01 gedora systemd[1]: Starting User Manager for UID 0...
Sij 28 12:01:01 gedora systemd[3630]: Failed at step CGROUP spawning /usr/lib/sy
stemd/systemd: No such file or directory
Sij 28 12:01:01 gedora systemd[1]: Started User Manager for UID 0.
Sij 28 12:01:01 gedora CROND[3631]: (root) CMD (run-parts /etc/cron.hourly)
Sij 28 12:01:01 gedora run-parts[3634]: (/etc/cron.hourly) starting 0anacron
Sij 28 12:01:01 gedora run-parts[3640]: (/etc/cron.hourly) finished 0anacron
Sij 28 12:01:01 gedora systemd[1]: Stopped User Manager for UID 0.
Sij 28 12:01:01 gedora systemd[1]: Stopping user-0.slice.
Sij 28 12:01:01 gedora systemd[1]: Removed slice user-0.slice.

So the regression happened between -14 and -15.

Comment 31 Tim Waugh 2015-02-09 17:47:28 UTC
I see this too, systemd-216-17.fc21.x86_64. It's pretty intermittent:

$ journalctl MESSAGE_ID=641257651c1b4ec9a8624d7a40a9e1e7
-- Logs begin at Sat 2015-01-10 16:37:07 GMT, end at Mon 2015-02-09 17:45:40 GMT. --
Jan 25 00:43:45 snoopy systemd[4779]: Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory
-- Reboot --
Jan 28 11:01:01 snoopy systemd[1786]: Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory
-- Reboot --
Jan 28 17:22:11 snoopy systemd[2209]: Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory
-- Reboot --
Jan 29 22:01:01 snoopy systemd[2560]: Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory
-- Reboot --
Jan 31 16:01:01 snoopy systemd[3751]: Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory
-- Reboot --
Feb 01 18:01:01 snoopy systemd[2384]: Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory
-- Reboot --
Feb 09 14:01:01 snoopy systemd[2573]: Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory

...but most often seems to happen after crond runs cron.hourly jobs.

Comment 32 Vedran Miletić 2015-02-11 00:42:54 UTC
Tim, yes, cron jobs trigger it most often. But not necessarily.

Comment 33 Jack 2015-02-17 09:01:19 UTC
no journal entries for 7 minutes previous to this
-- snip --

Feb 17 09:01:01 builder2 kernel: SELinux: initialized (dev tmpfs, type tmpfs), uses transition SIDs
Feb 17 09:01:01 builder2 systemd[31058]: Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory
Feb 17 09:01:01 builder2 CROND[31059]: (root) CMD (run-parts /etc/cron.hourly)
Feb 17 09:01:01 builder2 run-parts[31062]: (/etc/cron.hourly) starting 0anacron
Feb 17 09:01:01 builder2 run-parts[31068]: (/etc/cron.hourly) finished 0anacron
Feb 17 09:01:01 builder2 run-parts[31070]: (/etc/cron.hourly) starting mcelog.cron
Feb 17 09:01:01 builder2 run-parts[31074]: (/etc/cron.hourly) finished mcelog.cron

-- snip --
no activity for 13 minutes after that

Comment 34 Harald Reindl 2015-02-17 09:14:21 UTC
no SELinux here - system boots with selinux=0

Comment 35 Nerijus Baliūnas 2015-02-18 21:07:19 UTC
I got it too:
Feb 18 23:02:01 nerijus systemd: Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory

# uname -a
Linux nerijus 3.18.7-200.fc21.x86_64 #1 SMP Wed Feb 11 21:53:17 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

Not a VM. systemd-216-20.fc21.x86_64

Comment 36 Andrew J. Schorr 2015-02-23 14:15:50 UTC
I'm getting it also when a remote user ssh's into the system to run a simple command (but this is not always repeatable):

Feb 23 08:41:01 ti139 systemd[1]: Starting user-360.slice.
Feb 23 08:41:01 ti139 systemd[1]: Created slice user-360.slice.
Feb 23 08:41:01 ti139 systemd[1]: Starting User Manager for UID 360...
Feb 23 08:41:01 ti139 systemd[13080]: Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory
Feb 23 08:41:01 ti139 systemd[1]: Started User Manager for UID 360.
Feb 23 08:41:01 ti139 systemd[1]: Starting Session 113 of user zero.
Feb 23 08:41:01 ti139 systemd[1]: Started Session 113 of user zero.
Feb 23 08:41:01 ti139 systemd[1]: Stopped User Manager for UID 360.
Feb 23 08:41:01 ti139 systemd[1]: Stopping user-360.slice.
Feb 23 08:41:01 ti139 systemd[1]: Removed slice user-360.slice.

This is vanilla F21 just installed this weekend on a physical server.

[root@ti139 ~]# uname -a
Linux ti139 3.18.7-200.fc21.x86_64 #1 SMP Wed Feb 11 21:53:17 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

[root@ti139 ~]# rpm -q systemd
systemd-216-20.fc21.x86_64

SElinux is disabled.

Comment 37 Harald Reindl 2015-02-25 20:03:58 UTC
____________________________________________________________

that's new or at least the first time due reboot after regular updates

Feb 25 20:49:26 testserver systemd: Unit systemd-exit.service entered failed state.
Feb 25 20:49:26 testserver systemd: systemd-exit.service failed

Feb 25 20:43:56 Updated: libcom_err-1.42.12-3.fc21.x86_64
Feb 25 20:43:58 Updated: 2:samba-libs-4.1.17-1.fc21.x86_64
Feb 25 20:43:59 Updated: 2:libwbclient-4.1.17-1.fc21.x86_64
Feb 25 20:43:59 Updated: libcurl-7.37.0-13.fc21.x86_64
Feb 25 20:43:59 Updated: 32:bind-license-9.9.6-8.P1.fc21.noarch
Feb 25 20:44:00 Updated: 32:bind-libs-lite-9.9.6-8.P1.fc21.x86_64
Feb 25 20:44:00 Updated: 32:bind-libs-9.9.6-8.P1.fc21.x86_64
Feb 25 20:44:01 Updated: 2:samba-common-4.1.17-1.fc21.x86_64
Feb 25 20:44:01 Updated: libss-1.42.12-3.fc21.x86_64
Feb 25 20:44:01 Updated: e2fsprogs-libs-1.42.12-3.fc21.x86_64
Feb 25 20:44:02 Updated: freetype-2.5.3-16.fc21.x86_64
Feb 25 20:44:03 Updated: freetype-devel-2.5.3-16.fc21.x86_64
Feb 25 20:44:04 Updated: e2fsprogs-1.42.12-3.fc21.x86_64
Feb 25 20:44:04 Updated: 2:libsmbclient-4.1.17-1.fc21.x86_64
Feb 25 20:44:05 Updated: 32:bind-utils-9.9.6-8.P1.fc21.x86_64
Feb 25 20:44:05 Updated: curl-7.37.0-13.fc21.x86_64
Feb 25 20:44:05 Updated: libcurl-devel-7.37.0-13.fc21.x86_64
Feb 25 20:44:06 Updated: libcom_err-devel-1.42.12-3.fc21.x86_64
Feb 25 20:44:09 Updated: initscripts-9.56.1-7.fc21.x86_64
Feb 25 20:44:09 Updated: perl-Getopt-Long-2.45-1.fc21.noarch
Feb 25 20:44:09 Updated: setup-2.9.0-4.fc21.noarch
Feb 25 20:44:10 Updated: xdg-utils-1.1.0-0.39.rc3.fc21.noarch
____________________________________________________________

Feb 25 18:45:01 testserver systemd: Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory
Feb 25 18:55:01 testserver systemd: Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory
Feb 25 19:00:01 testserver systemd: Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory
Feb 25 19:03:01 testserver systemd: Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory
Feb 25 19:15:01 testserver systemd: Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory
Feb 25 19:30:01 testserver systemd: Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory
Feb 25 19:40:01 testserver systemd: Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory
Feb 25 19:50:01 testserver systemd: Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory
Feb 25 20:01:01 testserver systemd: Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory
Feb 25 20:10:01 testserver systemd: Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory
Feb 25 20:20:01 testserver systemd: Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory
Feb 25 20:30:01 testserver systemd: Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory
Feb 25 20:49:26 testserver systemd: Failed at step CHDIR spawning /usr/bin/kill: Permission denied
Feb 25 20:49:26 testserver systemd: Failed to start Exit the Session.
Feb 25 20:49:26 testserver systemd: Dependency failed for Exit the Session.
Feb 25 20:49:26 testserver systemd: Unit systemd-exit.service entered failed state.
Feb 25 20:49:26 testserver systemd: systemd-exit.service failed.
Feb 25 21:00:27 testserver systemd: Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory

Comment 38 Thomas Müller 2015-03-03 07:02:01 UTC
Same here with out-of-the-box stock cron jobs...

Mär 03 04:01:01 nuc systemd[18183]: pam_unix(systemd-user:session): session closed for user root
Mär 03 05:01:01 nuc kernel: SELinux: initialized (dev tmpfs, type tmpfs), uses transition SIDs
Mär 03 05:01:01 nuc systemd[18295]: Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory
Mär 03 05:01:01 nuc CROND[18296]: (root) CMD (run-parts /etc/cron.hourly)
Mär 03 05:01:01 nuc run-parts[18299]: (/etc/cron.hourly) starting 0anacron
Mär 03 05:01:01 nuc run-parts[18305]: (/etc/cron.hourly) finished 0anacron
Mär 03 05:01:01 nuc run-parts[18307]: (/etc/cron.hourly) starting mcelog.cron
Mär 03 05:01:01 nuc run-parts[18311]: (/etc/cron.hourly) finished mcelog.cron
Mär 03 06:01:01 nuc kernel: SELinux: initialized (dev tmpfs, type tmpfs), uses transition SIDs
Mär 03 06:01:01 nuc systemd[18404]: pam_unix(systemd-user:session): session opened for user root by (uid=0)


Clean install of F21 on a physical system.

[root@nuc ~]# uname -a
Linux nuc 3.18.7-200.fc21.x86_64 #1 SMP Wed Feb 11 21:53:17 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

[root@nuc ~]# rpm -q systemd
systemd-216-20.fc21.x86_64

Comment 39 Petr Spacek 2015-03-21 17:35:43 UTC
I confirm that this happens on F21 with systemd-216-21.fc21.x86_64.

This is clean installation (no upgrades) on bare metal.

Comment 40 Harald Reindl 2015-03-24 16:18:45 UTC
that happens on *any* Fedora 21 setup and nobody of the developers can tell me they won't see if they just read their logs, especially given that 6 different people are confirming this over 2 months now

Comment 41 Michal Schmidt 2015-03-26 14:44:46 UTC
On my F21 system "journalctl MESSAGE_ID=641257651c1b4ec9a8624d7a40a9e1e7" shows me only three log entries indicating this bug (on Feb 04, 10, and 25).
Do not assume this is easy to reproduce for the developers.

Comment 42 Harald Reindl 2015-03-26 15:36:14 UTC
try to setup a few cronjobs per minute calling a simple bash script doing a "sleep 1" and it will get visible, until now i hvae not faced a single Fedora 21 setup (virtual and physical) without that bug all day long

and to be honest "Failed at step CGROUP spawning" sounds terrible

Mar 26 12:40:01 srv-rhsoft systemd: Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory
Mar 26 13:00:01 srv-rhsoft systemd: Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory
Mar 26 13:00:01 srv-rhsoft systemd: Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory
Mar 26 13:02:01 srv-rhsoft systemd: Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory
Mar 26 13:20:01 srv-rhsoft systemd: Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory
Mar 26 13:40:01 srv-rhsoft systemd: Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory
Mar 26 14:00:01 srv-rhsoft systemd: Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory
Mar 26 14:02:01 srv-rhsoft systemd: Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory
Mar 26 14:20:01 srv-rhsoft systemd: Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory
Mar 26 14:40:01 srv-rhsoft systemd: Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory
Mar 26 15:00:01 srv-rhsoft systemd: Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory
Mar 26 15:02:01 srv-rhsoft systemd: Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory
Mar 26 15:20:01 srv-rhsoft systemd: Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory
Mar 26 15:40:01 srv-rhsoft systemd: Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory
Mar 26 16:01:01 srv-rhsoft systemd: Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory
Mar 26 16:20:01 srv-rhsoft systemd: Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory

Comment 43 Thomas Müller 2015-03-28 07:59:46 UTC
(In reply to Michal Schmidt from comment #41)
> Do not assume this is easy to reproduce for the developers.
It is not about this being easy to reproduce, but that there appears to be no indication that the maintainer even accept this as a valid problem.

(In reply to Jóhann B. Guðmundsson from comment #25)
> Apparently it's to hard for you to do what you are asked. 
> 
> For example this could be because a spesific user is running the cron job (
> or not ) that triggers this and relates to PAM.
> 
> In anycase I got better things to do with my time let's see if Zbigniew can
> better deal with you and proceed further in trying to find the root cause
> for this.
It is nice that Jóhann apparently works for an employer that has no problem with internal code being posted in public. Expecting this to be the case for everyone is pretty arrogant.

In any case, several users (including myself) have commented that this happens even for the most basic cron jobs on a new and clean installation, basically ruling out any local misconfiguration or weird interaction with exotic tools or scripts.

Still, there is no indication whatsoever that indicates that someone is actually looking into this. (Probably better things to do)

Comment 44 Michal Schmidt 2015-03-30 08:45:27 UTC
(In reply to Thomas Müller from comment #43)
> It is not about this being easy to reproduce, but that there appears to be
> no indication that the maintainer even accept this as a valid problem.

I accept this is a valid problem. (Otherwise I would have already closed it as NOTABUG.)
We do not tend to post comments that merely acknowledge the validity of bugs, unless we already have some insight to share.

> Still, there is no indication whatsoever that indicates that someone is
> actually looking into this. (Probably better things to do)

I have been looking into this.
I don't know what makes the occurrence of the bug so common for some users, while it's so rare for others. With a cron job scheduled to run every minute, it still took hours to reproduce for me. Thankfully I was able to find faster reproducer. I'm running a service like this:

[Service]
ExecStart=/bin/sh -c 'while :; do su - root -c "sleep 1"; done'

This takes a few minutes to trigger the bug.

Based on my findings so far, the problem seems to be that sometimes when recursively trimming cgroups of user-0.slice, the cgroup .../user-0.slice/user@0.service is still present and gets removed too, making the service's 'cgroup_realized' attribute out of sync with reality.

Comment 45 Jóhann B. Guðmundsson 2015-03-30 09:23:47 UTC
Michal you might want to have a look at and share your findings with David since he had been trying to narrow down this ( race ) trigger in upstream bug 89145

Comment 46 Harald Reindl 2015-04-19 20:36:50 UTC
any news here?
looks like this also affcets F22

for a highly critical component like systemd bugs visible by just read regular syslog happen way too often and stay way too long

Apr 19 22:25:37 rawhide systemd: Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory

[root@rawhide ~]# rpm -q systemd
systemd-219-11.fc22.x86_64

Comment 47 Vedran Miletić 2015-04-19 21:03:10 UTC
Indeed it does. But it's much rarer.

Comment 48 Harald Reindl 2015-04-19 21:04:45 UTC
true - on F22 it happens not that often while on F21 i see that messages all day long on any machine virtual as well as physical

Comment 49 David Rheinsberg 2015-09-23 09:04:02 UTC
Can someone tell me how to reproduce this on -git? Running Michal's unit triggers a `systemd --user` restart every second (and I fixed several unrelated bugs now), but I could not trigger the CGROUP issue. I also played with some explicit preemption points, but it seems I can no longer trigger it after the cgroup-rewrite in systemd-225.

Comment 50 Fedora End Of Life 2015-11-04 10:27:28 UTC
This message is a reminder that Fedora 21 is nearing its end of life.
Approximately 4 (four) weeks from now Fedora will stop maintaining
and issuing updates for Fedora 21. It is Fedora's policy to close all
bug reports from releases that are no longer maintained. At that time
this bug will be closed as EOL if it remains open with a Fedora  'version'
of '21'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version.

Thank you for reporting this issue and we are sorry that we were not 
able to fix it before Fedora 21 is end of life. If you would still like 
to see this bug fixed and are able to reproduce it against a later version 
of Fedora, you are encouraged  change the 'version' to a later Fedora 
version prior this bug is closed as described in the policy above.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events. Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.

Comment 51 Fedora End Of Life 2015-12-02 07:53:24 UTC
Fedora 21 changed to end-of-life (EOL) status on 2015-12-01. Fedora 21 is
no longer maintained, which means that it will not receive any further
security or bug fix updates. As a result we are closing this bug.

If you can reproduce this bug against a currently maintained version of
Fedora please feel free to reopen this bug against that version. If you
are unable to reopen this bug, please file a new report against the
current release. If you experience problems, please add a comment to this
bug.

Thank you for reporting this bug and we are sorry it could not be fixed.

Comment 52 Dominik 'Rathann' Mierzejewski 2016-10-23 20:13:11 UTC
Still happening on Fedora 24 when running munin cron job intermittently:
[...]
Oct 23 15:55:01 amaterasu.greysector.net systemd[3033]: user@383.service: Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory
Oct 23 17:40:01 amaterasu.greysector.net systemd[20066]: user@383.service: Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory
Oct 23 19:45:01 amaterasu.greysector.net systemd[7980]: user@383.service: Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory

$ id munin
uid=383(munin) gid=381(munin) groups=381(munin)
$ rpm -qf /etc/cron.d/munin 
munin-2.0.26-1.fc24.noarch
$ rpm -q systemd
systemd-229-16.fc24.x86_64

Comment 53 Fedora End Of Life 2017-07-25 18:48:43 UTC
This message is a reminder that Fedora 24 is nearing its end of life.
Approximately 2 (two) weeks from now Fedora will stop maintaining
and issuing updates for Fedora 24. It is Fedora's policy to close all
bug reports from releases that are no longer maintained. At that time
this bug will be closed as EOL if it remains open with a Fedora  'version'
of '24'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version'
to a later Fedora version.

Thank you for reporting this issue and we are sorry that we were not
able to fix it before Fedora 24 is end of life. If you would still like
to see this bug fixed and are able to reproduce it against a later version
of Fedora, you are encouraged  change the 'version' to a later Fedora
version prior this bug is closed as described in the policy above.

Although we aim to fix as many bugs as possible during every release's
lifetime, sometimes those efforts are overtaken by events. Often a
more recent Fedora release includes newer upstream software that fixes
bugs or makes them obsolete.

Comment 54 Fedora End Of Life 2017-08-08 11:54:15 UTC
Fedora 24 changed to end-of-life (EOL) status on 2017-08-08. Fedora 24 is
no longer maintained, which means that it will not receive any further
security or bug fix updates. As a result we are closing this bug.

If you can reproduce this bug against a currently maintained version of
Fedora please feel free to reopen this bug against that version. If you
are unable to reopen this bug, please file a new report against the
current release. If you experience problems, please add a comment to this
bug.

Thank you for reporting this bug and we are sorry it could not be fixed.


Note You need to log in before you can comment on or make changes to this bug.