Is this with the latest systemd, or the one from F21 installation media?

Can you attach the full boot log?

systemd-216-17.fc21.x86_64

was also there with the latest build(s) and i upgraded from munal koji download before wrote the bugreport, dracut -f also called after update

Created attachment 983407 [details]
dmesg and /var/log/messages after reboot

keep in mind that is the same environment as https://bugzilla.redhat.com/show_bug.cgi?id=1184016 if you look for something in that case

OK, this is an issue I can reproduce, not bug 1184016. Sorry for bugspam.

Please attache the output from journalctl -b and confirm that your are not running an custom kernel

Created attachment 984842 [details]
output of journalctl

output attached, happens again around the *useless* log-flood of *useless* usermanager for ordinary cronjobs which worked decades before systemd and "user-manager"

hopefully somebody re-considers all that bload for simple tasks....

(In reply to Vedran Miletić from comment #5)
> OK, this is an issue I can reproduce, not bug 1184016. Sorry for bugspam.

Please attache the output from journalctl -b and confirm that your are not running an custom kernel

(In reply to Harald Reindl from comment #7)
> Created attachment 984842 [details]
> output of journalctl
> 
> output attached, happens again around the *useless* log-flood of *useless*
> usermanager for ordinary cronjobs which worked decades before systemd and
> "user-manager"
> 
> hopefully somebody re-considers all that bload for simple tasks....

I asked you for the output of journalctl -b and there is not a single kernel entry in that journal. 

Please provide the exact output of the journal which contains the complete bootlog as Zbigniew had already asked for in comment two and leave this rant of yours at home.

> I asked you for the output of journalctl -b and there 
> is not a single kernel entry in that journal

so what - a full bootlog is already attached for days now and i don't use journald for persistent storage - that won't change

> Please provide the exact output of the journal which contains 
> the complete bootlog as Zbigniew had already asked for

what about look at the attachment i made 2 hours after that request days ago?
https://bugzilla.redhat.com/attachment.cgi?id=983407

> and leave this rant of yours at home

given the ongoing troubles with log-flooding, still persistent bugs from F20 and the complete failing of systemd in F21 if someone is so crazy using systemd features my rant is more than moderate

I also asked you to confirm you are not running custom kernel and you did not answer that.

no - otherwise i would not write bugreports for fedora core components

[root@testserver:~]$ uname -r
3.18.3-201.fc21.x86_64

[root@testserver:~]$ cat yum.log | grep kernel
Jan 22 01:18:36 Installed: kernel-core-3.18.3-201.fc21.x86_64
Jan 22 01:18:49 Updated: kernel-headers-3.18.3-201.fc21.x86_64
Jan 22 01:40:38 Erased: kernel-core-3.18.2-200.fc21.x86_64

Enable persistent journal reboot and provide the full output from the journal and confirm you are not running this in containers or as virtual image 

Based on the hostname of the machine in the journal log your provide that should not be a problem since it's labeled as a test machine

this is a VMNware guest running from Fedora 9 to Fedora 20, recently upgraded to F21, since our whole production is running on top of VMware vSphere and before systemd upgrade of F21 it *must not* matter if this is a virtual machine - until F21 is not running smooth and without namespace troubles (see other bugreport) i even don't consider touch any of my physical hosts with it

persistent journal - somwhere around tomorrow, there is currently running a long DBMail stress test against MariaDB 10

It should not matter if it's VMWare but is anyone experiencing this running Fedora in VMware or running custom kernel which has not cgroups enable ( which usally triggers this error ) ? 

The other issue might be that your cron jobs are doing something weird or you have a wierd setup which locks them out from doing what they are supposed to do

You need to attach your cron script that is being run as apache here...

Jan 27 08:00:01 testserver.rhsoft.net systemd[1]: Starting User Manager for UID 48...
Jan 27 08:00:01 testserver.rhsoft.net systemd[4881]: Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory
Jan 27 08:00:01 testserver.rhsoft.net systemd[1]: Started User Manager for UID 48.
Jan 27 08:00:01 testserver.rhsoft.net systemd[1]: Starting Session 133 of user apache.
Jan 27 08:00:01 testserver.rhsoft.net systemd[1]: Started Session 133 of user apache.
Jan 27 08:00:01 testserver.rhsoft.net systemd[1]: Starting Session 134 of user root.
Jan 27 08:00:01 testserver.rhsoft.net systemd[1]: Started Session 134 of user root.
Jan 27 08:00:01 testserver.rhsoft.net CROND[4894]: (root) CMD (     nice php /scripts/dbmail-snapshot/build.php)
Jan 27 08:00:01 testserver.rhsoft.net CROND[4901]: (root) CMD (     nice -n 19 ionice -c 3 php /scripts/check-uploadtemp.php)
Jan 27 08:00:01 testserver.rhsoft.net CROND[4905]: (root) CMD (     nice php /usr/local/scripts/rh_watchdog.php)
Jan 27 08:00:01 testserver.rhsoft.net CROND[4902]: (apache) CMD (   nice -n 19 ionice -c 3 dash /scripts/cleanup-uploadtemp.cron)
Jan 27 08:00:01 testserver.rhsoft.net CROND[4906]: (root) CMD (     /usr/bin/rpm -q dbmail > /www/mailadmin/modules/dbmail-snapshot.txt)
Jan 27 08:00:01 testserver.rhsoft.net systemd[1]: Stopped User Manager for UID 48.
Jan 27 08:00:01 testserver.rhsoft.net systemd[1]: Stopping user-48.slice.

Same thing about this one

Jan 27 08:03:01 testserver.rhsoft.net systemd[5009]: Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory
Jan 27 08:03:01 testserver.rhsoft.net systemd[1]: Started User Manager for UID 0.
Jan 27 08:03:01 testserver.rhsoft.net systemd[1]: Starting Session 136 of user root.
Jan 27 08:03:01 testserver.rhsoft.net systemd[1]: Started Session 136 of user root.
Jan 27 08:03:01 testserver.rhsoft.net CROND[5014]: (root) CMD (     nice -n 19 ionice -c 3 php /scripts/check-uploadtemp.php)
Jan 27 08:03:01 testserver.rhsoft.net systemd[1]: Stopped User Manager for UID 0.

This one runs fine

Jan 27 08:05:01 testserver.rhsoft.net systemd[1]: Started User Manager for UID 0.
Jan 27 08:05:01 testserver.rhsoft.net CROND[5058]: (root) CMD (     nice php /scripts/dbmail-snapshot/build.php)
Jan 27 08:05:01 testserver.rhsoft.net CROND[5059]: (root) CMD (     /usr/bin/rpm -q dbmail > /www/mailadmin/modules/dbmail-snapshot.txt)
Jan 27 08:05:01 testserver.rhsoft.net systemd[1]: Stopping User Manager for UID 0...

So what's the difference between those cron jobs $PATH issues maybe?

the cronjob running as "apache" is as simple as a cronjob can be, that's just a replacement for manage expired session-files in random webserver-workers and also remove remaining old temp-uploads in the previuos shared folder

*/30 * * * *  apache    nice -n 19 ionice -c 3 dash /scripts/cleanup-uploadtemp.cron

#!/usr/bin/dash
find /var/www/uploadtemp -type f -mmin +60 -delete
find /var/www/sessiondata -type f -mmin +45 -delete
__________________________________________________________________

/scripts/check-uploadtemp.php does nothing else than read the apache-vhost-configs, parse out the document root and make sure under each docroot a subfolder "uploadtemp" exists with the correct permissions

the reason behind is that our cms-installer refuses to rsync the template system if the document root is not empty and the script create vhosts may also create ones not became a cms-setup (in both cases the vhost and it's config is created with the same script-call)
__________________________________________________________________

/scripts/dbmail-snapshot/build.php just looks if there is a new upstream-coomit ID is stored in a database, fetch the tarball in that case, fires up the rpm-build and restarts dbmail services
__________________________________________________________________

$PATH issues are unlikely, that below is on top of /etc/crontab and all cronjobs are inside /etc/crontab

SHELL=/usr/bin/bash
PATH=/usr/bin:/usr/sbin
LANG=en_GB.UTF-8
MAILTO=root
HOME=/

One is set to bash while the cron job runs as dash ( #!/usr/bin/dash ) try changing that to "bash" and see if that fixes the issue

the dash one is the simple posted above

don't get me wrong but with that logic since bash is the default shell, it would not be possible to run crojobs with /usr/bin/perl, /usr/bin/php, /usr/bin/python or whatever interpreter in a clean way, others are running and it's happening randomly

if that would be the reason it would happen every 30 minutes, the machine has a uptime of 18 hours (23:11:17 up 18:56,  1 user,  load average: 0,11, 0,19, 0,21) and the journal don't have entries for each call

[root@testserver:~]$ journalctl | grep "spawning /usr/lib/systemd/systemd"
Jan 27 12:45:01 testserver.rhsoft.net systemd[9346]: Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory
Jan 27 13:00:01 testserver.rhsoft.net systemd[9535]: Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory
Jan 27 13:05:01 testserver.rhsoft.net systemd[9718]: Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory
Jan 27 13:30:01 testserver.rhsoft.net systemd[10058]: Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory
Jan 27 14:01:01 testserver.rhsoft.net systemd[10585]: Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory
Jan 27 14:20:01 testserver.rhsoft.net systemd[10871]: Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory
Jan 27 14:40:01 testserver.rhsoft.net systemd[11162]: Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory
Jan 27 15:00:01 testserver.rhsoft.net systemd[11733]: Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory
Jan 27 17:00:01 testserver.rhsoft.net systemd[13269]: Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory
Jan 27 18:03:01 testserver.rhsoft.net systemd[14196]: Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory
Jan 27 18:03:46 testserver.rhsoft.net systemd[14338]: Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory
Jan 27 18:03:47 testserver.rhsoft.net systemd[14523]: Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory
Jan 27 19:00:02 testserver.rhsoft.net systemd[31202]: Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory
Jan 27 19:15:01 testserver.rhsoft.net systemd[5079]: Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory
Jan 27 21:30:01 testserver.rhsoft.net systemd[8480]: Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory
Jan 27 22:30:01 testserver.rhsoft.net systemd[15933]: Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory

I did not ask you to spew nonsense into the bug report I asked you to run the cron jobs that triggered this as bash( the apache job you refered to )  and see if it triggers it. 

Your Dash setup could be broken, your php setup could be broken etc there is a difference in your cron jobs as I mentioned in comment 16

Some run without problems some do not and trigger this so surely you can use those administrative skills of yours change the cron job that triggered this to bash and reduce that particular cron job to a minute to see if it happens.

besides i did now change the one to bash and run every minute, don't call others to spew nonsense until you re-think about about your assumptions

if my PHP setup would be broken the other cronjobs running every 5 minutes would trigger the same error and if my bash setup would be broken where where the problem  at 15:30, 16:00, and 17:30

Jan 27 15:00:01 testserver.rhsoft.net systemd[11733]: Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory
Jan 27 17:00:01 testserver.rhsoft.net systemd[13269]: Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory
Jan 27 18:03:01 testserver.rhsoft.net systemd[14196]: Failed at step CGROUP

and EVEN IF my php and dash setup would be broken (broken in what way) systemd would have a massive problem if it's cgroups-controller got shoot in the foot by a userland script

that's the result of running the small shell-script with bash instead das every minute, as you can see the error is triggered only randomly

Jan 27 23:35:01 testserver systemd: Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory
Jan 27 23:38:01 testserver systemd: Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory
Jan 27 23:41:01 testserver systemd: Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory

Jan 27 23:40:01 testserver CROND[18543]: (apache) CMD (   nice -n 19 ionice -c 3 bash /scripts/cleanup-uploadtemp.cron)
Jan 27 23:41:01 testserver CROND[18607]: (apache) CMD (   nice -n 19 ionice -c 3 bash /scripts/cleanup-uploadtemp.cron)
Jan 27 23:42:01 testserver CROND[18649]: (apache) CMD (   nice -n 19 ionice -c 3 bash /scripts/cleanup-uploadtemp.cron)
Jan 27 23:43:01 testserver CROND[18704]: (apache) CMD (   nice -n 19 ionice -c 3 bash /scripts/cleanup-uploadtemp.cron)
Jan 27 23:44:01 testserver CROND[18758]: (apache) CMD (   nice -n 19 ionice -c 3 bash /scripts/cleanup-uploadtemp.cron)

Jan 27 23:41:01 testserver.rhsoft.net systemd[1]: Starting user-48.slice.
Jan 27 23:41:01 testserver.rhsoft.net systemd[1]: Created slice user-48.slice.
Jan 27 23:41:01 testserver.rhsoft.net systemd[1]: Starting User Manager for UID 48...
Jan 27 23:41:01 testserver.rhsoft.net systemd[18602]: Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory
Jan 27 23:41:01 testserver.rhsoft.net systemd[1]: Started User Manager for UID 48.
Jan 27 23:41:01 testserver.rhsoft.net systemd[1]: Starting Session 712 of user apache.
Jan 27 23:41:01 testserver.rhsoft.net systemd[1]: Started Session 712 of user apache.
Jan 27 23:41:01 testserver.rhsoft.net CROND[18607]: (apache) CMD (   nice -n 19 ionice -c 3 bash /scripts/cleanup-uploadtemp.cron)
Jan 27 23:41:01 testserver.rhsoft.net systemd[1]: Stopped User Manager for UID 48.
Jan 27 23:41:01 testserver.rhsoft.net systemd[1]: Stopping user-48.slice.
Jan 27 23:41:01 testserver.rhsoft.net systemd[1]: Removed slice user-48.slice.

Then we can rule out dash and it's $PATH/Environment as the underlying problem which just leaves potential PAM problems. 

Can you paste all the cron jobs running comment 16. 

They seem to be similar in nature

/scripts/check-uploadtemp.php from #16 depends on a lot of libraries and /scripts/dbmail-snapshot/build.php too - i doubt they are similar in nature given that a simple dash or bash 2-liner has the same behavior

since we ruled out one script and it's environment the scripts itself are completly innocent and keep in mind that it don't happen at every run

if you look at these two links you see SYSTEMD has a major problem and not single scripts independent in which language they are written

https://bugzilla.redhat.com/show_bug.cgi?id=1184016
https://bugzilla.redhat.com/show_bug.cgi?id=1184016#c7

Apparently it's to hard for you to do what you are asked. 

For example this could be because a spesific user is running the cron job ( or not ) that triggers this and relates to PAM.

In anycase I got better things to do with my time let's see if Zbigniew can better deal with you and proceed further in trying to find the root cause for this.

it's not a matter of "hard for you to" - it just makes no sense to paste thousands of line wired PHP code not simple runable on a different environment and containing sensible code 

since it also affects the *default apache user* running a 2-liner shell script with not more than two find-calls which i already posted so why do you need *thousands of lines php code* instead focus on that 2 liner?

so keep your polemic "I got better things to do" for yourself when it is pretty clear that you are asking for nonsense meaning a lot fo work to review it before publish while knowing it don't help at all

Jóhann, the error happens with stock cron jobs as well. So it looks like a systemd issue indeed. And it's random (i.e. doesn't happen _every_ time), like Harald said.

I also have good news. I installed systemd 218-3 from F22 on F21 ~16 hours ago and I have not had "Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory" error since. I'm running a machine on DigitalOcean.

Update: I see it on workstation machine as well:

Sij 24 09:01:01 clover CROND[11661]: (root) CMD (run-parts /etc/cron.hourly)
Sij 24 09:01:01 clover run-parts[11665]: (/etc/cron.hourly) starting 0anacron
Sij 24 09:01:01 clover run-parts[11671]: (/etc/cron.hourly) finished 0anacron
Sij 24 09:01:01 clover run-parts[11673]: (/etc/cron.hourly) starting mcelog.cron
Sij 24 09:01:01 clover run-parts[11678]: (/etc/cron.hourly) finished mcelog.cron
Sij 24 09:01:01 clover systemd[11659]: Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory

Sij 25 04:01:01 clover kernel: SELinux: initialized (dev tmpfs, type tmpfs), uses transition SIDs
Sij 25 04:01:01 clover CROND[14080]: (root) CMD (run-parts /etc/cron.hourly)
Sij 25 04:01:01 clover systemd[14078]: Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory
Sij 25 04:01:01 clover run-parts[14084]: (/etc/cron.hourly) starting 0anacron
Sij 25 04:01:01 clover run-parts[14090]: (/etc/cron.hourly) finished 0anacron
Sij 25 04:01:01 clover run-parts[14093]: (/etc/cron.hourly) starting mcelog.cron
Sij 25 04:01:01 clover run-parts[14097]: (/etc/cron.hourly) finished mcelog.cron

Before Jan 24, I was running 216-14 and I didn't get that particular error.

OK, downgrade from 218-3.fc22 to 216-5.fc21 went nicely (I'm impressed, btw) so let's see if the change happened between -14 and -15 or -15 and -16.

Just got this on 216-15:

Sij 28 12:01:01 gedora kernel: SELinux: initialized (dev tmpfs, type tmpfs), use
s transition SIDs
Sij 28 12:01:01 gedora systemd[1]: Starting user-0.slice.
Sij 28 12:01:01 gedora systemd[1]: Created slice user-0.slice.
Sij 28 12:01:01 gedora systemd[1]: Starting Session 25 of user root.
Sij 28 12:01:01 gedora systemd[1]: Started Session 25 of user root.
Sij 28 12:01:01 gedora systemd[1]: Starting User Manager for UID 0...
Sij 28 12:01:01 gedora systemd[3630]: Failed at step CGROUP spawning /usr/lib/sy
stemd/systemd: No such file or directory
Sij 28 12:01:01 gedora systemd[1]: Started User Manager for UID 0.
Sij 28 12:01:01 gedora CROND[3631]: (root) CMD (run-parts /etc/cron.hourly)
Sij 28 12:01:01 gedora run-parts[3634]: (/etc/cron.hourly) starting 0anacron
Sij 28 12:01:01 gedora run-parts[3640]: (/etc/cron.hourly) finished 0anacron
Sij 28 12:01:01 gedora systemd[1]: Stopped User Manager for UID 0.
Sij 28 12:01:01 gedora systemd[1]: Stopping user-0.slice.
Sij 28 12:01:01 gedora systemd[1]: Removed slice user-0.slice.

So the regression happened between -14 and -15.

I see this too, systemd-216-17.fc21.x86_64. It's pretty intermittent:

$ journalctl MESSAGE_ID=641257651c1b4ec9a8624d7a40a9e1e7
-- Logs begin at Sat 2015-01-10 16:37:07 GMT, end at Mon 2015-02-09 17:45:40 GMT. --
Jan 25 00:43:45 snoopy systemd[4779]: Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory
-- Reboot --
Jan 28 11:01:01 snoopy systemd[1786]: Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory
-- Reboot --
Jan 28 17:22:11 snoopy systemd[2209]: Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory
-- Reboot --
Jan 29 22:01:01 snoopy systemd[2560]: Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory
-- Reboot --
Jan 31 16:01:01 snoopy systemd[3751]: Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory
-- Reboot --
Feb 01 18:01:01 snoopy systemd[2384]: Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory
-- Reboot --
Feb 09 14:01:01 snoopy systemd[2573]: Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory

...but most often seems to happen after crond runs cron.hourly jobs.

Tim, yes, cron jobs trigger it most often. But not necessarily.

no journal entries for 7 minutes previous to this
-- snip --

Feb 17 09:01:01 builder2 kernel: SELinux: initialized (dev tmpfs, type tmpfs), uses transition SIDs
Feb 17 09:01:01 builder2 systemd[31058]: Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory
Feb 17 09:01:01 builder2 CROND[31059]: (root) CMD (run-parts /etc/cron.hourly)
Feb 17 09:01:01 builder2 run-parts[31062]: (/etc/cron.hourly) starting 0anacron
Feb 17 09:01:01 builder2 run-parts[31068]: (/etc/cron.hourly) finished 0anacron
Feb 17 09:01:01 builder2 run-parts[31070]: (/etc/cron.hourly) starting mcelog.cron
Feb 17 09:01:01 builder2 run-parts[31074]: (/etc/cron.hourly) finished mcelog.cron

-- snip --
no activity for 13 minutes after that

no SELinux here - system boots with selinux=0

I got it too:
Feb 18 23:02:01 nerijus systemd: Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory

# uname -a
Linux nerijus 3.18.7-200.fc21.x86_64 #1 SMP Wed Feb 11 21:53:17 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

Not a VM. systemd-216-20.fc21.x86_64

I'm getting it also when a remote user ssh's into the system to run a simple command (but this is not always repeatable):

Feb 23 08:41:01 ti139 systemd[1]: Starting user-360.slice.
Feb 23 08:41:01 ti139 systemd[1]: Created slice user-360.slice.
Feb 23 08:41:01 ti139 systemd[1]: Starting User Manager for UID 360...
Feb 23 08:41:01 ti139 systemd[13080]: Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory
Feb 23 08:41:01 ti139 systemd[1]: Started User Manager for UID 360.
Feb 23 08:41:01 ti139 systemd[1]: Starting Session 113 of user zero.
Feb 23 08:41:01 ti139 systemd[1]: Started Session 113 of user zero.
Feb 23 08:41:01 ti139 systemd[1]: Stopped User Manager for UID 360.
Feb 23 08:41:01 ti139 systemd[1]: Stopping user-360.slice.
Feb 23 08:41:01 ti139 systemd[1]: Removed slice user-360.slice.

This is vanilla F21 just installed this weekend on a physical server.

[root@ti139 ~]# uname -a
Linux ti139 3.18.7-200.fc21.x86_64 #1 SMP Wed Feb 11 21:53:17 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

[root@ti139 ~]# rpm -q systemd
systemd-216-20.fc21.x86_64

SElinux is disabled.

____________________________________________________________

that's new or at least the first time due reboot after regular updates

Feb 25 20:49:26 testserver systemd: Unit systemd-exit.service entered failed state.
Feb 25 20:49:26 testserver systemd: systemd-exit.service failed

Feb 25 20:43:56 Updated: libcom_err-1.42.12-3.fc21.x86_64
Feb 25 20:43:58 Updated: 2:samba-libs-4.1.17-1.fc21.x86_64
Feb 25 20:43:59 Updated: 2:libwbclient-4.1.17-1.fc21.x86_64
Feb 25 20:43:59 Updated: libcurl-7.37.0-13.fc21.x86_64
Feb 25 20:43:59 Updated: 32:bind-license-9.9.6-8.P1.fc21.noarch
Feb 25 20:44:00 Updated: 32:bind-libs-lite-9.9.6-8.P1.fc21.x86_64
Feb 25 20:44:00 Updated: 32:bind-libs-9.9.6-8.P1.fc21.x86_64
Feb 25 20:44:01 Updated: 2:samba-common-4.1.17-1.fc21.x86_64
Feb 25 20:44:01 Updated: libss-1.42.12-3.fc21.x86_64
Feb 25 20:44:01 Updated: e2fsprogs-libs-1.42.12-3.fc21.x86_64
Feb 25 20:44:02 Updated: freetype-2.5.3-16.fc21.x86_64
Feb 25 20:44:03 Updated: freetype-devel-2.5.3-16.fc21.x86_64
Feb 25 20:44:04 Updated: e2fsprogs-1.42.12-3.fc21.x86_64
Feb 25 20:44:04 Updated: 2:libsmbclient-4.1.17-1.fc21.x86_64
Feb 25 20:44:05 Updated: 32:bind-utils-9.9.6-8.P1.fc21.x86_64
Feb 25 20:44:05 Updated: curl-7.37.0-13.fc21.x86_64
Feb 25 20:44:05 Updated: libcurl-devel-7.37.0-13.fc21.x86_64
Feb 25 20:44:06 Updated: libcom_err-devel-1.42.12-3.fc21.x86_64
Feb 25 20:44:09 Updated: initscripts-9.56.1-7.fc21.x86_64
Feb 25 20:44:09 Updated: perl-Getopt-Long-2.45-1.fc21.noarch
Feb 25 20:44:09 Updated: setup-2.9.0-4.fc21.noarch
Feb 25 20:44:10 Updated: xdg-utils-1.1.0-0.39.rc3.fc21.noarch
____________________________________________________________

Feb 25 18:45:01 testserver systemd: Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory
Feb 25 18:55:01 testserver systemd: Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory
Feb 25 19:00:01 testserver systemd: Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory
Feb 25 19:03:01 testserver systemd: Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory
Feb 25 19:15:01 testserver systemd: Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory
Feb 25 19:30:01 testserver systemd: Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory
Feb 25 19:40:01 testserver systemd: Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory
Feb 25 19:50:01 testserver systemd: Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory
Feb 25 20:01:01 testserver systemd: Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory
Feb 25 20:10:01 testserver systemd: Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory
Feb 25 20:20:01 testserver systemd: Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory
Feb 25 20:30:01 testserver systemd: Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory
Feb 25 20:49:26 testserver systemd: Failed at step CHDIR spawning /usr/bin/kill: Permission denied
Feb 25 20:49:26 testserver systemd: Failed to start Exit the Session.
Feb 25 20:49:26 testserver systemd: Dependency failed for Exit the Session.
Feb 25 20:49:26 testserver systemd: Unit systemd-exit.service entered failed state.
Feb 25 20:49:26 testserver systemd: systemd-exit.service failed.
Feb 25 21:00:27 testserver systemd: Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory

Same here with out-of-the-box stock cron jobs...

Mär 03 04:01:01 nuc systemd[18183]: pam_unix(systemd-user:session): session closed for user root
Mär 03 05:01:01 nuc kernel: SELinux: initialized (dev tmpfs, type tmpfs), uses transition SIDs
Mär 03 05:01:01 nuc systemd[18295]: Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory
Mär 03 05:01:01 nuc CROND[18296]: (root) CMD (run-parts /etc/cron.hourly)
Mär 03 05:01:01 nuc run-parts[18299]: (/etc/cron.hourly) starting 0anacron
Mär 03 05:01:01 nuc run-parts[18305]: (/etc/cron.hourly) finished 0anacron
Mär 03 05:01:01 nuc run-parts[18307]: (/etc/cron.hourly) starting mcelog.cron
Mär 03 05:01:01 nuc run-parts[18311]: (/etc/cron.hourly) finished mcelog.cron
Mär 03 06:01:01 nuc kernel: SELinux: initialized (dev tmpfs, type tmpfs), uses transition SIDs
Mär 03 06:01:01 nuc systemd[18404]: pam_unix(systemd-user:session): session opened for user root by (uid=0)


Clean install of F21 on a physical system.

[root@nuc ~]# uname -a
Linux nuc 3.18.7-200.fc21.x86_64 #1 SMP Wed Feb 11 21:53:17 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

[root@nuc ~]# rpm -q systemd
systemd-216-20.fc21.x86_64

I confirm that this happens on F21 with systemd-216-21.fc21.x86_64.

This is clean installation (no upgrades) on bare metal.

that happens on *any* Fedora 21 setup and nobody of the developers can tell me they won't see if they just read their logs, especially given that 6 different people are confirming this over 2 months now

On my F21 system "journalctl MESSAGE_ID=641257651c1b4ec9a8624d7a40a9e1e7" shows me only three log entries indicating this bug (on Feb 04, 10, and 25).
Do not assume this is easy to reproduce for the developers.

try to setup a few cronjobs per minute calling a simple bash script doing a "sleep 1" and it will get visible, until now i hvae not faced a single Fedora 21 setup (virtual and physical) without that bug all day long

and to be honest "Failed at step CGROUP spawning" sounds terrible

Mar 26 12:40:01 srv-rhsoft systemd: Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory
Mar 26 13:00:01 srv-rhsoft systemd: Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory
Mar 26 13:00:01 srv-rhsoft systemd: Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory
Mar 26 13:02:01 srv-rhsoft systemd: Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory
Mar 26 13:20:01 srv-rhsoft systemd: Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory
Mar 26 13:40:01 srv-rhsoft systemd: Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory
Mar 26 14:00:01 srv-rhsoft systemd: Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory
Mar 26 14:02:01 srv-rhsoft systemd: Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory
Mar 26 14:20:01 srv-rhsoft systemd: Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory
Mar 26 14:40:01 srv-rhsoft systemd: Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory
Mar 26 15:00:01 srv-rhsoft systemd: Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory
Mar 26 15:02:01 srv-rhsoft systemd: Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory
Mar 26 15:20:01 srv-rhsoft systemd: Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory
Mar 26 15:40:01 srv-rhsoft systemd: Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory
Mar 26 16:01:01 srv-rhsoft systemd: Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory
Mar 26 16:20:01 srv-rhsoft systemd: Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory

(In reply to Michal Schmidt from comment #41)
> Do not assume this is easy to reproduce for the developers.
It is not about this being easy to reproduce, but that there appears to be no indication that the maintainer even accept this as a valid problem.

(In reply to Jóhann B. Guðmundsson from comment #25)
> Apparently it's to hard for you to do what you are asked. 
> 
> For example this could be because a spesific user is running the cron job (
> or not ) that triggers this and relates to PAM.
> 
> In anycase I got better things to do with my time let's see if Zbigniew can
> better deal with you and proceed further in trying to find the root cause
> for this.
It is nice that Jóhann apparently works for an employer that has no problem with internal code being posted in public. Expecting this to be the case for everyone is pretty arrogant.

In any case, several users (including myself) have commented that this happens even for the most basic cron jobs on a new and clean installation, basically ruling out any local misconfiguration or weird interaction with exotic tools or scripts.

Still, there is no indication whatsoever that indicates that someone is actually looking into this. (Probably better things to do)

(In reply to Thomas Müller from comment #43)
> It is not about this being easy to reproduce, but that there appears to be
> no indication that the maintainer even accept this as a valid problem.

I accept this is a valid problem. (Otherwise I would have already closed it as NOTABUG.)
We do not tend to post comments that merely acknowledge the validity of bugs, unless we already have some insight to share.

> Still, there is no indication whatsoever that indicates that someone is
> actually looking into this. (Probably better things to do)

I have been looking into this.
I don't know what makes the occurrence of the bug so common for some users, while it's so rare for others. With a cron job scheduled to run every minute, it still took hours to reproduce for me. Thankfully I was able to find faster reproducer. I'm running a service like this:

[Service]
ExecStart=/bin/sh -c 'while :; do su - root -c "sleep 1"; done'

This takes a few minutes to trigger the bug.

Based on my findings so far, the problem seems to be that sometimes when recursively trimming cgroups of user-0.slice, the cgroup .../user-0.slice/user is still present and gets removed too, making the service's 'cgroup_realized' attribute out of sync with reality.

Michal you might want to have a look at and share your findings with David since he had been trying to narrow down this ( race ) trigger in upstream bug 89145

any news here?
looks like this also affcets F22

for a highly critical component like systemd bugs visible by just read regular syslog happen way too often and stay way too long

Apr 19 22:25:37 rawhide systemd: Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory

[root@rawhide ~]# rpm -q systemd
systemd-219-11.fc22.x86_64

Indeed it does. But it's much rarer.

true - on F22 it happens not that often while on F21 i see that messages all day long on any machine virtual as well as physical

Can someone tell me how to reproduce this on -git? Running Michal's unit triggers a `systemd --user` restart every second (and I fixed several unrelated bugs now), but I could not trigger the CGROUP issue. I also played with some explicit preemption points, but it seems I can no longer trigger it after the cgroup-rewrite in systemd-225.

This message is a reminder that Fedora 21 is nearing its end of life.
Approximately 4 (four) weeks from now Fedora will stop maintaining
and issuing updates for Fedora 21. It is Fedora's policy to close all
bug reports from releases that are no longer maintained. At that time
this bug will be closed as EOL if it remains open with a Fedora  'version'
of '21'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version.

Thank you for reporting this issue and we are sorry that we were not 
able to fix it before Fedora 21 is end of life. If you would still like 
to see this bug fixed and are able to reproduce it against a later version 
of Fedora, you are encouraged  change the 'version' to a later Fedora 
version prior this bug is closed as described in the policy above.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events. Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.

Fedora 21 changed to end-of-life (EOL) status on 2015-12-01. Fedora 21 is
no longer maintained, which means that it will not receive any further
security or bug fix updates. As a result we are closing this bug.

If you can reproduce this bug against a currently maintained version of
Fedora please feel free to reopen this bug against that version. If you
are unable to reopen this bug, please file a new report against the
current release. If you experience problems, please add a comment to this
bug.

Thank you for reporting this bug and we are sorry it could not be fixed.

Still happening on Fedora 24 when running munin cron job intermittently:
[...]
Oct 23 15:55:01 amaterasu.greysector.net systemd[3033]: user: Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory
Oct 23 17:40:01 amaterasu.greysector.net systemd[20066]: user: Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory
Oct 23 19:45:01 amaterasu.greysector.net systemd[7980]: user: Failed at step CGROUP spawning /usr/lib/systemd/systemd: No such file or directory

$ id munin
uid=383(munin) gid=381(munin) groups=381(munin)
$ rpm -qf /etc/cron.d/munin 
munin-2.0.26-1.fc24.noarch
$ rpm -q systemd
systemd-229-16.fc24.x86_64

This message is a reminder that Fedora 24 is nearing its end of life.
Approximately 2 (two) weeks from now Fedora will stop maintaining
and issuing updates for Fedora 24. It is Fedora's policy to close all
bug reports from releases that are no longer maintained. At that time
this bug will be closed as EOL if it remains open with a Fedora  'version'
of '24'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version'
to a later Fedora version.

Thank you for reporting this issue and we are sorry that we were not
able to fix it before Fedora 24 is end of life. If you would still like
to see this bug fixed and are able to reproduce it against a later version
of Fedora, you are encouraged  change the 'version' to a later Fedora
version prior this bug is closed as described in the policy above.

Although we aim to fix as many bugs as possible during every release's
lifetime, sometimes those efforts are overtaken by events. Often a
more recent Fedora release includes newer upstream software that fixes
bugs or makes them obsolete.

Fedora 24 changed to end-of-life (EOL) status on 2017-08-08. Fedora 24 is
no longer maintained, which means that it will not receive any further
security or bug fix updates. As a result we are closing this bug.

If you can reproduce this bug against a currently maintained version of
Fedora please feel free to reopen this bug against that version. If you
are unable to reopen this bug, please file a new report against the
current release. If you experience problems, please add a comment to this
bug.

Thank you for reporting this bug and we are sorry it could not be fixed.