I cannot reproduce this with systemd-219-30.el7.x86_64 on a RHEL-7.3 machine. Nothing fails when TEMPORARY_STATE is set to 'yes'.

The problem seems to be that all those services set PrivateDevices=yes. If you comment that line, they start.

When stracing pid 1 and its forked processes (systemd-hostnamed in this particular case), this part reveals what failed:

[pid 12578] mount(NULL, "/dev/shm/lldpad.state", NULL, MS_REMOUNT|MS_BIND, NULL) = -1 EINVAL (Invalid argument)
[pid 12578] close(3)                    = 0
[pid 12578] munmap(0x7f03475be000, 4096) = 0
[pid 12578] socket(PF_LOCAL, SOCK_DGRAM|SOCK_CLOEXEC, 0) = 3
[pid 12578] getsockopt(3, SOL_SOCKET, SO_SNDBUF, [212992], [4]) = 0
[pid 12578] setsockopt(3, SOL_SOCKET, 0x20 /* SO_??? */, [8388608], 4) = 0
[pid 12578] setsockopt(3, SOL_SOCKET, SO_SNDTIMEO, "\n\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0", 16) = 0
[pid 12578] connect(3, {sa_family=AF_LOCAL, sun_path="/run/systemd/journal/socket"}, 29) = 0
[pid 12578] sendmsg(3, {msg_name(0)=NULL, msg_iov(11)=[{"PRIORITY=3\nSYSLOG_FACILITY=3\nCOD"..., 124}, {"USER_UNIT=systemd-hostnamed.serv"..., 35}, {"\n", 1}, {"MESSAGE_ID=641257651c1b4ec9a8624"..., 43}, {"\n", 1}, {"EXECUTABLE=/usr/lib/systemd/syst"..., 45}, {"\n", 1}, {"MESSAGE=Failed at step NAMESPACE"..., 94}, {"\n", 1}, {"ERRNO=22", 8}, {"\n", 1}], msg_controllen=0, msg_flags=0}, MSG_NOSIGNAL) = 354
[pid 12578] exit_group(226)             = ?

$ mount | grep /dev/shm
tmpfs on /dev/shm type tmpfs (rw,relatime,seclabel)
none on /dev/shm/lldpad.state type tmpfs (rw,relatime,seclabel)

I think that this might be a problem. Why is /dev/shm/lldpad.state mounted like that?

(In reply to Jan Synacek from comment #5)
> $ mount | grep /dev/shm
> tmpfs on /dev/shm type tmpfs (rw,relatime,seclabel)
> none on /dev/shm/lldpad.state type tmpfs (rw,relatime,seclabel)
> 
> I think that this might be a problem. Why is /dev/shm/lldpad.state mounted
> like that?

No idea, AFAIK we don't modify it in any way.

(In reply to Lev Veyde from comment #6)
> No idea, AFAIK we don't modify it in any way.

Is it "we the image creators"? I'm asking because I would like to know if it has to be mounted (or what was the reasoning) like that and if the problem disappears when the mount is not there. Then the solution would be simple. If the mount line really has to be there, then we would have to come up with something else. It doesn't look (to me, at least) that there's a bug in systemd in this case.

(In reply to Jan Synacek from comment #7)
> (In reply to Lev Veyde from comment #6)
> > No idea, AFAIK we don't modify it in any way.
> 
> Is it "we the image creators"? I'm asking because I would like to know if it
> has to be mounted (or what was the reasoning) like that and if the problem
> disappears when the mount is not there. Then the solution would be simple.
> If the mount line really has to be there, then we would have to come up with
> something else. It doesn't look (to me, at least) that there's a bug in
> systemd in this case.

Yes, that's what I meant - we don't change anything in that area as part of the oVirt-Live image creation.

From what I see it looks like that if that mount is removed then the services load normally (or rather can be restarted to work fine).

BTW, how you have concluded that the issue is specifically with that mount?
I couldn't find any useful information in the logs that could point to that direction.

A potential fix: https://github.com/systemd/systemd/commit/98df8089bea1b2407c46495b6c2eb76dda46c658
I'm not sure if it will fix this particular case.

If I provide a test build, would it be possible to rebuild the image?

(In reply to Jan Synacek from comment #9)
> A potential fix:
> https://github.com/systemd/systemd/commit/
> 98df8089bea1b2407c46495b6c2eb76dda46c658
> I'm not sure if it will fix this particular case.
> 
> If I provide a test build, would it be possible to rebuild the image?

Yes, we could try to do a test build using an updated systemd package.

https://github.com/lnykryn/systemd-rhel/pull/103

fix merged to upstream staging branch ->
https://github.com/lnykryn/systemd-rhel/commit/8d166597076d87aae9d5f98144103386c79d6446
-> post

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2017:2297