Bug 1002806

In either F20 or Rawhide, if I boot using a runlevel ("3" or "5") as a boot option, then the runlevel command returns the proper value ("N 3" or "N 5", resp.). Otherwise, it returns "unknown".

This bug only appears on systems installed using an install image. When running the 20 Alpha TC2 Desktop Live, or after installing from it, the runlevel command returns "N 5" even though it was not passed as a boot argument. So the quickest way to reproduce this is to do a minimal F20 install from netinst or DVD.

I hit this on TC2 w/ 32bit trying to install KDE Desktop, runlevel returns unknown, X wont start because there's no runlevel for systemd to determine when to start up graphical.target (Runlevel 5)

I did a PXE net install from the TC2 http://dl.fedoraproject.org/pub/alt/stage/20-Alpha-TC2/Fedora/i386/os/ repo directly, I did not try 64bit yet to confirm if this is platform specific or not.

KS file generated, this is based on a bootstrapped ks.cfg and then I manually select the installation type, I chose 'KDE Plasma Workspaces' option then installed with this setup:

On boot, there was no X started since no graphical.target was configured:

lrwxrwxrwx. 1 root root 37 Sep  3 23:07 /etc/systemd/system/default.target -> /lib/systemd/system/multi-user.target

[root@t42 ~]# runlevel
unknown

In Fedora 19, you would indeed see the runlevel.

anaconda-ks.cfg:

--- SNIP ---
#version=DEVEL
# System authorization information
auth --enableshadow --enablemd5
# Install OS instead of upgrade
install

# Use network installation
url --url="http://dl.fedoraproject.org/pub/alt/stage/20-Alpha-TC2/Fedora/i386/os/"
# Firewall configuration
firewall --disabled
ignoredisk --only-use=sda
# Keyboard layouts
# old format: keyboard us
# new format:
keyboard --vckeymap=us --xlayouts='us'
# System language
lang en_US.UTF-8

# Network information
network  --bootproto=dhcp --device=enp2s1 --noipv6
network  --hostname=t42
# Root password
rootpw --plaintext letmein
# SELinux configuration
selinux --disabled
# System services
services --enabled="chronyd"
# System timezone
timezone America/Toronto --isUtc
# System bootloader configuration
bootloader --location=mbr --driveorder="sda" --timeout=5 --boot-drive=sda
# Clear the Master Boot Record
zerombr
# Partition clearing information
clearpart --all  --drives=sda
# Disk partitioning information
part / --asprimary --fstype="ext2" --size=76018
part /boot --asprimary --fstype="ext2" --size=300

%pre
%end

%post --logfile /root/dump.log
echo "Disabling services: mcelog, smartd, firewalld, atd, crond"
chkconfig mcelog off
chkconfig smartd off
chkconfig firewalld off
chkconfig atd off
chkconfig crond off
chkconfig chronyd on

echo "nameserver 172.18.2.6" >> /etc/resolv.conf
echo "1 letmein" >> /etc/chrony.keys

%end

%packages --excludedocs --nobase
@admin-tools
@base-x
@core
@dial-up
@fonts
@guest-desktop-agents
@hardware-support
@input-methods
@kde-desktop
@multimedia
@printing
@standard
chrony
grub2
kernel
net-tools
-sendmail

%end
--- SNIP ---

Shawn, I think this is a different problem. The problem is that multi-user.target is used by default in your case (instead of graphical.target), not that runlevel command displays "unknown". Please file a separate bug report. Also don't forget to try the installation without the kickstart file.

Hi Kamil,

There may be two separate bugs though as a result of this issue however as you say.

1) systemd showing unknown
2) X not starting because systemd is not setting the target correctly (at least for KDE Plasma Workspaces) which may or may not be related to #1

I'll file a 2nd bug for the issue #2 if it turns out they are related we can link the two

So back to what this bug is initially about...

As has been in the man page for runlevel command. 

"This is a legacy command available for compatibility only. It should not be used anymore, as the concept of runlevels is obsolete."

Given that we need users/administrators to adapt to new concept and ways of doing things we at some point we should start removing that obsoleted backward compatibility so I think we should rather come up with a firm legacy removal policy for commands targets etc which systemd obsoletes which we can put under the systemd namespace on the wiki as well as list there which those commands targets etc are and when we plan on obsoleting them as opposed to patch these things up and have them continue to work in their limited/broken way.

Arguably 2 full GA releases should suffice as an migration/obsolete period for those things.

I will disagree, runlevels are still quite useful and the concept is not 'gone'. Switching the system from different runlevels is still handled:

See:

telinit
telinit [OPTIONS...] {COMMAND}

Send control commands to the init daemon.

     --help      Show this help
     --no-wall   Don't send wall message before halt/power-off/reboot

Commands:
  0              Power-off the machine
  6              Reboot the machine
  2, 3, 4, 5     Start runlevelX.target unit
  1, s, S        Enter rescue mode
  q, Q           Reload init daemon configuration
  u, U           Reexecute init daemon


Its just now the runlevels are targets.

Not to mention that this bug only affects installs from install images, not running live or installs from live, so it may be caused by something trivial.

(In reply to Shawn Starr from comment #9)
> I will disagree, runlevels are still quite useful and the concept is not
> 'gone'. Switching the system from different runlevels is still handled:
> 
> See:

There is not 1 to 1 mapping between run levels and targets and alternating the old commands to providing limited backwards compatibility to the new stuff is only hindering adoption as in change of mindset and workflow since administrators and users alike have certain expectation to the behaviour and workflow using those commands.

telinit for example is no longer working in the same way as it has for the last decade. You no longer go and edit /etc/inittab you do not use it the same way when setting and troubleshooting problem so it's existence only confuse administrators and their expectation regarding it's behaviour. ( And we have had plethora of bugs related to it )

At least that's my point of view as in we should have gotten the users to get accustom to the new commands by essentially forcing them to use the native systemd commands by having them backward incompatible instead of alternating and introducing limit backward compatibility and behaviour with the old commands and the old way administrate and doing things.

Hi,
the 'runlevel' command in systemd works a bit differently than sysvinit runlevels, because the a specific runlevel is only returned when it is reached, while sysvinit would display it as soon as the switch was requested. So,
to have an answer like '3', two things are required:

1. the target (multi-user.target or graphical.target) must be reached
2. runlevel3.target must be an alias for that target

So, can you do:
$  ls -l /usr/lib/systemd/system/runlevel?.target
$  systemctl status multi-user.target graphical.target default.target runlevel{3,5}.target

?

On one of my F20 VirtualBox guests, which originally gave "unknown", it now shows "N 5". I don't know when it fixed itself. On a F20 KVM guest I installed from the 20 Alpha RC2 DVD yesterday, it still shows "unknown". Here is the output from that:

[andre@localhost ~]$ ls -l /usr/lib/systemd/system/runlevel?.target
lrwxrwxrwx. 1 root root 15 Sep 12 09:51 /usr/lib/systemd/system/runlevel0.target -> poweroff.target
lrwxrwxrwx. 1 root root 13 Sep 12 09:51 /usr/lib/systemd/system/runlevel1.target -> rescue.target
lrwxrwxrwx. 1 root root 17 Sep 12 09:51 /usr/lib/systemd/system/runlevel2.target -> multi-user.target
lrwxrwxrwx. 1 root root 17 Sep 12 09:51 /usr/lib/systemd/system/runlevel3.target -> multi-user.target
lrwxrwxrwx. 1 root root 17 Sep 12 09:51 /usr/lib/systemd/system/runlevel4.target -> multi-user.target
lrwxrwxrwx. 1 root root 16 Sep 12 09:51 /usr/lib/systemd/system/runlevel5.target -> graphical.target
lrwxrwxrwx. 1 root root 13 Sep 12 09:51 /usr/lib/systemd/system/runlevel6.target -> reboot.target
[andre@localhost ~]$ systemctl status multi-user.target graphical.target default.target runlevel{3,5}.target
multi-user.target - Multi-User System
   Loaded: loaded (/usr/lib/systemd/system/multi-user.target; disabled)
   Active: active since Fri 2013-09-13 08:17:19 EDT; 4min 29s ago
     Docs: man:systemd.special(7)

Sep 13 08:17:19 localhost.localdomain systemd[1]: Starting Multi-User System.
Sep 13 08:17:19 localhost.localdomain systemd[1]: Reached target Multi-User S...

graphical.target - Graphical Interface
   Loaded: loaded (/lib/systemd/system/graphical.target; enabled)
   Active: active since Fri 2013-09-13 08:17:19 EDT; 4min 29s ago
     Docs: man:systemd.special(7)

Sep 13 08:17:19 localhost.localdomain systemd[1]: Starting Graphical Interface.
Sep 13 08:17:19 localhost.localdomain systemd[1]: Reached target Graphical In...

graphical.target - Graphical Interface
   Loaded: loaded (/lib/systemd/system/graphical.target; enabled)
   Active: active since Fri 2013-09-13 08:17:19 EDT; 4min 29s ago
     Docs: man:systemd.special(7)

Sep 13 08:17:19 localhost.localdomain systemd[1]: Starting Graphical Interface.
Sep 13 08:17:19 localhost.localdomain systemd[1]: Reached target Graphical In...

multi-user.target - Multi-User System
   Loaded: loaded (/usr/lib/systemd/system/multi-user.target; disabled)
   Active: active since Fri 2013-09-13 08:17:19 EDT; 4min 29s ago
     Docs: man:systemd.special(7)

Sep 13 08:17:19 localhost.localdomain systemd[1]: Starting Multi-User System.
Sep 13 08:17:19 localhost.localdomain systemd[1]: Reached target Multi-User S...

graphical.target - Graphical Interface
   Loaded: loaded (/lib/systemd/system/graphical.target; enabled)
   Active: active since Fri 2013-09-13 08:17:19 EDT; 4min 29s ago
     Docs: man:systemd.special(7)

Sep 13 08:17:19 localhost.localdomain systemd[1]: Starting Graphical Interface.
Sep 13 08:17:19 localhost.localdomain systemd[1]: Reached target Graphical In...
[andre@localhost ~]$ runlevel 
unknown
[andre@localhost ~]$

Hm, and what does 'utmpdump < /var/run/utmp' say?

Just so you know, after my last comment, I first tried updating just systemd, and then everything, and rebooting, to see if it made any difference. It didn't. On the other hand, my 20 and Rawhide VirtualBox guests, which have existed and been continuously updated for a few weeks, both "fixed" themselves at some point. I have no idea how or when.

Anyway, the requested output on my KVM guest (which is now fully updated, and still returns "unknown") is

[andre@localhost ~]$ utmpdump < /var/run/utmp
Utmp dump of /dev/stdin
[2] [00000] [~~  ] [reboot  ] [~           ] [3.11.0-300.fc20.x86_64] [0.0.0.0        ] [Sat Sep 14 03:56:46 2013 EDT]
[7] [01214] [:0  ] [andre   ] [:0          ] [:0                  ] [0.0.0.0        ] [Sat Sep 14 03:57:44 2013 EDT]
[7] [01988] [/0  ] [andre   ] [pts/0       ] [:0                  ] [0.0.0.0        ] [Sat Sep 14 03:58:26 2013 EDT]
[andre@localhost ~]$

running "telinit 3" or "init 3" will result in the runlevel changing, but in the tty session you ran the command the prompt will never return, nor the session reset.

I must also strongly encourage us to NOT try and obsolete the idea of runlevels.  They are indeed most useful as a fast way to diagnose, and/or fix problems.

If "runlevel" returns "unknown" this usually means that the boot is not complete yet. You probably still have some jobs queued. Check which "systemctl list-jobs" which ones those are.

This is from a clean install from Fedora-20-Beta-TC2-x86_64-DVD.iso (minimal, no updates).

[root@localhost ~]# systemctl list-jobs
No jobs running.
[root@localhost ~]# runlevel
unknown
[root@localhost ~]#

I'm aware that sometimes it takes a while for runlevel to start showing the normal output, but I've given it plenty of time. This never happened before, and I've been running it on each compose for a long time since https://fedoraproject.org/wiki/QA:Testcase_Package_Sets_Minimal_Package_Install specifically suggests using the runlevel command. It might spontaneously "fix" itself eventually, as I noted above, but normal behavior is that it should work almost immediately after the first boot.

This really looks like systemd bug, probably something to do with the fact that default.target is also known as runlevel5.target. systemd-update-utmp-runlevel.service is not run, even though default.target has it in Wants=, and default.target is "started", i.e. completed. Starting systemd-update-utmp-runlevel.service manually causes that utmp entry to be inserted.

[test@fedora20 ~]$ sudo systemctl start default.target
[test@fedora20 ~]$ runlevel
unknown
[test@fedora20 ~]$ sudo systemctl start runlevel5.target
[test@fedora20 ~]$ runlevel
N 5

[test@fedora20 ~]$ systemctl show --no-pager -p Wants default.target
Wants=gdm.service systemd-readahead-replay.service systemd-readahead-collect.service accounts-daemon.service rtkit-daemon.service
[test@fedora20 ~]$ systemctl show --no-pager -p Wants graphical.target
Wants=gdm.service systemd-readahead-replay.service systemd-readahead-collect.service accounts-daemon.service rtkit-daemon.service
[test@fedora20 ~]$ systemctl show --no-pager -p Wants runlevel5.target
Wants=gdm.service systemd-readahead-replay.service systemd-readahead-collect.service accounts-daemon.service rtkit-daemon.service systemd-update-utmp-runlevel.service

WAT?

So... runlevel5.target isn't actually referenced by anything, and systemd will happily reach default.target = graphical.target without ever attempting to load runlevel5.target. This means that drop-ins and .wants directories attached to runlevel5.target are not processed. If at any point, runlevel5.target is loaded (systemctl status runlevel5.target is enough), runlevel5.target becomes another name for graphical.target.

I'm not sure what the proper fix here is:

1. disallow .d and .wants and .requires dirs on aliases
2. 1., but only if the alias is not wanted or required somewhere else
3. include .d and .wants and .requires for all aliases of a unit. This would require looking for all aliases in advance, an O(n) operation, unless a cache of aliases is used.

systemd-208-4.fc20 has been submitted as an update for Fedora 20.
https://admin.fedoraproject.org/updates/systemd-208-4.fc20

Package systemd-208-4.fc20:
* should fix your issue,
* was pushed to the Fedora 20 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing systemd-208-4.fc20'
as soon as you are able to.
Please go to the following url:
https://admin.fedoraproject.org/updates/FEDORA-2013-19680/systemd-208-4.fc20
then log in and leave karma (feedback).

still occurs in systemd-208-4.fc20

(In reply to lnie from comment #23)
> still occurs in systemd-208-4.fc20
Is this after a reboot?

What does 'runlevel; systemctl list-jobs; runlevel; systemctl start runlevel5.target; runlevel' say?

God forgive~,I forgot to reboot it:(
seems be fixed already:)

@ lnie: If you change your karma on https://admin.fedoraproject.org/updates/FEDORA-2013-19680/systemd-208-4.fc20 to +1, it should reach +3 and go to stable.

@Zbigniew Jędrzejewski-Szmek: In the changelog, you said it was a "temporary fix" - how did you fix it (option 1, 2, 3, or something else)?

(In reply to Andre Robatino from comment #26)
> @ lnie: If you change your karma on
> https://admin.fedoraproject.org/updates/FEDORA-2013-19680/systemd-208-4.fc20
> to +1, it should reach +3 and go to stable.
I pushed it anyway, assuming that lnie's negative karma doesn't count.

> @Zbigniew Jędrzejewski-Szmek: In the changelog, you said it was a "temporary
> fix" - how did you fix it (option 1, 2, 3, or something else)?
I copied the links from runlevel5.wants to graphical.wants, and the same for other runlevels. Far from ideal, but it seems that people don't use .wants for aliases all that often, and systemd itself was one of the main users.

(In reply to Andre Robatino from comment #26)
> @ lnie: If you change your karma on
> https://admin.fedoraproject.org/updates/FEDORA-2013-19680/systemd-208-4.fc20
> to +1, it should reach +3 and go to stable.
> 
going to add one :)

systemd-208-4.fc20 has been pushed to the Fedora 20 stable repository.  If problems still persist, please make note of it in this bug report.