Bug 766935

Summary: systemd hardcoded to sysv runlevel 5
Product: [Fedora] Fedora Reporter: Joost Ruijsch <joost>
Component: systemdAssignee: systemd-maint
Status: CLOSED NOTABUG QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: high Docs Contact:
Priority: unspecified    
Version: 15CC: johannbg, johannbg, lpoetter, metherid, mschmidt, notting, plautrba, systemd-maint
Target Milestone: ---   
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2012-01-26 09:12:31 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Joost Ruijsch 2011-12-12 20:09:43 UTC
Description of problem:
No matter what runlevel is used, systemd only executes and configures runlevel 5 for legacy initscripts.

Version-Release number of selected component (if applicable):
systemd-26-13.fc15.i686
(Bug has been in all versions afaik)

# systemctl is-enabled clamav.service
clamav.service is not a native service, redirecting to /sbin/chkconfig.
Executing /sbin/chkconfig clamav --level=5
# systemctl is-enabled clamav.service
clamav.service is not a native service, redirecting to /sbin/chkconfig.
Executing /sbin/chkconfig clamav --level=5
# runlevel
N 3
# ls -l /etc/systemd/system/default.target
lrwxrwxrwx 1 root root 36 Nov 18  2010 /etc/systemd/system/default.target -> /lib/systemd/system/runlevel3.target

clamav did not start during bootup. It was enabled for level 3, but not for level 5.

Expected results:
"Executing /sbin/chkconfig clamav --level=3"
(or just without any "--level=" )

Executing proper runlevel for sysv initscripts.

Additional info:
"--level=5" is hardcoded in systemctl.c

Comment 1 Jóhann B. Guðmundsson 2012-01-24 11:00:52 UTC
There are no such things as "runlevels" in systemd world and the backwards compatibility to it is limited. 

Each unit is configured and tied to a specific "target" through the [Install] section of those units. 

Now 

"clamav.service is not a native service, redirecting to /sbin/chkconfig."

Tells us that you are using the legacy sysv init script but as far as I can tell clamav already has native systemd units 

http://pkgs.fedoraproject.org/gitweb/?p=clamav.git;a=tree;h=eed51b8ded09188753fa8f84db5ce5f11afbcc91;hb=eed51b8ded09188753fa8f84db5ce5f11afbcc91

You should hence 

a) updated to a release of clamav that introduces those units  

or

b) copy the relevant units in git to /etc/systemd/system directory and run systemctl daemon-reload and systemctl enable foo.service and systemctl start foo.service

For better integration of clamav with systemd.

Comment 2 Joost Ruijsch 2012-01-24 14:15:02 UTC
Now that is exactly my point "backwards compatibility to it is limited".
It is not just limited, but (intentionally?) crippled. 

Systemd was introduced in F15 and at that time many of the services available had not been updated to systemd. Considering the Fedora update policy they never will be, these updates are only targeted for F16 or later.

Systemd has targets (like runlevel3.target) that should map to runlevels for backwards compatibility, only they DONT. The backwards compatibiliy is flawed and that is why I reported this.

Some testing revealed that a legacy service is started on boot if it is enabled for any of the runlevels. It is only not started if it is disabled for all runlevels. Effectively, systemd does not allow one to configure initscripts for different runlevels or targets.
I have not found any documentation or explanation for this behavior, so I assume it must be a bug, not a feature.

Bug 709254 has resulted in a workaround for this problem in 'ntsysv', but this is only available for F16. (But I successfully installed the F16 rpm on F15).

There are many problem reports on the net about services starting on boot when they are disabled. A proper runlevel mapping could have avoided a lot of trouble. Standard answer from the systemd people is to use the graphical service configuration tool, but they forget not everyone runs X. My server does not even have a graphics card installed.

Comment 3 Jóhann B. Guðmundsson 2012-01-24 14:46:53 UTC
(In reply to comment #2)
> Now that is exactly my point "backwards compatibility to it is limited".
> It is not just limited, but (intentionally?) crippled. 

It's the other way around backwards compatibly was added to systemd so it was "intentionally" added to the extent it made sense.

Systemd is a new technology thus it should be approached as one.  
 
> Systemd was introduced in F15 and at that time many of the services available
> had not been updated to systemd. Considering the Fedora update policy they
> never will be, these updates are only targeted for F16 or later.

True unit's aren't introduced into GA releases without granted permissions from fesco to do so but that's not something we decided nor is it our fault that there are so few units in the distribution. 

We have units for 50 packages that never got shipped for F15/F16 and we have units for additional ( currently ) new 66 packages for F17/rawhide combined this is 116 packages that have more than 116 units for them stuck in bugzilla.

Again that's not our fault it's the relevant maintainers things are getting hold upon and causing bad user experience to our user base.  

> Systemd has targets (like runlevel3.target) that should map to runlevels for
> backwards compatibility, only they DONT. The backwards compatibiliy is flawed
> and that is why I reported this.

The backward compatibility is sufficient 

Many of the legacy sysv init script contained broken LSB headers or simply did not contain them et all.

> Some testing revealed that a legacy service is started on boot if it is enabled
> for any of the runlevels. It is only not started if it is disabled for all
> runlevels. Effectively, systemd does not allow one to configure initscripts for
> different runlevels or targets.
> I have not found any documentation or explanation for this behavior, so I
> assume it must be a bug, not a feature.

Sysv init script are considered to be legacy and the time is better spent migrating those scripts to native systemd units for the best systemd intergration instead of trying to get them to work by either fixing the legacy sysv init script or with some hack.
 
> Bug 709254 has resulted in a workaround for this problem in 'ntsysv', but this
> is only available for F16. (But I successfully installed the F16 rpm on F15).

For the most part ( with few exeption ) you dont need install F16 rpms you only need the unit file(s).
 
> There are many problem reports on the net about services starting on boot when
> they are disabled. A proper runlevel mapping could have avoided a lot of
> trouble. Standard answer from the systemd people is to use the graphical
> service configuration tool, but they forget not everyone runs X. My server does
> not even have a graphics card installed.

Again there are no such things as runlevels in systemd user/administrator can if they want to create their own target and boot directly into that and or link the default target to that ( or multi-user.target ) if they want to.

Comment 4 Joost Ruijsch 2012-01-24 16:06:07 UTC
Thanks for your reply, I did not know it was so difficult to get all the unit files packaged and shipped.

But concerning the backwards compatibility issue I tried to address, you missed my point. Please dont answer "Again there are no such things as runlevels in systemd" again or my head will explode.

By 'runlevel' I refer to the legacy sysv runlevel concept and its relevance to systemd backwards compatibility. With 'target' I refer to the new systemd concept.

I try to explain again:

If you configure systemd for runlevel3.target as default on boot, then you would expect systemd to start the legacy services as configured in /etc/rc.d/rc3.d.
Same thing for runlevel5.target and /etc/rc.d/rc5.d
That is what I and many users expected to happen, but does not.

What does happen:
Systemd executes starts ANY legacy service that is enabled in ANY of the /etc/rc.d/rc#.d directories, regardless of whatever target is configured.
This means that the only way to disable a legacy service, is to disable it in ALL /etc/rc.d/rc#.d directories. 
(I wanted to say 'runlevels' here, but I know "there is no such thing as...")

This means that you cannot enable legacy services for specific targets only. It is enabled for either all or none of the targets in a system. There is no mapping between targets and runlevels.

Is this intentional and designed behavior? Is this explained and documented somewhere?

>> Bug 709254 has resulted in a workaround for this problem in 'ntsysv', but this
>> is only available for F16. (But I successfully installed the F16 rpm on F15).

>For the most part ( with few exeption ) you dont need install F16 rpms you only
>need the unit file(s).

I was referring to the 'ntysv' rpm, which in response to bug 709254 was changed to enable/disable services in ALL runlevels instead of just the current. 
(I mean whatever the 'runlevel' command returns corresponding to runlevel#.target, being fully aware that "Again there are no such things as runlevels in systemd...")

Comment 5 Jóhann B. Guðmundsson 2012-01-25 13:20:54 UTC
(In reply to comment #4)
> 
> What does happen:
> Systemd executes starts ANY legacy service that is enabled in ANY of the
> /etc/rc.d/rc#.d directories, regardless of whatever target is configured.
> This means that the only way to disable a legacy service, is to disable it in
> ALL /etc/rc.d/rc#.d directories. 
> (I wanted to say 'runlevels' here, but I know "there is no such thing as...")
> 
> This means that you cannot enable legacy services for specific targets only. It
> is enabled for either all or none of the targets in a system. There is no
> mapping between targets and runlevels.
> 
> Is this intentional and designed behavior? Is this explained and documented
> somewhere?
> 

Hmm 

Are you sure this is not happening to you because the legacy sysv init script in question does not have proper lsb headers as in chkconfig, Default-Start,Default-Stop etc... 

Many of them are utterly broken in that regard which might explain your unfortunate experience. 

Does this also happen with those that do have proper lsb headers?

Comment 6 Jóhann B. Guðmundsson 2012-01-25 13:30:30 UTC
With regards to your documentation question you might want to read this.

http://www.freedesktop.org/wiki/Software/systemd/Incompatibilities

Pay attention to the last section...

Comment 7 Jóhann B. Guðmundsson 2012-01-26 09:12:31 UTC
I think both comment 5 and comment 6 answer all the questions you might have had about your issue thus closing this as a not a bug.

Thanks.

Comment 8 Joost Ruijsch 2012-01-26 14:24:44 UTC
I have done some tests and that last "incompatibilities" section seems correct. multi-user.target is mapped to runlevels 234, not all runlevels as I thought.

But that still leaves my original question:

Why is systemctl hardcoded to "--level=5"?

When running multi-user.target that is mapped to runlevels 234, than manipulating level 5 makes no sense does it?

The LSB headers have nothing to do with the issue imho.

# systemctl is-enabled httpd.service
httpd.service is not a native service, redirecting to /sbin/chkconfig.
Executing /sbin/chkconfig httpd --level=5

This is still a bug.

Comment 9 Michal Schmidt 2012-01-26 14:47:40 UTC
(In reply to comment #8)
> Why is systemctl hardcoded to "--level=5"?

Because the alternative would be worse. Without it, "chkconfig <servicename>" would try to give an answer valid for the current runlevel, but there are uses of "systemctl is-enabled ..." in scripts during boot where there is no runlevel defined because no target has been reached yet.

Comment 10 Joost Ruijsch 2012-01-26 15:18:39 UTC
(In reply to comment #9)
> (In reply to comment #8)
> > Why is systemctl hardcoded to "--level=5"?
> 
> Because the alternative would be worse. Without it, "chkconfig <servicename>"
> would try to give an answer valid for the current runlevel, but there are uses
> of "systemctl is-enabled ..." in scripts during boot where there is no runlevel
> defined because no target has been reached yet.

I can see the point of that, but it is a little blunt to just assume a runlevel in all cases.

In /lib/systemd/system/ the following symlinks exist:

runlevel0.target -> poweroff.target
runlevel1.target -> rescue.target
runlevel2.target -> multi-user.target
runlevel3.target -> multi-user.target
runlevel4.target -> multi-user.target
runlevel5.target -> graphical.target
runlevel6.target -> reboot.target

I think that systemd checks these symlinks on boot and finds that runlevels 2,3 and 4 are symlinked to the curent multi-user.target. Then it checks for services that are enabled for any of those three runlevels.

Would it not be possible for systemctl to follow the same logic?
If no target is defined, than it could just follow the default target and only as a last resort assume runlevel 5.
 
The 'ntsysv' tool defaults to the current runlevel (whatever /sbin/runlevel returns) and that is 3 for multi-user.target. When it disables a service, then it is still enabled for level 2 or 4 and effectively still enabled on boot.

Both of these tools do not properly function for 'multi-user.target'. Users running 'graphical.target' have no problems.

Considering how long it may take before all services are migrated to systemd, it could make sense to improve compatibility.. just my 2 cents..