Bug 734268 - Plese remove /etc/rc.local or chmod -x it
Summary: Plese remove /etc/rc.local or chmod -x it
Keywords:
Status: CLOSED NOTABUG
Alias: None
Product: Fedora
Classification: Fedora
Component: initscripts
Version: 16
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: Bill Nottingham
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2011-08-30 00:15 UTC by Lennart Poettering
Modified: 2014-03-17 03:28 UTC (History)
9 users (show)

Fixed In Version: initscripts-9.32-1.fc16
Doc Type: Bug Fix
Doc Text:
Clone Of:
: 752901 (view as bug list)
Environment:
Last Closed: 2011-11-10 18:32:17 UTC
Type: ---
Embargoed:


Attachments (Terms of Use)

Description Lennart Poettering 2011-08-30 00:15:03 UTC
I'd really like to see /etc/rc.local go away by default. The systemd unit we ship will execute it only if the file exists and is executable. Thus I'd like to ask you to either remove the file or at least "chmod -x" it, so that we don't start it anymore.

In both cases just creating it/chmod +x will magically bring it back.

If you decide to chmod -x it by default, it might be a good idea to add a comment to the file explaining that users have to chmod +x it before it takes effect.

Comment 1 Lennart Poettering 2011-08-30 13:00:35 UTC
Thinking about it there might be a nicer third option: we turn rc-local.service into a service that is not statically enabled but dynamically enabled. We'd then enable it for upgrades but leave it disabled for new installs. We'd leave /etc/rc.local in place the way it is right now. To make use of it people would then just have to edit it and run "systemctl enable rc-local.service". This scheme has one big advantage: the synchronization point that rc-local.service is would not be included at all in the initial transaction which would be quite an improvement in paralellization.

Or, the fourth option would be to remove this thing entirely: /etc/rc.local as well as rc-local.service.

I'd vote for the fourth or third option (in this order). What do you prefer?

Comment 2 Bill Nottingham 2011-08-31 22:03:01 UTC
Honestly, I'd go for the first... it's far far simpler to just remove it in packaging, but keep it around and enabled such that it will be run if it shows up.

Comment 4 Fedora Update System 2011-09-02 20:11:05 UTC
initscripts-9.32-1.fc16 has been submitted as an update for Fedora 16.
https://admin.fedoraproject.org/updates/initscripts-9.32-1.fc16

Comment 5 Fedora Update System 2011-09-06 18:08:01 UTC
Package initscripts-9.32-1.fc16:
* should fix your issue,
* was pushed to the Fedora 16 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing initscripts-9.32-1.fc16'
as soon as you are able to.
Please go to the following url:
https://admin.fedoraproject.org/updates/initscripts-9.32-1.fc16
then log in and leave karma (feedback).

Comment 6 Fedora Update System 2011-09-11 01:44:09 UTC
initscripts-9.32-1.fc16 has been pushed to the Fedora 16 stable repository.  If problems still persist, please make note of it in this bug report.

Comment 7 Rodrigo de Farias Gomes 2011-10-11 21:17:49 UTC
The command systemctl enable rc-local.service not work in f16:
"Warning: unit files do not carry install information. No operation executed."

Comment 8 Bill Nottingham 2011-10-12 17:56:52 UTC
You don't have to enable it. You just need to create the actual /etc/rc.d/rc.local file.

Comment 9 Benoît Gaurier 2011-11-03 11:45:08 UTC
When file /etc/rc.local is missing or is not executable, service rc-local fails to start at startup. File /etc/rc.d/rc.local does not seem to be looked at. On the other hand, when /etc/rc.local exists and is executable, service rc-local seems to have no problem at startup (no error message), however the commands it contains are not executed ! (tested with simple commands such as touch file or echo "test" > file)

Comment 10 DiEOrLivE 2011-11-03 15:13:08 UTC
>You just need to create the actual /etc/rc.d/rc.local file.

It does not work for me either.

Comment 11 Ingo Molnar 2011-11-10 06:27:22 UTC
It does not work.

Creating the /etc/rc.d/rc.local file and making it executable:

-rwxr-xr-x. 1 root root 31 Nov 10 07:05 /etc/rc.d/rc.local

Still results in:

# systemctl enable rc-local.service
Warning: unit files do not carry install information. No operation executed.

Also, the systemd error message here is absolutely lacking. It does not explain anything what is wrong, it just prints an opaque message that gives the user no actionable path to resolve the problem.

User facing system tools must *always* print actionable messages.

After reading the systemd source code and scratching my head i finally found the problem, there's no [Install] section in the rc-local.service file. Adding this:

[Install]
WantedBy=multi-user.target

To the end of the /lib/systemd/system/rc-local.service file resolves the bug. (But i'm not sure whether multi-user.target is the desired thing here.)

Please improve the error message, it should at minimum say:

Error: the unit file does not have an [Install] section. No operation executed.

Otherwise users have no idea what's wrong.

Comment 12 Ingo Molnar 2011-11-10 06:54:29 UTC
rc.local is giving more trouble. While 'systemctl enable rc-local.service' works, 'systemctl start rc-local.service' does not:

# systemctl start rc-local.service
Job failed. See system logs and 'systemctl status' for details.

Now, neither the system logs nor 'systemctl status' tells us what's wrong though:

 systemctl status rc-local.service
rc-local.service - /etc/rc.local Compatibility
	  Loaded: loaded (/lib/systemd/system/rc-local.service; enabled)
	  Active: failed since Thu, 10 Nov 2011 07:32:09 +0100; 3min 8s ago
	 Process: 5078 ExecStart=/etc/rc.d/rc.local start (code=exited, status=203/EXEC)
	  CGroup: name=systemd:/system/rc-local.service

Nov 10 07:32:09 meteor systemd[1]: rc-local.service: control process exited, code=exited status=203
Nov 10 07:32:09 meteor systemd[1]: Unit rc-local.service entered failed state.

The messages are rather meaningless and non-actionable. So how is a systemd user supposed to be able to debug such bugs?

The strace suggests that systemd is talking to the systemd process via /run/systemd/private but gets a message it does not like.

After stracing the systemd daemon itself, it turns out that:

5259  execve("/etc/rc.d/rc.local", ["/etc/rc.d/rc.local", "start"], [/* 6 vars */] <unfinished ...>
5259  <... execve resumed> )            = -1 ENOEXEC (Exec format error)

The problem was that while /etc/rc.d/rc.local was executable and could be executed from a shell prompt, it did not come with a #!/bin/bash starting line that marks it exec()able.

Fixing that made it start up.

There's two systemd problems/bugs here:

1) the message was *WAY* too opaque and indirect to figure out what's going on. There's a marked difference between not being able to execute a file and the script itself failing. The -ENOEXEC was not propagated back to the user - this is a basic FAIL, i suspect such simple mistakes happen all the time and users are kept wondering about what's wrong with their startup scripts.

2) systemd should probably consider executing executable text files.

3) when systemctl executes something successfully, it should print an informational line about it instead of just returning silently with an exit status of 0. This is command line UI 101: just returning silently keeps the user wondering whether anything happened. The old 'service XYZ start' mechanism returned a message in all cases for this precise reason.

Comment 13 Ingo Molnar 2011-11-10 06:58:41 UTC
4)

The most user-friendly behavior would be for systemd to check whether all relevant scripts are executable to systemd at the 'enable' stage, and warn about any mistakes at that stage: both the lack of execute permission and the wrong script format should be warned about.

A user/admin/developer might mistakenly conclude that enabling a service implies that it will be attempted to run on the next bootup, etc. - while systemd already knows at the 'enable' stage that this will be impossible because the file won't be executed.

Comment 14 Bill Nottingham 2011-11-10 17:25:55 UTC
So, I'm at a loss here.

1) As already stated, *you do not need to enable rc-local.service*. It is shipped in always-enabled state; it merely has a check to only run if /etc/rc.d/rc.local is executable. (To be precise: /lib/systemd/system/multi-user.target.wants/rc-local.service is shipped as a symlink with systemd itself.)

You can check this with

# systemctl list-unit-files | grep rc-local.service
rc-local.service          static

'static' means it's statically enabled; no configuration required.

Now, if it's not this way out of the box for you, we can try and figure out why. But given that the symlink is packaged in systemd-units, I'm surprised if that's the case.

2) If you're seriously suggesting that systemd should inspect everything marked with Exec=, ExecStartPre=, and so on, compare it against the kernel's currently supported binary formats (all the way through to binfmt_misc configuration), and not enable things that may not run... I respectfully disagree. That's a lot of complex code, inspecting the binaries runs into potential security exposure, and the kernel's binfmt configuration can change between installation and execution significantly. (Especially if you're installing to a chroot, or to a system image, or...)

rc.local has always been a shell script. I'd really classify creating it as an invalid script as admin error.

3) You mention that the ENOEXEC isn't propagated back to the user... it is:

 systemctl status rc-local.service
rc-local.service - /etc/rc.local Compatibility
   Loaded: loaded (/lib/systemd/system/rc-local.service; enabled)
   Active: failed since Thu, 10 Nov 2011 07:32:09 +0100; 3min 8s ago
  Process: 5078 ExecStart=/etc/rc.d/rc.local start (code=exited,
status=203/EXEC)
   CGroup: name=systemd:/system/rc-local.service

Specifically: (code=exited, status=203/EXEC)

Comment 15 Ingo Molnar 2011-11-10 18:32:17 UTC
1)

I think you are right that most of the symptoms that I saw were the result of /etc/rc.d/rc.local being an executable file to a shell but not a shell script per se. It was downhill from there.

Previously this was never a problem because there was a template in place. In F16 this template got removed, which removed the leading #!/bin/bash and thus made it easy to create a script there that is not actually executable to systemd.

2)

Well, the 'file' utility can tell them apart:

[root@meteor ~]# file /etc/rc.d/rc.local 
/etc/rc.d/rc.local: Bourne-Again shell script, ASCII text executable
[root@meteor ~]# file /etc/rc.d/rc.local 
/etc/rc.d/rc.local: ASCII text

So this test might be enough in practice:

  file /etc/rc.d/rc.local | grep -q executable 

but i can understand if you don't want to go there - was just a suggestion.

3)

I missed the 203/EXEC bit because it wasn't a familiar pattern (i mistakenly thought that it simply means that things were attempted to be executed) - would it be possible to prefix that output with -ENO, resulting in -ENOEXEC or so, signalling that it's a *failure*?

Also, the output is pretty meaningless because it's not structured, unless you really are experienced in systemctl output (which most of the people are not):

   Active: failed since Thu, 10 Nov 2011 07:32:09 +0100; 3min 8s ago
  Process: 5078 ExecStart=/etc/rc.d/rc.local start (code=exited, status=203/EXEC)

Something like this:

   Active: failed since Thu, 10 Nov 2011 07:32:09 +0100; 3min 8s ago
   Process: 5078 ExecStart=/etc/rc.d/rc.local start exited abnormally with status 203/-ENOEXEC - file not executable via exec()?

would have led me to the problem straight away.

The fact remains, i spent more than half an hour on something that should have been far more obvious to analyse at first sight. I only solved it because i straced the systemctl command, strace -s 100000 it once again to see all the message, saw the writes() to the socket resulting in a failure being passed back, then identified where the pipe was openened, realized that it was used by PID 1, then straced PID 1 and also looked at the systemd source code and man page while doing this.

If i have trouble with that and have to go to such lengths to figure out a pretty common failure then what will the typical sysadmin experience?

Comment 16 Bill Nottingham 2011-11-10 18:47:01 UTC
Cloned the last bit as a RFE for systemd.

Comment 17 John Mev 2011-11-13 10:41:24 UTC
Don't understand:

You have closed the bug as fixed in initscripts-9.32-1.fc16

My release:

[root@Fedora16 rc.d]# rpm -q initscripts
initscripts-9.34-2.fc16.i686

and rc.local still doesn't work:

[root@Fedora16 rc.d]# pwd
/etc/rc.d
[root@Fedora16 rc.d]# ll rc.local
-rwxr-xr-x. 1 root root 150 13 nov.  11:36 rc.local

Now what sould I do to have this command run ?

Thx

Comment 18 Bill Nottingham 2011-11-15 04:25:09 UTC
What's your rc.local look like?

Comment 19 John Mev 2011-11-15 10:39:45 UTC
So:

[Fedora16 VM]/etc/rc.d # ls -l rc.*
-rwxr-xr-x. 1 root root 157 13 nov.  12:52 rc.local


[Fedora16 VM]/etc/rc.d # cat rc.local
# rc.local rajouté
#!/bin/bash

/bin/mount -t vboxsf opt /opt2 -o ro
/bin/mount -t vboxsf local /usr/local -o ro

echo "c'est fait à `date`" >> /root/local



And nothing more is written in /root/local


Thx

Comment 20 Bill Nottingham 2011-11-16 18:40:53 UTC
That's invalid syntax.

"#!/bin/bash" needs to be the *first* line of the script.

Comment 21 John Mev 2011-11-16 22:28:06 UTC
In fact it would be better to continue to ship this file .

Thx anyway

Comment 22 DiEOrLivE 2011-11-17 15:53:33 UTC
>"#!/bin/bash" needs to be the *first* line of the script.

Now it works for me! Thanks!


Note You need to log in before you can comment on or make changes to this bug.