Bug 1655902

Hi mezhang,

I believe this is expected behavior. Pacemaker changes the sysrq contents when it starts, to make sure it can panic the kernel in case of severe error, but it does not change the user's permanent configuration.

Is there some negative effect on the cluster from not having it set at boot time?

Hi Gaillot,

Many thanks for your update.

>Is there some negative effect on the cluster from not having it set at boot time?

Manually restart the cluster can cause sysrq be changed. I know this is the expected behavior to make sure it can panic kernel.

But the sysrq isn't changed at the boot time. After the boot time if I never restart the cluster the sysrq cannot be changed, so that it cannot panic kernel.
I don't think this is an expected behavior. 

Mengyi

(In reply to mezhang from comment #3)
> Hi Gaillot,
> 
> Many thanks for your update.
> 
> >Is there some negative effect on the cluster from not having it set at boot time?
> 
> Manually restart the cluster can cause sysrq be changed. I know this is the
> expected behavior to make sure it can panic kernel.
> 
> But the sysrq isn't changed at the boot time. After the boot time if I never
> restart the cluster the sysrq cannot be changed, so that it cannot panic
> kernel.
> I don't think this is an expected behavior. 
> 
> Mengyi

I'm not sure I understand. If the cluster isn't running, how could it panic the kernel?

If your goal is to be able to panic the kernel by other means when the cluster is not running, then the sysctl.conf change in the Description is appropriate.

Hi Gaillot,

I am sorry for the confusing description.

>I'm not sure I understand. If the cluster isn't running, how could it panic the kernel?

I mean that when reboot the system, cluster is started by default. In this situation sysrq should be changed. In other words, cluster is running for the boot. But sysrq is not changed to 136 because sysctl.conf is worked.

mengyi

(In reply to mezhang from comment #5)
> Hi Gaillot,
> 
> I am sorry for the confusing description.
> 
> >I'm not sure I understand. If the cluster isn't running, how could it panic the kernel?
> 
> I mean that when reboot the system, cluster is started by default. In this
> situation sysrq should be changed. In other words, cluster is running for
> the boot. But sysrq is not changed to 136 because sysctl.conf is worked.
> 
> mengyi

Ah, I see. I didn't realize the cluster was enabled at boot, sorry.

Pacemaker should set the sysrq in this case as well, but I suspect something later in the boot sequence may be resetting it.

Does your /etc/sysctl.conf or any .conf file in /etc/sysctl.d have an explicit setting for kernel.sysrq?

I see the standard file /usr/lib/sysctl.d/50-default.conf contains "kernel.sysrq = 16", so that may be where it is coming from.

To confirm whether this is the cause, please remove your workaround, run this, and reboot:

mkdir -p /etc/systemd/system/pacemaker.service.d/
cat >/etc/systemd/system/pacemaker.service.d/after-sysctl.conf <<EOF
[Unit]
After=systemd-sysctl
EOF

Your workaround is actually better, but this should help confirm whether my suspicion is correct.

I think I see a good way to fix this. Pacemaker calls its sysrq_init() function at start-up, and again just before panicking the host if the need arises. However sysrq_init() has a static variable to ensure it only modifies the kernel setting once. If we simply remove this control, it will reset the setting just before it is needed.

It would still be interesting to know if the unit override helps, but I think I will go with that fix. In the meantime, either your original workaround (preferred) or the unit override workaround (if it works) will be helpful.

Hi Gaillot,

Thanks you so much for your update.

>Does your /etc/sysctl.conf or any .conf file in /etc/sysctl.d have an explicit setting for kernel.sysrq?

yes. I configured kernel.sysrq in /etc/sysctl.conf explicitly.

>To confirm whether this is the cause, please remove your workaround, run this, and reboot:
>mkdir -p /etc/systemd/system/pacemaker.service.d/
>cat >/etc/systemd/system/pacemaker.service.d/after-sysctl.conf <<EOF
>[Unit]
>After=systemd-sysctl
>EOF

I added "After=systemd-sysctl" in pacemaker.service directly. It helps to fix the issue!

Mengyi

Hi Mengyi,

I'm having trouble reproducing the issue. I have a patch that reliably resets kernel.sysrq when needed, but as far as I can tell, setting kernel.sysrq has never been necessary for pacemaker, since it writes to /proc/sysrq-trigger with root privileges, which is not constrained by kernel.sysrq.

In your testing, was pacemaker actually unable to perform a reboot? What steps did you take to get that to happen?

Specifically, from the kernel "sysrq.txt" documentation: "Note that the value of /proc/sys/kernel/sysrq influences only the invocation via a keyboard. Invocation of any operation via /proc/sysrq-trigger is always allowed (by a user with admin privileges)."

When Pacemaker uses sysrq, it's always via /proc/sysrq-trigger as root, so I do not think that Pacemaker needs to set kernel.sysrq at all.

>In your testing, was pacemaker actually unable to perform a reboot? What steps did you take to get that to happen?

I am not sure if pacemaker perform a reboot correctly.

I just realized that the kernal.sysrq don't show the right output.
So I raise this bug ticket.

Great, thanks!

Based on my testing and the kernel documentation, pacemaker should never have changed kernel.sysrq at all. I will remove that code for a future release. Since the code doesn't really cause any problems, I'm going to close this bz. Thanks for reporting it, I am glad to know we don't need to change the system settings.