cpuplugd service failed to start on LPAR:

[root@s83lp68kvm ~]# systemctl status cpuplugd
? cpuplugd.service - LSB: Start the cpu and memory hotplug daemon for Linux on System z
   Loaded: loaded (/etc/rc.d/init.d/cpuplugd; bad; vendor preset: disabled)
   Active: inactive (dead)
     Docs: man:systemd-sysv-generator(8)
[root@s83lp68kvm ~]# systemctl start cpuplugd
Job for cpuplugd.service failed because the control process exited with error code. See "systemctl status cpuplugd.service" and "journalctl -xe" for details.
[root@s83lp68kvm ~]# systemctl status cpuplugd -l
? cpuplugd.service - LSB: Start the cpu and memory hotplug daemon for Linux on System z
   Loaded: loaded (/etc/rc.d/init.d/cpuplugd; bad; vendor preset: disabled)
   Active: failed (Result: exit-code) since Wed 2016-09-28 14:39:32 CEST; 30s ago
     Docs: man:systemd-sysv-generator(8)
  Process: 2314 ExecStart=/etc/rc.d/init.d/cpuplugd start (code=exited, status=150)

Sep 28 14:39:32 s83lp68kvm systemd[1]: Starting LSB: Start the cpu and memory hotplug daemon for Linux on System z...
Sep 28 14:39:32 s83lp68kvm systemd[1]: cpuplugd.service: control process exited, code=exited status=150
Sep 28 14:39:32 s83lp68kvm systemd[1]: Failed to start LSB: Start the cpu and memory hotplug daemon for Linux on System z.
Sep 28 14:39:32 s83lp68kvm systemd[1]: Unit cpuplugd.service entered failed state.
Sep 28 14:39:32 s83lp68kvm systemd[1]: cpuplugd.service failed.
[root@s83lp68kvm ~]#

The problem is in the cpuplugd init script /etc/init.d/cpuplugd, which does not look like an init script for a service from the s390-tools package.
See the attached cpuplugd init script.
The problematic lines are:
    # explicitly load cmm module, needed for memory hotplug
    modprobe $module > /dev/null 2>&1 || exit 150

This modprobe will always fail on LPAR, and there is no error message for this in the init script to explain anything, which is very confusing.

Created attachment 1205927 [details]
cpuplugd init script

------- Comment From thorsten.diehl.com 2016-10-18 05:05 EDT-------
I ran into this problem yesterday and found this bug already opened.

@Red Hat: Please consider my attached patch, which I already tested for
a) z/VM with successful load of cmm
b) z/VM with unsuccessful load of cmm
c) LPAR

Created attachment 1211645 [details]
patch to make cpuplugd startable in LPAR also

Hi Thorsten, thanks for the patch, it's how I intended to fix it. Could you please retry my version with "systemd-detect-virt" instead of grep? It's RHEL-7 and we can rely on systemd.

--- a/cpuplugd.initd
+++ b/cpuplugd.initd
@@ -32,8 +32,8 @@ start() {
     [ -f $config ] || exit 6
     [ `id -u` -eq 0 ] || exit 4
 
-    # explicitly load cmm module, needed for memory hotplug
-    modprobe $module > /dev/null 2>&1 || exit 150
+    # explicitly load cmm module on z/VM, needed for memory hotplug
+    [ `systemd-detect-virt` = 'zvm' ] && { modprobe $module > /dev/null 2>&1 || exit 150; }
 
     echo -n $"Starting $prog: "
     daemon $exec -c $config

------- Comment From geraldsc.com 2016-10-18 05:20 EDT-------
(In reply to comment #7)
> Created attachment 113096 [details]
> patch to make cpuplugd startable in LPAR also

I don't think it is correct to load the cmm module from the init script at all. It certainly is not correct to exit if module load fails, even on z/VM, because cmm is only needed for the memory hotplug (ballooning) rules of cpuplugd, and those rules are disabled by default. So, exiting when cmm cannot be loaded will also prevent the CPU hotplug feature, even if no memory hotplug rules will be used.

If anyone wants to use the MEM(UN)PLUG rules, the cmm module needs to be loaded manually upfront, as described in the documentation. The only way to do this correctly from the init script would be to parse the cpuplugd config file and check for the MEM(UN)PLUG rules, and of course also the z/VM check as in this patch, but I don't think it is worth the effort.

So I would suggest to simply remove the cmm module loading code from the init script, or at least remove the "exit 150" and just try to load it silently, maybe with the z/VM check as done in this patch.

I have two questions
- what are drawbacks if the CMM module is loaded and not used?
- shouldn't the cpuplugd daemon itself load the module if its config contains any memory hotplug rules?

For RHEL we want a conservative approach, but we can experiment with a better solution in Fedora using latest upstream s390-tools.

------- Comment From brueckner.com 2016-10-18 05:52 EDT-------
(In reply to comment #9)
> Hi Thorsten, thanks for the patch, it's how I intended to fix it. Could you
> please retry my version with "systemd-detect-virt" instead of grep? It's
> RHEL-7 and we can rely on systemd.
>
> --- a/cpuplugd.initd
> +++ b/cpuplugd.initd
> @@ -32,8 +32,8 @@ start() {
> [ -f $config ] || exit 6
> [ `id -u` -eq 0 ] || exit 4
>
> -    # explicitly load cmm module, needed for memory hotplug
> -    modprobe $module > /dev/null 2>&1 || exit 150
> +    # explicitly load cmm module on z/VM, needed for memory hotplug
> +    [ `systemd-detect-virt` = 'zvm' ] && { modprobe $module > /dev/null
> 2>&1 || exit 150; }
>
> echo -n $"Starting $prog: "
> daemon $exec -c $config

Another idea what be to load the cmm module outside the context of the cpuplugd using systemd'd module loading as described in:

https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/System_Administrators_Guide/sec-Persistent_Module_Loading.html

(In reply to IBM Bug Proxy from comment #12)
> ------- Comment From brueckner.com 2016-10-18 05:52 EDT-------
> (In reply to comment #9)
> > Hi Thorsten, thanks for the patch, it's how I intended to fix it. Could you
> > please retry my version with "systemd-detect-virt" instead of grep? It's
> > RHEL-7 and we can rely on systemd.
> >
> > --- a/cpuplugd.initd
> > +++ b/cpuplugd.initd
> > @@ -32,8 +32,8 @@ start() {
> > [ -f $config ] || exit 6
> > [ `id -u` -eq 0 ] || exit 4
> >
> > -    # explicitly load cmm module, needed for memory hotplug
> > -    modprobe $module > /dev/null 2>&1 || exit 150
> > +    # explicitly load cmm module on z/VM, needed for memory hotplug
> > +    [ `systemd-detect-virt` = 'zvm' ] && { modprobe $module > /dev/null
> > 2>&1 || exit 150; }
> >
> > echo -n $"Starting $prog: "
> > daemon $exec -c $config
> 
> Another idea what be to load the cmm module outside the context of the
> cpuplugd using systemd'd module loading as described in:
> 
> https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/
> html/System_Administrators_Guide/sec-Persistent_Module_Loading.html

but this is unconditional load if I read it correctly, another option could be load by udev like the SCSI sg module or create a special systemd unit that would be a dependency of the cpluplugd service or ...

------- Comment From geraldsc.com 2016-10-18 06:36 EDT-------
(In reply to comment #11)
> I have two questions
> - what are drawbacks if the CMM module is loaded and not used?
> - shouldn't the cpuplugd daemon itself load the module if its config
> contains any memory hotplug rules?
> For RHEL we want a conservative approach, but we can experiment with a
> better solution in Fedora using latest upstream s390-tools.

There are no drawbacks if the CMM module is loaded and not used. The only issue that we have here is the "exit 150", which prevents normal CPU hotplug operation on LPAR (and also on z/VM, if for any reason the load would fail there). So an unconditional (and unchecked) load from the init script seems like the easiest way to go.

Not sure why we decided not to load the module from the daemon itself, but it "works as designed" and documented. It probably was considered to be something that can be done easily outside the daemon.

------- Comment From thorsten.diehl.com 2016-10-18 07:18 EDT-------
(In reply to comment #9)
> Hi Thorsten, thanks for the patch, it's how I intended to fix it. Could you
> please retry my version with "systemd-detect-virt" instead of grep? It's
> RHEL-7 and we can rely on systemd.
> --- a/cpuplugd.initd
> +++ b/cpuplugd.initd
> @@ -32,8 +32,8 @@ start() {
> [ -f $config ] || exit 6
> [ `id -u` -eq 0 ] || exit 4
>
> -    # explicitly load cmm module, needed for memory hotplug
> -    modprobe $module > /dev/null 2>&1 || exit 150
> +    # explicitly load cmm module on z/VM, needed for memory hotplug
> +    [ `systemd-detect-virt` = 'zvm' ] && { modprobe $module > /dev/null
> 2>&1 || exit 150; }
>
> echo -n $"Starting $prog: "
> daemon $exec -c $config

Hi Dan,

verified, this works the same way, as expected.

Reading the other discussions, it's (for sure) up to you how to implement a solution. I have no preference at the moment.

------- Comment From ruddk.com 2016-12-27 13:22 EDT-------
This bug appears to have fallen between the cracks.  Did this ever get resolved in an official update?

(In reply to IBM Bug Proxy from comment #16)
> ------- Comment From ruddk.com 2016-12-27 13:22 EDT-------
> This bug appears to have fallen between the cracks.  Did this ever get
> resolved in an official update?

The fix is planned for 7.4.

fyi ... a related bugzilla is now also open for RHEL6.8 in
LTC bug 1416831 - RHEL6.8 - cpuplugd failed to start on LPAR (s390utils/s390-tools) ...

------- Comment From Irina.Yashina1.com 2017-05-04 10:39 EDT-------
The fix was verified in RHEL7.4 Alpha, cpuplugd successfully started on LPAR.
The problem has been solved, thank you.

Verified by partner. Patch confirmed in GIT (the one from Comment #9, not the version from the attachment). Setting verified.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2017:2045