Bug 1027605 - engine-cleanup (and possible engine-setup) does not affect runtime value of shmmax
engine-cleanup (and possible engine-setup) does not affect runtime value of s...
Status: CLOSED CURRENTRELEASE
Product: Red Hat Enterprise Virtualization Manager
Classification: Red Hat
Component: ovirt-engine-setup (Show other bugs)
3.3.0
Unspecified Unspecified
medium Severity high
: ---
: 3.3.0
Assigned To: Sandro Bonazzola
Jiri Belka
integration
: Triaged
Depends On: 1026926
Blocks: 3.3snap3
  Show dependency treegraph
 
Reported: 2013-11-07 03:19 EST by Sandro Bonazzola
Modified: 2014-01-21 17:27 EST (History)
11 users (show)

See Also:
Fixed In Version: is25
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: 1026926
Environment:
Last Closed:
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)


External Trackers
Tracker ID Priority Status Summary Last Updated
oVirt gerrit 21139 None None None Never
oVirt gerrit 21283 None None None Never
oVirt gerrit 21512 None None None Never
oVirt gerrit 21533 None None None Never

  None (edit)
Description Sandro Bonazzola 2013-11-07 03:19:29 EST
+++ This bug was initially created as a clone of Bug #1026926 +++

Description of problem:
If you run engine-setup; engine-cleanup; engine-setup (which might easily occur due to an error in the initial configuration), the final engine-setup may not properly store a persistent configuration for shmmax in sysctl, so that a subsequent reboot will cause postgresql to fail to come up so oVirt is non-operational.

This is because when engine-setup decides whether or not to store a kernel.shmmax value in sysctl.d/ovirt-engine.conf it checks the current runtime value of shmmax. So you can wind up with the following scenario:

1. setup detects default/low runtime value, and stores kernel.shmmax in ovirt-engine.conf
2. Cleanup removes the file (but does not adjust the current runtime value)
3. 2nd setup detects adequate runtime value, and does not write a new ovirt-engine.conf
4. Upon the next reboot, the system comes up with the default/low value

Postgresql will then be non-operational.

Version-Release number of selected component (if applicable):
3.3

How reproducible:
Completely

Steps to Reproduce:
1. engine-setup
2. engine-cleanup
3. engine-setup
4. reboot

Actual results:
oVirt fails to come up cleanly, due to postgresql failure to start due to error allocating shm segment. You may find a general postgresql error in syslog. You may find more details via 'systemctl status postgresql'.

Expected results:
oVirt operational


Additional info:

--- Additional comment from Bob Doolittle on 2013-11-05 11:42:15 EST ---

Note that determining an appropriate value to set in engine-cleanup is problematic.

The simplest approach would be to set shmmax to the system default, but:
1. system default may change across future kernel releases
2. system default may have been altered prior to engine-cleanup, to meet requirements unrelated to oVirt

Another approach might be to store the value prior to engine-setup, and restore that value during engine-cleanup. However this is still vulnerable to administrative changes which occurred after engine-setup and before engine-cleanup.

I can't think of a general solution to this, other than requiring a reboot after engine-cleanup (i.e. don't even try to set a runtime value, simply remove ovirt-engine.conf and instruct the administrator to reboot after cleanup completion). That's not pretty, unfortunately. We'd sort of like the kernel to record a stack of runtime shmmax update operations, so that we could replay it without the oVirt change :)
Comment 3 Sandro Bonazzola 2013-11-15 02:12:22 EST
patches merged upstream on master and 3.3 branches
Comment 4 Sandro Bonazzola 2013-11-21 07:29:28 EST
For testing this:

ensure you've less than 35554432 as kernel.shmmax:
if you have:
# sysctl kernel.shmmax
kernel.shmmax = 35554432

you need to lower it:
sysctl kernel.shmmax=30000000

then run setup:
it will create /etc/sysctl.d/ovirt-engine.conf

at the end of the setup you should have 
# sysctl kernel.shmmax
kernel.shmmax = 35554432
(or higher if some other conf is raising it)

run engine-cleanup:
it should say:

The file /etc/sysctl.d/ovirt-engine.conf configuring sysctl kernel.shmmax has been left in place in order to allow postgres to run at next reboot.
If you want to restore default kernel.shmmax  value please remove the file and reboot.

and the file /etc/sysctl.d/ovirt-engine.conf should still exists.
Comment 5 Jiri Belka 2013-11-21 08:39:24 EST
Impossible to verify on RHEL 6.5 because if kernel.shmmax is below acceptable values for postgresql, engine-setup fails. If kernel.shmmax value is OK for postgresql then no /etc/sysctl.d/ovirt-engine.conf is created.
Comment 6 Jiri Belka 2013-11-21 08:40:39 EST
BZ1033043 created for kernel.shmmax check before starting postgresql during engine-setup.
Comment 7 Jiri Belka 2013-11-21 08:55:49 EST
Something is odd here. If kernel.shmmax value _is_ OK, then empty sysctl.d dir is created but there's no file.

If kernel.shmmax value is _NOT_ OK, then engine-setup fails, see BZ1033043.

Anyway, the check seems to take care only about runtime kernel.shmmax value, i still have too low value in sysctl.conf.

# sysctl kernel.shmmax                                               
kernel.shmmax = 39280640

# grep -R kernel.shmmax /etc/sysctl*                                                                                                         
kernel.shmmax = 30000000

This BZ is un-verificable now.
Comment 8 Bob Doolittle 2013-11-21 09:05:03 EST
(In reply to Jiri Belka from comment #5)
> Impossible to verify on RHEL 6.5 because if kernel.shmmax is below
> acceptable values for postgresql, engine-setup fails. If kernel.shmmax value
> is OK for postgresql then no /etc/sysctl.d/ovirt-engine.conf is created.

If this is true, that's a (serious) regression which should probably block 3.3.1.

Before this fix, if kernel.shmmax was low the sysctl.d file was created and the value was raised, so engine-setup succeeded.
Comment 9 Sandro Bonazzola 2013-11-21 09:35:53 EST
(In reply to Bob Doolittle from comment #8)
> (In reply to Jiri Belka from comment #5)
> > Impossible to verify on RHEL 6.5 because if kernel.shmmax is below
> > acceptable values for postgresql, engine-setup fails. If kernel.shmmax value
> > is OK for postgresql then no /etc/sysctl.d/ovirt-engine.conf is created.
> 
> If this is true, that's a (serious) regression which should probably block
> 3.3.1.
> 
> Before this fix, if kernel.shmmax was low the sysctl.d file was created and
> the value was raised, so engine-setup succeeded.

Bob, Jiri, this is as it was before the patch.
The status here is:

 1) On RHEL-6.5 posgresql requires more than 35554432 in shmmax.
 2) On RHEL-6.5, defaul shmmax is 68719476736
 3) Jiri set shmmax manually to 30000000
 4) engine-setup failed running postgres so the transaction has been aborted reverting the write of /etc/sysctl.d/ovirt-engine.conf
 5) runtime still have 35554432 becuase it's set before trying to run postgresql.
Comment 10 Sandro Bonazzola 2013-11-21 09:42:22 EST
I'll push a patch rising shmmax to something more than 39280640, let's round to 40Mb, 41943040.
Comment 11 Bob Doolittle 2013-11-21 09:49:30 EST
I see, thanks Sandro. It's really disgusting that engine-setup has to guess what postgresql wants.

If postgresql requires a higher value of shmmax, it should be postgresql's responsibility to do something like this. Consumers of postgresql should not have to guess, and try to take care of it. Besides the risks around such guesses, there can also arise conflicts, such as were discussed earlier in this and the related oVirt bug comments. All of those issues would be cleaned up if postgresql did the right thing for itself upon config/unconfig (or install/uninstall).
Comment 12 Jiri Belka 2013-11-21 09:51:52 EST
You do not need to guess. You can (somehow) counted that from postgresql.conf. But I'm not DBA.
Comment 13 Bob Doolittle 2013-11-21 09:53:07 EST
(In reply to Jiri Belka from comment #12)
> You do not need to guess. You can (somehow) counted that from
> postgresql.conf. But I'm not DBA.

That's no better, IMO. If postgresql has a requirement, it should be it's responsibility to make sure the requirement is met, not the responsibility of every and all consumers of postgresql.
Comment 14 Bob Doolittle 2013-11-21 09:56:02 EST
For example, all of this dance could be avoided, if postgresql simply raised the runtime value to what it's configured to use, at startup time. Nobody would have to be changing system parameters to make sure that the right things happen on the next reboot, and nobody would be having to guess what the right thing to do is upon uninstall/unconfig of a consumer.
Comment 15 Jiri Belka 2013-11-21 10:00:47 EST
My opinion is that all this is stupid. I would prefer if it would fail with nice error output (to let sysadmin to repair it) instead of doing some magic in the background. But this is not enough enterprise style. Thus, RHEVM is doing all dance in the background. I think postgresql is OK, it fails and output in the log is clear.
Comment 16 Bob Doolittle 2013-11-21 10:17:36 EST
On reflection there's some truth to what you say: shmmax should mean something, and apps should not change it on demand. But at the same time only postgresql knows its requirements well. RHEVM is doing the wrong dance moves, because it doesn't (can't?) know what postgresql wants. It is just guessing. We may raise the value to deal with the new default, but as soon as the default changes again, we could have another bug just like this.

*Can* we determine postgresql's requirements from the conf file? Or is there a hard-coded default within the program? There should be some way to determine its requirements, and meet them (perhaps after prompting the admin whether they want the shmmax raised to meet the need). If you can't easily tell from the conf file, there should be an option to postgresql to report its requirement (or maybe some parseable failure message that reports the failed requirement).
Comment 17 Sandro Bonazzola 2013-11-21 10:34:27 EST
(In reply to Jiri Belka from comment #15)
> My opinion is that all this is stupid. I would prefer if it would fail with
> nice error output (to let sysadmin to repair it) instead of doing some magic
> in the background. But this is not enough enterprise style. Thus, RHEVM is
> doing all dance in the background. I think postgresql is OK, it fails and
> output in the log is clear.

We're trying to be nice to admins coming from windows environments not really understanding what those configs are. However, on RHEL the default is high enough to just skip this whole thing. I think we should have a separate bug for discussing a change in how the shmmax has to be determined. Here we're trying to avoid the deletion of a "parachute" config file on cleanup.
Comment 18 Yedidyah Bar David 2013-11-21 15:17:18 EST
Is there a problem with setting shmmax "too high"? I now searched a bit and could not find evidence for that.

One might claim that the real bug is also not in postgresql but in the kernel - that there should really be no such maximum.
Comment 19 Sandro Bonazzola 2013-11-25 03:55:53 EST
patches merged on upstream master and 3.3 branch.

Didi: I don't know if there may be problems setting shmmax "too high". For now just raised to 40Mb shmmax.
Comment 20 Jiri Belka 2013-11-29 04:38:55 EST
This overriding of kernel.shmmax conflicts with installed tuned. In fact, tuned has a service ktune which loads (for me oddly thus I submited BZ1036049) reversed sysctl rules and thus sets back "original" kernel.shmmax value from sysctl.conf which was overriden by /etc/sysctl.d/ovirt-engine.conf.

# grep SYSCTL_ /etc/sysconfig/ktune
SYSCTL_POST="/etc/sysctl.d/* /etc/sysctl.conf"

Otherwise ok is25.

(If one has tuned installed a workaround is to switch order in /etc/sysctl/ktune or comment that line out.)
Comment 21 Itamar Heim 2014-01-21 17:22:44 EST
Closing - RHEV 3.3 Released
Comment 22 Itamar Heim 2014-01-21 17:23:52 EST
Closing - RHEV 3.3 Released
Comment 23 Itamar Heim 2014-01-21 17:27:40 EST
Closing - RHEV 3.3 Released

Note You need to log in before you can comment on or make changes to this bug.