+++ This bug was initially created as a clone of Bug #1026926 +++
Description of problem:
If you run engine-setup; engine-cleanup; engine-setup (which might easily occur due to an error in the initial configuration), the final engine-setup may not properly store a persistent configuration for shmmax in sysctl, so that a subsequent reboot will cause postgresql to fail to come up so oVirt is non-operational.
This is because when engine-setup decides whether or not to store a kernel.shmmax value in sysctl.d/ovirt-engine.conf it checks the current runtime value of shmmax. So you can wind up with the following scenario:
1. setup detects default/low runtime value, and stores kernel.shmmax in ovirt-engine.conf
2. Cleanup removes the file (but does not adjust the current runtime value)
3. 2nd setup detects adequate runtime value, and does not write a new ovirt-engine.conf
4. Upon the next reboot, the system comes up with the default/low value
Postgresql will then be non-operational.
Version-Release number of selected component (if applicable):
Steps to Reproduce:
oVirt fails to come up cleanly, due to postgresql failure to start due to error allocating shm segment. You may find a general postgresql error in syslog. You may find more details via 'systemctl status postgresql'.
--- Additional comment from Bob Doolittle on 2013-11-05 11:42:15 EST ---
Note that determining an appropriate value to set in engine-cleanup is problematic.
The simplest approach would be to set shmmax to the system default, but:
1. system default may change across future kernel releases
2. system default may have been altered prior to engine-cleanup, to meet requirements unrelated to oVirt
Another approach might be to store the value prior to engine-setup, and restore that value during engine-cleanup. However this is still vulnerable to administrative changes which occurred after engine-setup and before engine-cleanup.
I can't think of a general solution to this, other than requiring a reboot after engine-cleanup (i.e. don't even try to set a runtime value, simply remove ovirt-engine.conf and instruct the administrator to reboot after cleanup completion). That's not pretty, unfortunately. We'd sort of like the kernel to record a stack of runtime shmmax update operations, so that we could replay it without the oVirt change :)
patches merged upstream on master and 3.3 branches
For testing this:
ensure you've less than 35554432 as kernel.shmmax:
if you have:
# sysctl kernel.shmmax
kernel.shmmax = 35554432
you need to lower it:
then run setup:
it will create /etc/sysctl.d/ovirt-engine.conf
at the end of the setup you should have
# sysctl kernel.shmmax
kernel.shmmax = 35554432
(or higher if some other conf is raising it)
it should say:
The file /etc/sysctl.d/ovirt-engine.conf configuring sysctl kernel.shmmax has been left in place in order to allow postgres to run at next reboot.
If you want to restore default kernel.shmmax value please remove the file and reboot.
and the file /etc/sysctl.d/ovirt-engine.conf should still exists.
Impossible to verify on RHEL 6.5 because if kernel.shmmax is below acceptable values for postgresql, engine-setup fails. If kernel.shmmax value is OK for postgresql then no /etc/sysctl.d/ovirt-engine.conf is created.
BZ1033043 created for kernel.shmmax check before starting postgresql during engine-setup.
Something is odd here. If kernel.shmmax value _is_ OK, then empty sysctl.d dir is created but there's no file.
If kernel.shmmax value is _NOT_ OK, then engine-setup fails, see BZ1033043.
Anyway, the check seems to take care only about runtime kernel.shmmax value, i still have too low value in sysctl.conf.
# sysctl kernel.shmmax
kernel.shmmax = 39280640
# grep -R kernel.shmmax /etc/sysctl*
kernel.shmmax = 30000000
This BZ is un-verificable now.
(In reply to Jiri Belka from comment #5)
> Impossible to verify on RHEL 6.5 because if kernel.shmmax is below
> acceptable values for postgresql, engine-setup fails. If kernel.shmmax value
> is OK for postgresql then no /etc/sysctl.d/ovirt-engine.conf is created.
If this is true, that's a (serious) regression which should probably block 3.3.1.
Before this fix, if kernel.shmmax was low the sysctl.d file was created and the value was raised, so engine-setup succeeded.
(In reply to Bob Doolittle from comment #8)
> (In reply to Jiri Belka from comment #5)
> > Impossible to verify on RHEL 6.5 because if kernel.shmmax is below
> > acceptable values for postgresql, engine-setup fails. If kernel.shmmax value
> > is OK for postgresql then no /etc/sysctl.d/ovirt-engine.conf is created.
> If this is true, that's a (serious) regression which should probably block
> Before this fix, if kernel.shmmax was low the sysctl.d file was created and
> the value was raised, so engine-setup succeeded.
Bob, Jiri, this is as it was before the patch.
The status here is:
1) On RHEL-6.5 posgresql requires more than 35554432 in shmmax.
2) On RHEL-6.5, defaul shmmax is 68719476736
3) Jiri set shmmax manually to 30000000
4) engine-setup failed running postgres so the transaction has been aborted reverting the write of /etc/sysctl.d/ovirt-engine.conf
5) runtime still have 35554432 becuase it's set before trying to run postgresql.
I'll push a patch rising shmmax to something more than 39280640, let's round to 40Mb, 41943040.
I see, thanks Sandro. It's really disgusting that engine-setup has to guess what postgresql wants.
If postgresql requires a higher value of shmmax, it should be postgresql's responsibility to do something like this. Consumers of postgresql should not have to guess, and try to take care of it. Besides the risks around such guesses, there can also arise conflicts, such as were discussed earlier in this and the related oVirt bug comments. All of those issues would be cleaned up if postgresql did the right thing for itself upon config/unconfig (or install/uninstall).
You do not need to guess. You can (somehow) counted that from postgresql.conf. But I'm not DBA.
(In reply to Jiri Belka from comment #12)
> You do not need to guess. You can (somehow) counted that from
> postgresql.conf. But I'm not DBA.
That's no better, IMO. If postgresql has a requirement, it should be it's responsibility to make sure the requirement is met, not the responsibility of every and all consumers of postgresql.
For example, all of this dance could be avoided, if postgresql simply raised the runtime value to what it's configured to use, at startup time. Nobody would have to be changing system parameters to make sure that the right things happen on the next reboot, and nobody would be having to guess what the right thing to do is upon uninstall/unconfig of a consumer.
My opinion is that all this is stupid. I would prefer if it would fail with nice error output (to let sysadmin to repair it) instead of doing some magic in the background. But this is not enough enterprise style. Thus, RHEVM is doing all dance in the background. I think postgresql is OK, it fails and output in the log is clear.
On reflection there's some truth to what you say: shmmax should mean something, and apps should not change it on demand. But at the same time only postgresql knows its requirements well. RHEVM is doing the wrong dance moves, because it doesn't (can't?) know what postgresql wants. It is just guessing. We may raise the value to deal with the new default, but as soon as the default changes again, we could have another bug just like this.
*Can* we determine postgresql's requirements from the conf file? Or is there a hard-coded default within the program? There should be some way to determine its requirements, and meet them (perhaps after prompting the admin whether they want the shmmax raised to meet the need). If you can't easily tell from the conf file, there should be an option to postgresql to report its requirement (or maybe some parseable failure message that reports the failed requirement).
(In reply to Jiri Belka from comment #15)
> My opinion is that all this is stupid. I would prefer if it would fail with
> nice error output (to let sysadmin to repair it) instead of doing some magic
> in the background. But this is not enough enterprise style. Thus, RHEVM is
> doing all dance in the background. I think postgresql is OK, it fails and
> output in the log is clear.
We're trying to be nice to admins coming from windows environments not really understanding what those configs are. However, on RHEL the default is high enough to just skip this whole thing. I think we should have a separate bug for discussing a change in how the shmmax has to be determined. Here we're trying to avoid the deletion of a "parachute" config file on cleanup.
Is there a problem with setting shmmax "too high"? I now searched a bit and could not find evidence for that.
One might claim that the real bug is also not in postgresql but in the kernel - that there should really be no such maximum.
patches merged on upstream master and 3.3 branch.
Didi: I don't know if there may be problems setting shmmax "too high". For now just raised to 40Mb shmmax.
This overriding of kernel.shmmax conflicts with installed tuned. In fact, tuned has a service ktune which loads (for me oddly thus I submited BZ1036049) reversed sysctl rules and thus sets back "original" kernel.shmmax value from sysctl.conf which was overriden by /etc/sysctl.d/ovirt-engine.conf.
# grep SYSCTL_ /etc/sysconfig/ktune
Otherwise ok is25.
(If one has tuned installed a workaround is to switch order in /etc/sysctl/ktune or comment that line out.)
Closing - RHEV 3.3 Released