Bug 1027605

patches merged upstream on master and 3.3 branches

For testing this:

ensure you've less than 35554432 as kernel.shmmax:
if you have:
# sysctl kernel.shmmax
kernel.shmmax = 35554432

you need to lower it:
sysctl kernel.shmmax=30000000

then run setup:
it will create /etc/sysctl.d/ovirt-engine.conf

at the end of the setup you should have 
# sysctl kernel.shmmax
kernel.shmmax = 35554432
(or higher if some other conf is raising it)

run engine-cleanup:
it should say:

The file /etc/sysctl.d/ovirt-engine.conf configuring sysctl kernel.shmmax has been left in place in order to allow postgres to run at next reboot.
If you want to restore default kernel.shmmax  value please remove the file and reboot.

and the file /etc/sysctl.d/ovirt-engine.conf should still exists.

Impossible to verify on RHEL 6.5 because if kernel.shmmax is below acceptable values for postgresql, engine-setup fails. If kernel.shmmax value is OK for postgresql then no /etc/sysctl.d/ovirt-engine.conf is created.

BZ1033043 created for kernel.shmmax check before starting postgresql during engine-setup.

Something is odd here. If kernel.shmmax value _is_ OK, then empty sysctl.d dir is created but there's no file.

If kernel.shmmax value is _NOT_ OK, then engine-setup fails, see BZ1033043.

Anyway, the check seems to take care only about runtime kernel.shmmax value, i still have too low value in sysctl.conf.

# sysctl kernel.shmmax                                               
kernel.shmmax = 39280640

# grep -R kernel.shmmax /etc/sysctl*                                                                                                         
kernel.shmmax = 30000000

This BZ is un-verificable now.

(In reply to Jiri Belka from comment #5)
> Impossible to verify on RHEL 6.5 because if kernel.shmmax is below
> acceptable values for postgresql, engine-setup fails. If kernel.shmmax value
> is OK for postgresql then no /etc/sysctl.d/ovirt-engine.conf is created.

If this is true, that's a (serious) regression which should probably block 3.3.1.

Before this fix, if kernel.shmmax was low the sysctl.d file was created and the value was raised, so engine-setup succeeded.

(In reply to Bob Doolittle from comment #8)
> (In reply to Jiri Belka from comment #5)
> > Impossible to verify on RHEL 6.5 because if kernel.shmmax is below
> > acceptable values for postgresql, engine-setup fails. If kernel.shmmax value
> > is OK for postgresql then no /etc/sysctl.d/ovirt-engine.conf is created.
> 
> If this is true, that's a (serious) regression which should probably block
> 3.3.1.
> 
> Before this fix, if kernel.shmmax was low the sysctl.d file was created and
> the value was raised, so engine-setup succeeded.

Bob, Jiri, this is as it was before the patch.
The status here is:

 1) On RHEL-6.5 posgresql requires more than 35554432 in shmmax.
 2) On RHEL-6.5, defaul shmmax is 68719476736
 3) Jiri set shmmax manually to 30000000
 4) engine-setup failed running postgres so the transaction has been aborted reverting the write of /etc/sysctl.d/ovirt-engine.conf
 5) runtime still have 35554432 becuase it's set before trying to run postgresql.

I'll push a patch rising shmmax to something more than 39280640, let's round to 40Mb, 41943040.

I see, thanks Sandro. It's really disgusting that engine-setup has to guess what postgresql wants.

If postgresql requires a higher value of shmmax, it should be postgresql's responsibility to do something like this. Consumers of postgresql should not have to guess, and try to take care of it. Besides the risks around such guesses, there can also arise conflicts, such as were discussed earlier in this and the related oVirt bug comments. All of those issues would be cleaned up if postgresql did the right thing for itself upon config/unconfig (or install/uninstall).

You do not need to guess. You can (somehow) counted that from postgresql.conf. But I'm not DBA.

(In reply to Jiri Belka from comment #12)
> You do not need to guess. You can (somehow) counted that from
> postgresql.conf. But I'm not DBA.

That's no better, IMO. If postgresql has a requirement, it should be it's responsibility to make sure the requirement is met, not the responsibility of every and all consumers of postgresql.

For example, all of this dance could be avoided, if postgresql simply raised the runtime value to what it's configured to use, at startup time. Nobody would have to be changing system parameters to make sure that the right things happen on the next reboot, and nobody would be having to guess what the right thing to do is upon uninstall/unconfig of a consumer.

My opinion is that all this is stupid. I would prefer if it would fail with nice error output (to let sysadmin to repair it) instead of doing some magic in the background. But this is not enough enterprise style. Thus, RHEVM is doing all dance in the background. I think postgresql is OK, it fails and output in the log is clear.

On reflection there's some truth to what you say: shmmax should mean something, and apps should not change it on demand. But at the same time only postgresql knows its requirements well. RHEVM is doing the wrong dance moves, because it doesn't (can't?) know what postgresql wants. It is just guessing. We may raise the value to deal with the new default, but as soon as the default changes again, we could have another bug just like this.

*Can* we determine postgresql's requirements from the conf file? Or is there a hard-coded default within the program? There should be some way to determine its requirements, and meet them (perhaps after prompting the admin whether they want the shmmax raised to meet the need). If you can't easily tell from the conf file, there should be an option to postgresql to report its requirement (or maybe some parseable failure message that reports the failed requirement).

(In reply to Jiri Belka from comment #15)
> My opinion is that all this is stupid. I would prefer if it would fail with
> nice error output (to let sysadmin to repair it) instead of doing some magic
> in the background. But this is not enough enterprise style. Thus, RHEVM is
> doing all dance in the background. I think postgresql is OK, it fails and
> output in the log is clear.

We're trying to be nice to admins coming from windows environments not really understanding what those configs are. However, on RHEL the default is high enough to just skip this whole thing. I think we should have a separate bug for discussing a change in how the shmmax has to be determined. Here we're trying to avoid the deletion of a "parachute" config file on cleanup.

Is there a problem with setting shmmax "too high"? I now searched a bit and could not find evidence for that.

One might claim that the real bug is also not in postgresql but in the kernel - that there should really be no such maximum.

patches merged on upstream master and 3.3 branch.

Didi: I don't know if there may be problems setting shmmax "too high". For now just raised to 40Mb shmmax.

This overriding of kernel.shmmax conflicts with installed tuned. In fact, tuned has a service ktune which loads (for me oddly thus I submited BZ1036049) reversed sysctl rules and thus sets back "original" kernel.shmmax value from sysctl.conf which was overriden by /etc/sysctl.d/ovirt-engine.conf.

# grep SYSCTL_ /etc/sysconfig/ktune
SYSCTL_POST="/etc/sysctl.d/* /etc/sysctl.conf"

Otherwise ok is25.

(If one has tuned installed a workaround is to switch order in /etc/sysctl/ktune or comment that line out.)

Closing - RHEV 3.3 Released

Closing - RHEV 3.3 Released

Closing - RHEV 3.3 Released