Bug 1026926 - engine-cleanup (and possible engine-setup) does not affect runtime value of shmmax
engine-cleanup (and possible engine-setup) does not affect runtime value of s...
Status: CLOSED CURRENTRELEASE
Product: oVirt
Classification: Community
Component: ovirt-engine-installer (Show other bugs)
3.3
Unspecified Unspecified
medium Severity high
: ---
: 3.3.2
Assigned To: Sandro Bonazzola
integration
:
Depends On:
Blocks: 1027605
  Show dependency treegraph
 
Reported: 2013-11-05 11:34 EST by Bob Doolittle
Modified: 2013-12-19 09:24 EST (History)
7 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 1027605 (view as bug list)
Environment:
Last Closed: 2013-12-19 09:24:35 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)


External Trackers
Tracker ID Priority Status Summary Last Updated
oVirt gerrit 21139 None None None Never
oVirt gerrit 21283 None None None Never

  None (edit)
Description Bob Doolittle 2013-11-05 11:34:14 EST
Description of problem:
If you run engine-setup; engine-cleanup; engine-setup (which might easily occur due to an error in the initial configuration), the final engine-setup may not properly store a persistent configuration for shmmax in sysctl, so that a subsequent reboot will cause postgresql to fail to come up so oVirt is non-operational.

This is because when engine-setup decides whether or not to store a kernel.shmmax value in sysctl.d/ovirt-engine.conf it checks the current runtime value of shmmax. So you can wind up with the following scenario:

1. setup detects default/low runtime value, and stores kernel.shmmax in ovirt-engine.conf
2. Cleanup removes the file (but does not adjust the current runtime value)
3. 2nd setup detects adequate runtime value, and does not write a new ovirt-engine.conf
4. Upon the next reboot, the system comes up with the default/low value

Postgresql will then be non-operational.

Version-Release number of selected component (if applicable):
3.3

How reproducible:
Completely

Steps to Reproduce:
1. engine-setup
2. engine-cleanup
3. engine-setup
4. reboot

Actual results:
oVirt fails to come up cleanly, due to postgresql failure to start due to error allocating shm segment. You may find a general postgresql error in syslog. You may find more details via 'systemctl status postgresql'.

Expected results:
oVirt operational


Additional info:
Comment 1 Bob Doolittle 2013-11-05 11:42:15 EST
Note that determining an appropriate value to set in engine-cleanup is problematic.

The simplest approach would be to set shmmax to the system default, but:
1. system default may change across future kernel releases
2. system default may have been altered prior to engine-cleanup, to meet requirements unrelated to oVirt

Another approach might be to store the value prior to engine-setup, and restore that value during engine-cleanup. However this is still vulnerable to administrative changes which occurred after engine-setup and before engine-cleanup.

I can't think of a general solution to this, other than requiring a reboot after engine-cleanup (i.e. don't even try to set a runtime value, simply remove ovirt-engine.conf and instruct the administrator to reboot after cleanup completion). That's not pretty, unfortunately. We'd sort of like the kernel to record a stack of runtime shmmax update operations, so that we could replay it without the oVirt change :)
Comment 2 Sandro Bonazzola 2013-11-11 04:40:46 EST
Proposed solutions:
1) don't remove shmmax configuration file on cleanup, just add a final note to the user to remove manually if needed and reboot.

this will leave existing runtime configuration and existing saved configuration as is, ensuring postgres will work also on reboot

2) save existing shmmax in /etc/sysctl.d/00-pre-ovirt-engine.conf and new values in /etc/sysctl.d/10-post-ovirt-engine.conf. On cleanup:
  - remove 10-post-ovirt-engine.conf
  - run sysctl --system
  - remove 00-pre-ovirt-engine.conf

this will lead to having postgres going down and maybe not working at reboot but will restore original values at runtime.

Note that just running sysctl --system doesn't reset the values to kernel defaults.

I personally prefer 1. What do you think?
Comment 3 Alon Bar-Lev 2013-11-11 04:44:55 EST
(In reply to Sandro Bonazzola from comment #2)
> I personally prefer 1. What do you think?

I agree... better to leave leftover then trying to revert to kernel defaults our-selves.

We put our file only if there is no other configuration of shmmax.
Comment 4 Yedidyah Bar David 2013-11-11 05:29:19 EST
(In reply to Alon Bar-Lev from comment #3)
> (In reply to Sandro Bonazzola from comment #2)
> > I personally prefer 1. What do you think?
> 
> I agree... better to leave leftover then trying to revert to kernel defaults
> our-selves.

I agree too.

> 
> We put our file only if there is no other configuration of shmmax.

We do? Not sure. Actually, we might get bitten if user has /etc/sysctl.d/zzz_shmmax with a low value - it will override our file. Perhaps we should also 'sysctl --system' and then check again, and alert/exit if value is still too low.
Comment 5 Alon Bar-Lev 2013-11-11 06:02:57 EST
(In reply to Yedidyah Bar David from comment #4)
> > 
> > We put our file only if there is no other configuration of shmmax.
> 
> We do? Not sure. Actually, we might get bitten if user has
> /etc/sysctl.d/zzz_shmmax with a low value - it will override our file.
> Perhaps we should also 'sysctl --system' and then check again, and
> alert/exit if value is still too low.

We can, however, this way we might override runtime tunning... there are components that tune system at runtime without persisting anything... reloading all may create undesired effect. Not sure what is better.
Comment 6 Sandro Bonazzola 2013-11-11 06:57:37 EST
(In reply to Alon Bar-Lev from comment #5)
> (In reply to Yedidyah Bar David from comment #4)
> > > 
> > > We put our file only if there is no other configuration of shmmax.
> > 
> > We do? Not sure. Actually, we might get bitten if user has
> > /etc/sysctl.d/zzz_shmmax with a low value - it will override our file.
> > Perhaps we should also 'sysctl --system' and then check again, and
> > alert/exit if value is still too low.
> 
> We can, however, this way we might override runtime tunning... there are
> components that tune system at runtime without persisting anything...
> reloading all may create undesired effect. Not sure what is better.

I think that if an admin is configuring sysctl with a zzz_shmmax, the admin is able fo solve shmmax issues himself.
However, the only way I can think of to cover all of them is to change the postgresql service for setting shmmax at runtime when it starts or create a service to do the same thing just before starting postgresql avoiding to have something to do with sysctl files at all.
Comment 7 Yedidyah Bar David 2013-11-11 09:22:16 EST
(In reply to Sandro Bonazzola from comment #6)
> (In reply to Alon Bar-Lev from comment #5)
> > (In reply to Yedidyah Bar David from comment #4)
> > > > 
> > > > We put our file only if there is no other configuration of shmmax.
> > > 
> > > We do? Not sure. Actually, we might get bitten if user has
> > > /etc/sysctl.d/zzz_shmmax with a low value - it will override our file.
> > > Perhaps we should also 'sysctl --system' and then check again, and
> > > alert/exit if value is still too low.
> > 
> > We can, however, this way we might override runtime tunning... there are
> > components that tune system at runtime without persisting anything...
> > reloading all may create undesired effect. Not sure what is better.
> 
> I think that if an admin is configuring sysctl with a zzz_shmmax, the admin
> is able fo solve shmmax issues himself.

Agreed.

> However, the only way I can think of to cover all of them is to change the
> postgresql service for setting shmmax at runtime when it starts or create a
> service to do the same thing just before starting postgresql avoiding to
> have something to do with sysctl files at all.

But then you interfere with this admin's manual changes.

I think best will be for postgresql to simply be able to work with a low shmmax :-)

Seriously, not sure which is best. I think we'll go with your (1) above and ignore zzz_shmmax.
Comment 8 Sandro Bonazzola 2013-11-11 09:27:20 EST
Ok, going with "don't remove shmmax configuration file on cleanup, just add a final note to the user to remove manually if needed and reboot." solution.
Comment 9 Bob Doolittle 2013-11-11 11:15:49 EST
(In reply to Yedidyah Bar David from comment #7)
> (In reply to Sandro Bonazzola from comment #6)
> > (In reply to Alon Bar-Lev from comment #5)
> > > (In reply to Yedidyah Bar David from comment #4)
> > > > > 
> > > > > We put our file only if there is no other configuration of shmmax.
> > > > 
> > > > We do? Not sure. Actually, we might get bitten if user has
> > > > /etc/sysctl.d/zzz_shmmax with a low value - it will override our file.
> > > > Perhaps we should also 'sysctl --system' and then check again, and
> > > > alert/exit if value is still too low.
> > > 
> > > We can, however, this way we might override runtime tunning... there are
> > > components that tune system at runtime without persisting anything...
> > > reloading all may create undesired effect. Not sure what is better.
> > 
> > I think that if an admin is configuring sysctl with a zzz_shmmax, the admin
> > is able fo solve shmmax issues himself.
> 
> Agreed.

Respectfully disagree. oVirt itself is a counterexample: changes are not necessarily made manually by an admin. Sometimes installing a layered feature/app can make such a change automatically. In such a case an admin may be totally unaware that a change has been applied, and therefore unable to easily make a good decision about how to recover.

That said, I think we're "between a rock and a hard place" because trying to automatically recover from this would be complex. So simply removing our configuration and requiring a reboot is our most viable option, much as I dislike forced reboots.

> 
> > However, the only way I can think of to cover all of them is to change the
> > postgresql service for setting shmmax at runtime when it starts or create a
> > service to do the same thing just before starting postgresql avoiding to
> > have something to do with sysctl files at all.
> 
> But then you interfere with this admin's manual changes.
> 
> I think best will be for postgresql to simply be able to work with a low
> shmmax :-)
> 
> Seriously, not sure which is best. I think we'll go with your (1) above and
> ignore zzz_shmmax.
Comment 10 Sandro Bonazzola 2013-11-12 07:46:35 EST
(In reply to Bob Doolittle from comment #9)
> Respectfully disagree. oVirt itself is a counterexample: changes are not
> necessarily made manually by an admin. Sometimes installing a layered
> feature/app can make such a change automatically. In such a case an admin
> may be totally unaware that a change has been applied, and therefore unable
> to easily make a good decision about how to recover.

Well, the application should be documented about kernel tuning changes.
However, about this I can see only one solution without changing too much the runtime environment: set our own conf and let the user discover that something else is playing with kernel tuning after reboot. Otherwise we must reload the configuration, affecting those services changing tuning at runtime.

> That said, I think we're "between a rock and a hard place" because trying to
> automatically recover from this would be complex. So simply removing our
> configuration and requiring a reboot is our most viable option, much as I
> dislike forced reboots.

I really prefer to leave the configuration there on cleanup and ask the user to take care of removing it ensuring it's not needed anymore, avoiding to break postgresql at reboot.
Comment 11 Sandro Bonazzola 2013-12-13 11:02:03 EST
rc build done, moving to ON_QA, packages will be available on updates-testing repository.
Comment 12 Sandro Bonazzola 2013-12-19 09:24:35 EST
oVirt 3.3.2 has been released resolving the problem described in this bug report.

Note You need to log in before you can comment on or make changes to this bug.