Description of problem: If you run engine-setup; engine-cleanup; engine-setup (which might easily occur due to an error in the initial configuration), the final engine-setup may not properly store a persistent configuration for shmmax in sysctl, so that a subsequent reboot will cause postgresql to fail to come up so oVirt is non-operational. This is because when engine-setup decides whether or not to store a kernel.shmmax value in sysctl.d/ovirt-engine.conf it checks the current runtime value of shmmax. So you can wind up with the following scenario: 1. setup detects default/low runtime value, and stores kernel.shmmax in ovirt-engine.conf 2. Cleanup removes the file (but does not adjust the current runtime value) 3. 2nd setup detects adequate runtime value, and does not write a new ovirt-engine.conf 4. Upon the next reboot, the system comes up with the default/low value Postgresql will then be non-operational. Version-Release number of selected component (if applicable): 3.3 How reproducible: Completely Steps to Reproduce: 1. engine-setup 2. engine-cleanup 3. engine-setup 4. reboot Actual results: oVirt fails to come up cleanly, due to postgresql failure to start due to error allocating shm segment. You may find a general postgresql error in syslog. You may find more details via 'systemctl status postgresql'. Expected results: oVirt operational Additional info:
Note that determining an appropriate value to set in engine-cleanup is problematic. The simplest approach would be to set shmmax to the system default, but: 1. system default may change across future kernel releases 2. system default may have been altered prior to engine-cleanup, to meet requirements unrelated to oVirt Another approach might be to store the value prior to engine-setup, and restore that value during engine-cleanup. However this is still vulnerable to administrative changes which occurred after engine-setup and before engine-cleanup. I can't think of a general solution to this, other than requiring a reboot after engine-cleanup (i.e. don't even try to set a runtime value, simply remove ovirt-engine.conf and instruct the administrator to reboot after cleanup completion). That's not pretty, unfortunately. We'd sort of like the kernel to record a stack of runtime shmmax update operations, so that we could replay it without the oVirt change :)
Proposed solutions: 1) don't remove shmmax configuration file on cleanup, just add a final note to the user to remove manually if needed and reboot. this will leave existing runtime configuration and existing saved configuration as is, ensuring postgres will work also on reboot 2) save existing shmmax in /etc/sysctl.d/00-pre-ovirt-engine.conf and new values in /etc/sysctl.d/10-post-ovirt-engine.conf. On cleanup: - remove 10-post-ovirt-engine.conf - run sysctl --system - remove 00-pre-ovirt-engine.conf this will lead to having postgres going down and maybe not working at reboot but will restore original values at runtime. Note that just running sysctl --system doesn't reset the values to kernel defaults. I personally prefer 1. What do you think?
(In reply to Sandro Bonazzola from comment #2) > I personally prefer 1. What do you think? I agree... better to leave leftover then trying to revert to kernel defaults our-selves. We put our file only if there is no other configuration of shmmax.
(In reply to Alon Bar-Lev from comment #3) > (In reply to Sandro Bonazzola from comment #2) > > I personally prefer 1. What do you think? > > I agree... better to leave leftover then trying to revert to kernel defaults > our-selves. I agree too. > > We put our file only if there is no other configuration of shmmax. We do? Not sure. Actually, we might get bitten if user has /etc/sysctl.d/zzz_shmmax with a low value - it will override our file. Perhaps we should also 'sysctl --system' and then check again, and alert/exit if value is still too low.
(In reply to Yedidyah Bar David from comment #4) > > > > We put our file only if there is no other configuration of shmmax. > > We do? Not sure. Actually, we might get bitten if user has > /etc/sysctl.d/zzz_shmmax with a low value - it will override our file. > Perhaps we should also 'sysctl --system' and then check again, and > alert/exit if value is still too low. We can, however, this way we might override runtime tunning... there are components that tune system at runtime without persisting anything... reloading all may create undesired effect. Not sure what is better.
(In reply to Alon Bar-Lev from comment #5) > (In reply to Yedidyah Bar David from comment #4) > > > > > > We put our file only if there is no other configuration of shmmax. > > > > We do? Not sure. Actually, we might get bitten if user has > > /etc/sysctl.d/zzz_shmmax with a low value - it will override our file. > > Perhaps we should also 'sysctl --system' and then check again, and > > alert/exit if value is still too low. > > We can, however, this way we might override runtime tunning... there are > components that tune system at runtime without persisting anything... > reloading all may create undesired effect. Not sure what is better. I think that if an admin is configuring sysctl with a zzz_shmmax, the admin is able fo solve shmmax issues himself. However, the only way I can think of to cover all of them is to change the postgresql service for setting shmmax at runtime when it starts or create a service to do the same thing just before starting postgresql avoiding to have something to do with sysctl files at all.
(In reply to Sandro Bonazzola from comment #6) > (In reply to Alon Bar-Lev from comment #5) > > (In reply to Yedidyah Bar David from comment #4) > > > > > > > > We put our file only if there is no other configuration of shmmax. > > > > > > We do? Not sure. Actually, we might get bitten if user has > > > /etc/sysctl.d/zzz_shmmax with a low value - it will override our file. > > > Perhaps we should also 'sysctl --system' and then check again, and > > > alert/exit if value is still too low. > > > > We can, however, this way we might override runtime tunning... there are > > components that tune system at runtime without persisting anything... > > reloading all may create undesired effect. Not sure what is better. > > I think that if an admin is configuring sysctl with a zzz_shmmax, the admin > is able fo solve shmmax issues himself. Agreed. > However, the only way I can think of to cover all of them is to change the > postgresql service for setting shmmax at runtime when it starts or create a > service to do the same thing just before starting postgresql avoiding to > have something to do with sysctl files at all. But then you interfere with this admin's manual changes. I think best will be for postgresql to simply be able to work with a low shmmax :-) Seriously, not sure which is best. I think we'll go with your (1) above and ignore zzz_shmmax.
Ok, going with "don't remove shmmax configuration file on cleanup, just add a final note to the user to remove manually if needed and reboot." solution.
(In reply to Yedidyah Bar David from comment #7) > (In reply to Sandro Bonazzola from comment #6) > > (In reply to Alon Bar-Lev from comment #5) > > > (In reply to Yedidyah Bar David from comment #4) > > > > > > > > > > We put our file only if there is no other configuration of shmmax. > > > > > > > > We do? Not sure. Actually, we might get bitten if user has > > > > /etc/sysctl.d/zzz_shmmax with a low value - it will override our file. > > > > Perhaps we should also 'sysctl --system' and then check again, and > > > > alert/exit if value is still too low. > > > > > > We can, however, this way we might override runtime tunning... there are > > > components that tune system at runtime without persisting anything... > > > reloading all may create undesired effect. Not sure what is better. > > > > I think that if an admin is configuring sysctl with a zzz_shmmax, the admin > > is able fo solve shmmax issues himself. > > Agreed. Respectfully disagree. oVirt itself is a counterexample: changes are not necessarily made manually by an admin. Sometimes installing a layered feature/app can make such a change automatically. In such a case an admin may be totally unaware that a change has been applied, and therefore unable to easily make a good decision about how to recover. That said, I think we're "between a rock and a hard place" because trying to automatically recover from this would be complex. So simply removing our configuration and requiring a reboot is our most viable option, much as I dislike forced reboots. > > > However, the only way I can think of to cover all of them is to change the > > postgresql service for setting shmmax at runtime when it starts or create a > > service to do the same thing just before starting postgresql avoiding to > > have something to do with sysctl files at all. > > But then you interfere with this admin's manual changes. > > I think best will be for postgresql to simply be able to work with a low > shmmax :-) > > Seriously, not sure which is best. I think we'll go with your (1) above and > ignore zzz_shmmax.
(In reply to Bob Doolittle from comment #9) > Respectfully disagree. oVirt itself is a counterexample: changes are not > necessarily made manually by an admin. Sometimes installing a layered > feature/app can make such a change automatically. In such a case an admin > may be totally unaware that a change has been applied, and therefore unable > to easily make a good decision about how to recover. Well, the application should be documented about kernel tuning changes. However, about this I can see only one solution without changing too much the runtime environment: set our own conf and let the user discover that something else is playing with kernel tuning after reboot. Otherwise we must reload the configuration, affecting those services changing tuning at runtime. > That said, I think we're "between a rock and a hard place" because trying to > automatically recover from this would be complex. So simply removing our > configuration and requiring a reboot is our most viable option, much as I > dislike forced reboots. I really prefer to leave the configuration there on cleanup and ask the user to take care of removing it ensuring it's not needed anymore, avoiding to break postgresql at reboot.
rc build done, moving to ON_QA, packages will be available on updates-testing repository.
oVirt 3.3.2 has been released resolving the problem described in this bug report.