Description of problem: See forum post: https://www.redhat.com/openshift/community/forums/openshift/postgresql-gets-stuck-in-the-database-system-is-shutting-down User: sontek Apptype: python DB: postgresql Problem: After a git push, app cannot connect to the database. Getting "OperationalError: (OperationalError) FATAL: the database system is shutting down" Reproducible: 2 times so far, many times, git push doesn't exhibit this behavior. Workaround: rhc-app force-stop -a {appName} resolves the problem. Env: action scripts not being used. The user is not manually stopping the database. NOTE: Since this is difficult to reproduce (I tried to git push my app many times, as well as tried rhc app restart -a {appName} ), I reproduced this by manually ssh'ing into the gear and running: $OPENSHIFT_DB_CTL_SCRIPT stop This puts the db in that 'stopping' state. I didn't have the patience to wait after 5+ minutes to see if it actually stops. Took the route the user took (force_stop).
Can't reproduce this bug But put in a resiliency fix to do an immediate shutdown if a normal shutdown doesn't succeed. Fixed with git commit: e35247690ff2ec9b50f60144d9894a23441cdec3
We have not find the better way to reproduce, while checking the code, found a typo in the above commit id : should be pg_ctl -m immediate. + if `pgrep -x postgres -u $(id -u) > /dev/null 2>&1`; then + pg_ctl stop -D "$CART_INSTANCE_DIR/data" -m immedate -w >> $pglogfile 2>&1 ^^^^^^^^ + fi
Good catch - thanks. Fixed typo with git commit 9670ff3421c4ded7841b19869a18cb927bdebb2b
Verified this bug with devenv_1752, and PASS. Typo is fixed. Due to this bug is difficult to reproduce, I do git push, and stop db manually via ssh into app for several times. It always works fine.