Bug 1010846 - Can not reload postgresql-8.4 after stopping DB cartridge with non-scablable app
Summary: Can not reload postgresql-8.4 after stopping DB cartridge with non-scablable app
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Containers
Version: 1.2.0
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: ---
: ---
Assignee: Miciah Dashiel Butler Masters
QA Contact: libra bugs
URL:
Whiteboard:
Depends On: 981528 981780
Blocks: 980739
TreeView+ depends on / blocked
 
Reported: 2013-09-23 07:52 UTC by Ma xiaoqiang
Modified: 2017-03-08 17:35 UTC (History)
9 users (show)

Fixed In Version: openshift-origin-cartridge-postgresql-0.2.8-1
Doc Type: Bug Fix
Doc Text:
Due to a bug in the PostgreSQL cartridge, non-scaled PostgreSQL databases would not reload after the cartridge was stopped and restarted. This has been fixed and the PostgreSQL databases now reload correctly after a cartridge restart.
Clone Of: 981528
Environment:
Last Closed: 2013-10-16 15:39:24 UTC
Target Upstream Version:
Embargoed:
xiama: needinfo+


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2013:1435 0 normal SHIPPED_LIVE OpenShift Enterprise 1.2.4 bug fix and enhancement update 2013-10-16 19:38:22 UTC

Description Ma xiaoqiang 2013-09-23 07:52:57 UTC
This problem is existing in 1.2.3-RHN-Stage puddle build 

openshift-origin-cartridge-postgresql-0.2.7-1.1.el6op.noarch

+++ This bug was initially created as a clone of Bug #981528 +++

Description of problem:
After stopping postgresql-8.4/postgresql-9.2 cartridge,tried to reload them,but it failed.It returned some error messages,like
Unable to complete the requested operation due to: Failed to correctly execute all parallel operations

Version-Release number of selected component (if applicable):
devenv_3449

How reproducible:
Always

Steps to Reproduce:
1.Create an application 
2.Add postgresql-8.4/postgresql-9.2
3.Stop postgresql-8.4/postgresql-9.2
4.Reload postgresql-8.4/postgresql-9.2

Actual results:
openshift@openshift-ubuntu:~$ rhc reload-cartridge postgresql-9.2 -a eapt
Unable to complete the requested operation due to: Failed to correctly execute all parallel operations - --DEBUG--
Failed to execute: 'control reload' for /var/lib/openshift/41d9221ce51e11e28abc22000aa43651/postgresql

pg_ctl: PID file "/var/lib/openshift/41d9221ce51e11e28abc22000aa43651/postgresql/data/postmaster.pid" does not exist
Is server running?

--RESULT--

--MESSAGE--

--ERROR--

--APP INFO--

--CART COMMANDS--

--CART PROPERTIES--
{}
--DATA--

--EXIT CODE--
1
--USER ACTIONABLE--
false
.
Reference ID: cd54f321c881dcadfcccd1ad019eaaf6


Expected results:
openshift@openshift-ubuntu:~$ rhc reload-cartridge postgresql-9.2 -a eapt
RESULT:
postgresql-9.2 reloaded


Additional info:
If db cartridge is started,can reload the db cartridge

--- Additional comment from Yan Du on 2013-07-05 02:01:30 EDT ---

it is ok with scalable app

[root@Daphne test]# rhc cartridge stop -a j2s -c postgresql-9.2
RESULT:
postgresql-9.2 stopped

[root@Daphne test]# rhc cartridge reload -a j2s -c postgresql-9.2
RESULT:
postgresql-9.2 reloaded

--- Additional comment from Hiro Asari on 2013-07-05 15:29:43 EDT ---

The fact that the scalable app case returns the "success" message is actually a bug in CLI. (See https://bugzilla.redhat.com/show_bug.cgi?id=981780)

--- Additional comment from Hiro Asari on 2013-07-05 15:32:49 EDT ---

'pg_ctl reload' will fail when postgresql is not running, so we need to avoid doing that. This is addressed by https://github.com/openshift/origin-server/pull/2993.

However, the PR won't address this bug entirely, since due to a couple of issues indicated above.

--- Additional comment from openshift-github-bot on 2013-07-05 16:01:41 EDT ---

Commit pushed to master at https://github.com/openshift/origin-server

https://github.com/openshift/origin-server/commit/ee8816e280c1aeead2351943cce7b8d2e3884f06
Bug 981528

PostgreSQL can reload the config only when it's running.

--- Additional comment from Hiro Asari on 2013-07-09 17:36:27 EDT ---



--- Additional comment from openshift-github-bot on 2013-07-10 10:04:32 EDT ---

Commit pushed to master at https://github.com/openshift/origin-server

https://github.com/openshift/origin-server/commit/3273645be4a4455e733f629369a933a12a127c76
Bug 981528

When the cartridge is down, it will be sent 'force-reload', not
'reload'.
Respond to that, and communicate with the user that it is invalid.

Use client_result for now, so that the user can see the message

See https://bugzilla.redhat.com/show_bug.cgi?id=981780

--- Additional comment from Hiro Asari on 2013-07-10 13:25:59 EDT ---

bx bin/rhc app create rack ruby-1.9 postgresql-9.2
bx bin/rhc cartridge stop postgresql-9.2 -a rack
bx bin/rhc cartridge reload postgresql-9.2 -a rack


bx bin/rhc app stop postgresql-9.2 -a rack
bx bin/rhc cartridge reload postgresql-9.2 -a rack

both 'reload' should result in:

$ bx bin/rhc cartridge reload postgresql-9.2 -a rack
Reloading postgresql-9.2 ... done

'reload' is valid only when the database is running

--- Additional comment from Hiro Asari on 2013-07-10 13:27:13 EDT ---

https://github.com/openshift/origin-server/pull/3042 should do this for good.

--- Additional comment from openshift-github-bot on 2013-07-10 14:35:51 EDT ---

Commit pushed to master at https://github.com/openshift/origin-server

https://github.com/openshift/origin-server/commit/187bae81024adcb4bd53b7334bc126152b7626d3
Bug 981528

reload/force-reload depends on application status, so both can be sent
to the cartridge

--- Additional comment from Wei Sun on 2013-07-10 22:56:22 EDT ---

Verified on INT(devenv_3481)

Result:
1.for postgresql-8.4
[sunwei@dhcp-8-229 ~]$ rhc stop-cartridge postgresql-8.4 -a perlt
Stopping postgresql-8.4 ... done
[sunwei@dhcp-8-229 ~]$ rhc reload-cartridge postgresql-8.4 -a perlt
Reloading postgresql-8.4 ... done

'reload' is valid only when the database is running
[sunwei@dhcp-8-229 ~]$ rhc status-cartridge postgresql-8.4 -a perlt
RESULT:
Postgres is stopped

2.for postgresql-9.2
[sunwei@dhcp-8-229 ~]$ rhc stop-cartridge postgresql-9.2 -a perlt
Stopping postgresql-9.2 ... done
[sunwei@dhcp-8-229 ~]$ rhc status-cartridge postgresql-9.2 -a perlt
RESULT:
Postgres is stopped
[sunwei@dhcp-8-229 ~]$ rhc reload-cartridge postgresql-9.2 -a perlt
Reloading postgresql-9.2 ... done

'reload' is valid only when the database is running
[sunwei@dhcp-8-229 ~]$ rhc status-cartridge postgresql-9.2 -a perlt
RESULT:
Postgres is stopped

Comment 2 Miciah Dashiel Butler Masters 2013-10-03 16:27:10 UTC
Can you check again that the bug exists on OSE 1.2? I cannot reproduce the problem:

  [root@broker ~]# rhc app create testapp2 ruby-1.9 postgresql -s --no-git
  Using postgresql-8.4 (PostgreSQL Database 8.4) for 'postgresql'
  Your authorization token has expired. Please sign in now to continue.
  Password: ********
  
  
  Application Options
  -------------------
    Namespace:  ose
    Cartridges: ruby-1.9, postgresql-8.4
    Gear Size:  default
    Scaling:    yes
  
  Creating application 'testapp2' ... done
  
  Waiting for your DNS name to be available ... done
  
  testapp2 @ http://testapp2-ose.example.com/ (uuid: 524d90a36892dfc384000027)
  ----------------------------------------------------------------------------
    Created: 8:43 AM
    Gears:   2 (defaults to small)
    Git URL: ssh://524d90a36892dfc384000027.com/~/git/testapp2.git/
    SSH:     524d90a36892dfc384000027.com
  
    postgresql-8.4 (PostgreSQL Database 8.4)
    ----------------------------------------
      Gears:          1 small
      Connection URL: postgresql://$OPENSHIFT_POSTGRESQL_DB_HOST:$OPENSHIFT_POSTGRESQL_DB_PORT
      Database Name:  testapp2
      Password:       CkTNjUkQASpC
      Username:       adminghqzyt1
  
    ruby-1.9 (Ruby 1.9)
    -------------------
      Scaling: x1 (minimum: 1, maximum: available) on small gears
  
    haproxy-1.4 (OpenShift Web Balancer)
    ------------------------------------
      Gears: Located with ruby-1.9
  
  RESULT:
  Application testapp2 was created.
  PostgreSQL 8.4 database added.  Please make note of these credentials:
     Root User: adminghqzyt1
     Root Password: CkTNjUkQASpC
     Database Name: testapp2
  Connection URL: postgresql://$OPENSHIFT_POSTGRESQL_DB_HOST:$OPENSHIFT_POSTGRESQL_DB_PORT
  HAProxy instance is started
  
  [root@broker ~]# rhc cartridge stop postgresql -a testapp2
  Using postgresql-8.4 (PostgreSQL Database 8.4) for 'postgresql'
  
  RESULT:
  postgresql-8.4 stopped
  [root@broker ~]# rhc cartridge reload postgresql -a testapp2
  Using postgresql-8.4 (PostgreSQL Database 8.4) for 'postgresql'
  
  RESULT:
  postgresql-8.4 reloaded
  [root@broker ~]# rhc cartridge stop postgresql -a testapp2
  Using postgresql-8.4 (PostgreSQL Database 8.4) for 'postgresql'
  
  RESULT:
  postgresql-8.4 stopped
  [root@broker ~]# rhc cartridge reload postgresql -a testapp2
  Using postgresql-8.4 (PostgreSQL Database 8.4) for 'postgresql'
  
  RESULT:
  postgresql-8.4 reloaded
  [root@broker ~]# rpm -qa |grep cartridge-postgresq
  openshift-origin-cartridge-postgresql-0.2.7-1.1.el6op.noarch
  [root@broker ~]#

Additionally, the reported error message does not exist in Enterprise.  It was introduced in commit ee550b2e3700f9777b884c4814c337aecc4eb6db to origin-server, which has not been backported to enterprise-server, and the text 'Failed to correctly execute all parallel operations' does not appear anywhere in the Enterprise codebase (I did a git grep on the enterprise-1.2 and enterprise-1.2.z branches).

Can you bug the problem again on OSE 1.2 and provide the exact commands and output of those commands? If I could get access to a devenv where you have reproduced the bug, that would be even better.

For what it's worth, I do have a local branch with the backport of the fix from origin-server, ready to push and merge.

Comment 3 Ma xiaoqiang 2013-10-08 01:01:47 UTC
I have check it. We should create a non-scalable app to reproduce this problem.there is not this problem with a scalable app.

1.create an non-scalable app
#rhc app create xiaom3 ruby-1.9
2.add postgresql cartridge to it
#rhc cartridge add postgre -a xiaom3
3.stop the postgresql cartridge 
#rhc cartridge stop postgresql -a xiaom3
4.reload cartirdge 
#rhc cartridge reload postgresql -a xiaom3    
Using postgresql-8.4 (PostgreSQL Database 8.4) for 'postgresql'
Unable to complete the requested operation due to: Node execution failure (invalid exit code from node).  If the problem persists please contact Red Hat support..
Reference ID: ff4c4ec4606bb77e8881d350c700ff26.

Comment 5 Ma xiaoqiang 2013-10-09 08:59:53 UTC
check this problem on puddle:[1.2/2013-10-08.1] 
1.create an non-scalable app
#rhc app create xiaom3 ruby-1.9
2.add postgresql cartridge to it
#rhc cartridge add postgre -a xiaom3
3.stop the postgresql cartridge 
#rhc cartridge stop postgresql -a xiaom3
4.reload cartirdge 
#rhc cartridge reload postgresql -a xiaom3
Using postgresql-8.4 (PostgreSQL Database 8.4) for 'post'
Reloading postgresql-8.4 ... 
'reload' is valid only when the database is running

Comment 8 errata-xmlrpc 2013-10-16 15:39:24 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2013-1435.html


Note You need to log in before you can comment on or make changes to this bug.