Bug 984837

Summary: Postgresql cartridges were not been upgraded successfully on some nodes in STG
Product: OpenShift Online Reporter: Zhe Wang <zhewang>
Component: ContainersAssignee: Hiro Asari <hasari>
Status: CLOSED CURRENTRELEASE QA Contact: libra bugs <libra-bugs>
Severity: medium Docs Contact:
Priority: medium    
Version: 2.xCC: admiller, hasari, jkeck, pmorie, xtian, zhewang
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2013-07-22 15:24:52 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Zhe Wang 2013-07-16 07:24:47 UTC 
Description of problem:
Adding a Postgresql-9.2 cartridge to an app sometimes fails with the following error:

Running Command - rhc cartridge add postgresql-9.2 -a nodejsph8v0av -l jizhao+1 -p 'redhat' --timeout 360
Adding postgresql-9.2 to application 'nodejsph8v0av' ... Unexpected error: key not found: (postgresql, 9.2, _)
Command Return: 1

The problem could root in that some nodes in STG did not get upgraded successfully, there might be  even no postgresql-9.2 cartridge yet.

Created an app with postgresql-8.4 added, which was still an old versioned postgresql according to the environment variables below:
OPENSHIFT_POSTGRES_VERSION=8.4
OPENSHIFT_POSTGRESQL_IDENT=redhat:postgresql:8.4:0.2.0

However, this is not reproduced when I tried again, possibly some nodes are upgraded successfully,  but some nodes fails

Version-Release number of selected component (if applicable):
STG(devenv-stage_406)

How reproducible:
Sometimes

Steps to Reproduce:
1. create an app, for example, a ruby-1.9 app
rhc app create rb9 ruby-1.9

2. Add a Postgresql-9.2 cartridge to this app
rhc cartridge add postgresql-9.2 -a rb9

Actual results:
Adding a Postgresql-9.2 sometimes fails with the error:

Adding postgresql-9.2 to application 'nodejsph8v0av' ... Unexpected error: key not found: (postgresql, 9.2, _)

Expected results:
Adding a Postgresql-9.2 cartridge should be successful in the scenario above.
The version in POSTGRESQL_IDENT should be updated to 0.0.3  at least.

Additional info:
Comment 1 Paul Morie 2013-07-16 15:36:47 UTC 
It sounds like the new postgres cartridge RPM is not available on all the nodes.  Ops needs to verify that the new package is installed everywhere.
Comment 2 Adam Miller 2013-07-16 15:37:58 UTC 
It is available and installed everywhere, consistency checks passed as well.
Comment 3 Hiro Asari 2013-07-16 15:39:14 UTC 
Could you test this again? This should be OK now, given Adam's comment above.

Thank you.
Comment 4 Adam Miller 2013-07-16 17:16:20 UTC 
No, nothing changed in STG since yesterday's deployment (not yet anyways, we do have a deploy scheduled for later today).

Also note, I suspect this to be related:
    https://bugzilla.redhat.com/show_bug.cgi?id=985063
Comment 5 Adam Miller 2013-07-16 19:33:11 UTC 
This is expected to be resolved, it's not that some nodes had an issue but there was a strange rails caching issue on the brokers which was causing cache responses to deliver the incorrect list of carts.
Comment 6 Zhe Wang 2013-07-17 10:24:35 UTC 
Referring to the results of Postgresql-9.2 cases in today's auto test against STG(devenv-stage_410), no failures were detected due to this bug. Thus, move it to VERIFIED.

Thanks,
Zhe Wang