Bug 891831

Summary: Failed to update namespace when the domain has jenkins/jboss/zend applications
Product: OKD Reporter: Jianwei Hou <jhou>
Component: ContainersAssignee: Paul Morie <pmorie>
Status: CLOSED CURRENTRELEASE QA Contact: libra bugs <libra-bugs>
Severity: high Docs Contact:
Priority: high    
Version: 2.xCC: chunchen, dmcphers, pmorie, xtian, yujzhang
Target Milestone: ---Keywords: Regression, Reopened
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2013-02-13 23:05:18 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
broker log
none
mcollective log none

Description Jianwei Hou 2013-01-04 06:10:29 UTC 
Description of problem:
When updating namespace with applications, the following error is displayed "Error updating namespace: ["ews1s"]Undo namespace update failed: ["ews1s"]. If the problem persists please contact support."

The namespace can be successfully updated if it has no application

Version-Release number of selected component (if applicable):
On devenv_2632

How reproducible:
Always

Steps to Reproduce:
1. Create domain and create an application
rhc domain create 2632t3
rhc app create jks jenkins-1.4

2. Update namespace
rhc domain udpate 2632t3 2632t3s

  
Actual results:

hjw@my devenv$ domain-update 2632t3 2632t3s
Changing namespace '2632t3' to '2632t3s'...
Error updating namespace: ["jks"] If the problem persists please contact support.

Expected results:


Additional info:
Comment 1 Paul Morie 2013-01-04 18:01:27 UTC 
Recreating in devenv.
Comment 2 Paul Morie 2013-01-04 19:00:00 UTC 
Unable to reproduce on devenv.  Re-open if you can reproduce.
Comment 3 Jianwei Hou 2013-01-05 02:12:24 UTC 
I have reproduced it on devenv_2634.

This problem can be reproduced if we have jenkins-1.4/jbossas-7/jbosseap-6.0/jbossews-1.0/zend-5.6 applications in domain

I seem to find out why this fails, when updating namespace, mcollective log reports that it can't find /etc/httpd/conf.d/openshift/$UUID_$DOMAIN_$APP/*.json

On client: 
hjw@my ~$ rhc domain show -px
Applications in 2634t1 domain
=============================

  as1 @ http://as1-2634t1.dev.rhcloud.com/ (uuid: 9cb9afd853944dc8b06300d713d8f3f2)
  =================================================================================
    Created:   Jan 04  8:59 PM
    Gear Size: small
    Git URL:   ssh://9cb9afd853944dc8b06300d713d8f3f2.rhcloud.com/~/git/as1.git/
    SSH:       9cb9afd853944dc8b06300d713d8f3f2.rhcloud.com

    jbossas-7 (JBoss Application Server 7.1)
    ========================================

You have 1 applications in your domain.

hjw@my ~$ rhc domain update 2634t1 2634t2 -px
Changing namespace '2634t1' to '2634t2'...
Error updating namespace: ["as1"] If the problem persists please contact support.



Errors in broker log:
2013-01-04 21:06:40.482 [ERROR] [REQ_ID=48c19f99eb394ec592aaa74899cbdc71] ACTION=UPDATE_DOMAIN Error updating namespace: ["as1"] If the problem persists please contact support. (pid:2252)
2013-01-04 21:06:40.482 [ERROR] [REQ_ID=48c19f99eb394ec592aaa74899cbdc71] ACTION=UPDATE_DOMAIN Error updating namespace: ["as1"] If the problem persists please contact support. (pid:2252)
2013-01-04 21:06:40.484 [INFO ] Completed 500 Internal Server Error in 7180ms (Views: 0.6ms) (pid:2252)


Errors in mcollective log:
I, [2013-01-04T21:06:38.317801 #1054]  INFO -- : openshift.rb:448:in `complete_process_gracefully' ERROR: (2)
------
sed: can't read /etc/httpd/conf.d/openshift/9cb9afd853944dc8b06300d713d8f3f2_2634t1_as1/*.json: No such file or directory

------)
I, [2013-01-04T21:06:38.318062 #1054]  INFO -- : openshift.rb:507:in `cartridge_do_action' cartridge_do_action failed (2)
------
sed: can't read /etc/httpd/conf.d/openshift/9cb9afd853944dc8b06300d713d8f3f2_2634t1_as1/*.json: No such file or directory

------)


I have attached both logs
Comment 4 Jianwei Hou 2013-01-05 02:12:57 UTC 
Created attachment 672649 [details]
broker log
Comment 5 Jianwei Hou 2013-01-05 02:13:24 UTC 
Created attachment 672650 [details]
mcollective log
Comment 6 Xiaoli Tian 2013-01-05 02:52:32 UTC 
*** Bug 891849 has been marked as a duplicate of this bug. ***
Comment 7 Paul Morie 2013-01-07 15:37:51 UTC 
Recreating in devenv.
Comment 8 Paul Morie 2013-01-07 20:16:43 UTC 
Pull request made to origin-server with workaround.  Added conditional to test for json files during update-namespace hook.  Eventually all gears will have json files and conditional can be removed.
Comment 9 Paul Morie 2013-01-08 18:07:23 UTC 
Fix will be present in next devenv AMI.
Comment 10 Jianwei Hou 2013-01-09 04:33:43 UTC 
Verified on devenv_2650

Steps:
1. Create domain 
rhc domain create 2650t

2. Create jenkins-1.4/jbossas-7/jbosseap-6.0/jbossews-1.0/zend-5.6 applications in domain

3. Update domain
rhc domain update 2650t 2650t1

4. Access apps' webpage after domain updating


Result:
Domain is updated successfully.
The apps remain available after update domain.