Bug 982436

Summary: Unable to use any functionality for existing applications i.e. 'rhc apps' or 'rhc create-app' etc.
Product: OpenShift Online Reporter: Edward Archibald <edward.archibald>
Component: PodAssignee: Rajat Chopra <rchopra>
Status: CLOSED CURRENTRELEASE QA Contact: libra bugs <libra-bugs>
Severity: urgent Docs Contact:
Priority: unspecified    
Version: 1.xCC: bhatiam, janez.troha, lmeyer, rchopra, riccardo.vianello, xtian
Target Milestone: ---Keywords: UpcomingRelease
Target Release: ---   
Hardware: x86_64   
OS: Mac OS   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2013-07-22 15:22:41 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Edward Archibald 2013-07-09 03:43:37 UTC
Description of problem:
Sometime around 11 AM PDT, Mon. Jul 8, 2013, I was unable to interact with OpenShift, wither via command line or via the UI, for operations like:

rhc apps
rhc port-forward 
rhc app delete
rhc create-app
etc.

For any of these command, I consistently get the same message:

The server did not respond correctly. This may be an issue with the server configuration or with your connection to the server
(such as a Web proxy or firewall). Please verify that you can access the OpenShift server
https://openshift.redhat.com/broker/rest/domains/continuent/applications/tungstencluster

When I check the liveness of that URL, I get a username/password challenge after which (I provide a valid username/password) I get this message:

We're sorry, but something went wrong.

We've been notified about this issue and we'll take a look at it shortly.

Clicking on the 'My Applications' or 'My Apps' buttons on the UI yield the message:

We appear to be having technical difficulties
You can try refreshing the page, if the problem is temporary.

This has been going on for about 10 hours at this point.  
 

Version-Release number of selected component (if applicable):
I am running against the current production online broker etc.  I am running rhc 1.9.6 command line tools.

My account ID edward.archibald

Steps to Reproduce:
1. Establish a session with OpenShift online via rhc.  I logged out and then back in just in case something was hosed in that path.
2. rhc apps

Actual results:
The server did not respond correctly. This may be an issue with the server configuration or with your connection to the server
(such as a Web proxy or firewall). Please verify that you can access the OpenShift server
https://openshift.redhat.com/broker/rest/domains/continuent/applications/tungstencluster


Expected results:
A listing of my current apps


Additional info:
I am able to log into the gear on which my app/cartridge is running if I log in directly using the username/host i.e. ssh 51da0957500446f29700005b.com but if I try to use anything that looks up apps, like 'rhc ssh tungstencluster', it fails.

Comment 1 Riccardo Vianello 2013-07-09 10:55:06 UTC
I think I'm experiencing the same issue. The reference reported by the web UI when accessing 'my apps' is the following: #335c5e33ffa8761f79d90922bd070829

Comment 2 dz0ny 2013-07-09 18:37:56 UTC
I also have the same problem. Reference #119a250574f10436db8f3acc7cfaf166

Comment 3 Edward Archibald 2013-07-09 18:40:49 UTC
I am using a downloadable cartridge in this app - it's a cartridge that I am developing.  The cartridge git is at:  narmitag/openshift-origin-cartridge-tungsten 

The cartridge work has been proceeding well up until I hit this issue - for about a week.

Comment 4 Edward Archibald 2013-07-09 18:46:40 UTC
Here's the exact command I used to create the app with the downloadable cartridge:

rhc create-app tungstencluster  php-5.3  http://cartreflect-claytondev.rhcloud.com/reflect?github=narmitag/openshift-origin-cartridge-tungsten --gear medium

Comment 5 Abhishek Gupta 2013-07-09 19:49:13 UTC
We have identified the problem and are working to fix the issue as well as the broken applications. The fix should not take more than a few hours. Will update this bug when the fix is complete.

Comment 6 Rajat Chopra 2013-07-09 22:55:13 UTC
There was a bug with a maintenance script that affected all applications with downloaded cartridges. The erring maintenance was run around 4pm July 8. 

All affected applications created before 2013-07-08 00:00:00EDT have been fixed.

If there was an application created (with downloadable cartridges) between 2013-07-08 00:00:00EDT and 2013-07-08 16:00:00 EDT, then it may still be broken. We are working on fixing them too.

Apologies for any hassle that this may have caused.

Comment 7 Rajat Chopra 2013-07-10 01:30:39 UTC
*** Bug 982658 has been marked as a duplicate of this bug. ***

Comment 8 Abhishek Gupta 2013-07-10 22:43:51 UTC
Assigning it to Rajat as he is working on identifying and fixing any remaining impacted applications.

Comment 9 Riccardo Vianello 2013-07-11 11:37:44 UTC
Thank you for working on this issue. The problematic application for my account is most likely to be 'mycart'. It's just a test application that doesn't currently provide any functionality, so please feel free to simply erase it if this may help a quicker resolution.

Comment 10 Rajat Chopra 2013-07-11 19:56:00 UTC
All known applications have been fixed. Please report if anything is still not working.

Comment 11 Riccardo Vianello 2013-07-12 06:00:56 UTC
My account (ID is riccardo dot vianello at gmail dot com) is still unusable. Latest reference displayed by the web UI is #50d9909ed3fc229803fa2b24327e3103.

Comment 12 Xiaoli Tian 2013-07-15 08:30:39 UTC
Move it to assigned to see if more fixing is needed according to comment 11 or if more migration is needed.

Comment 13 Rajat Chopra 2013-07-16 22:52:03 UTC
The erring app was deleted with user's permission (comment#9). This application apparently had more than one downloaded cartridges, and the fix reported in comment#10 missed that complication.

Hope everything is working now.

Comment 14 Riccardo Vianello 2013-07-17 06:02:19 UTC
Confirmed, my account is functional again. Thank you very much for your help.

Comment 15 Xiaoli Tian 2013-07-17 10:26:04 UTC
Move it to verified since the account is fixed according to comment 14