Bug 1270739

Summary:	Almost all pods keep in pending status on QE V3 Cluster
Product:	OpenShift Online	Reporter:	shliang <shliang>
Component:	Containers	Assignee:	Miciah Dashiel Butler Masters <mmasters>
Status:	CLOSED CURRENTRELEASE	QA Contact:	Chao Yang <chaoyang>
Severity:	high	Docs Contact:
Priority:	unspecified
Version:	3.x	CC:	aos-bugs, jokerman, mmccomas, xtian
Target Milestone:	---
Target Release:	3.x
Hardware:	Unspecified
OS:	Linux
Whiteboard:
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2016-05-23 15:10:01 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description shliang 2015-10-12 09:40:45 UTC

Description of problem:
Create application(cakephp,mysql,mongodb or postsql) in console,then the pod keeps pending for ever.

Version-Release number of selected component (if applicable):
QE v3's cluster

How reproducible:
on 2015.10.10, it was fine,but it starts to fail from 2015.10.12 morning.

Steps to Reproduce:
1.Login openshift in web console
2.create app by template such as mysql-ephemeral(or postsql and other app template)
3.then check  the pod‘s status

Actual results:
3.then the pod keep pending for ever

Expected results:

3. the pod should be running

Additional info:

Sometimes  it works well suddenly.for example, a pod kept pending morining ,but turn running afternoon.

Comment 1 shliang 2015-10-13 02:15:34 UTC

Additional info:

The events of the pod gives:

Time 	Name 	Kind 	Reason 	Message
10:10:59 AM 	mysql-1-c545x 	Pod 	scheduled 	Successfully assigned mysql-1-c545x to 172.31.53.216
10:10:59 AM 	mysql-1 	ReplicationController 	successfulCreate 	Created pod: mysql-1-c545x
10:10:54 AM 	mysql-1-deploy 	Pod 	scheduled 	Successfully assigned mysql-1-deploy to 172.31.53.214
10:10:54 AM 	mysql-1 	ReplicationController 	failedUpdate 	Error updating deployment mysql-project/mysql-1 status to Pending
10:10:17 AM 	mysql-1-deploy 	Pod 	killing 	Killing with docker id ab08bb744a8c
10:10:12 AM 	mysql-1-c545x 	Pod 	pulled 	Container image "openshift3/ose-pod:v3.0.2.0" already present on machine
10:10:09 AM 	mysql-1-deploy 	Pod 	created 	Created with docker id c62bfd572ff2
10:10:09 AM 	mysql-1-deploy 	Pod 	started 	Started with docker id c62bfd572ff2
10:10:08 AM 	mysql-1-deploy 	Pod 	created 	Created with docker id ab08bb744a8c
10:10:08 AM 	mysql-1-deploy 	Pod 	started 	Started with docker id ab08bb744a8c
10:10:08 AM 	mysql-1-deploy 	Pod 	pulled 	Container image "openshift3/ose-deployer:v3.0.2.0" already present on machine
10:10:07 AM 	mysql-1-deploy 	Pod 	pulled 	Container image "openshift3/ose-pod:v3.0.2.0" already present on machine

And the status of the pod:

mysql-1-c545x
Status: Pending
Node:172.31.53.216
IP on node:unknown
Restart policy: Always
Volumes:

    mysql-data
    default-token-5rq5d

Pod template:
Container: mysql
Image: openshift/mysql-55-centos7
Ports: 3306 (TCP)

Comment 3 Miciah Dashiel Butler Masters 2016-02-24 15:17:39 UTC

I could not reproduce the problem today.  I logged into the QE cluster's console, created a new project using the "mongodb-ephemeral" template, and checked the pod's status:

    % oc get pods
    NAME              READY     STATUS    RESTARTS   AGE
    mongodb-1-zovzr   1/1       Running   0          9m

I also tried with the "cakephp-mysql-example" and "dancer-mysql-example" templates:

    % oc get pods
    NAME                            READY     STATUS    RESTARTS   AGE
    cakephp-mysql-example-1-build   1/1       Running   0          6m
    mysql-1-qt833                   1/1       Running   0          7m

    % oc get pods
    NAME                           READY     STATUS    RESTARTS   AGE
    dancer-mysql-example-1-build   1/1       Running   0          2m
    database-1-0zyd1               1/1       Running   0          2m

Is anyone still seeing the problem? If so, with which template or templates? I did not see "mysql-ephemeral" or "postsql" as options.

Comment 4 Chao Yang 2016-02-29 07:32:10 UTC

This issue not existed in the puddle kubernetes v1.2.0-alpha.7-703-gbc4550d

Comment 5 Chao Yang 2016-02-29 10:27:02 UTC

This issue not existed in openshift dedicated env