Bug 1323096

Summary: [Dedicated]Template mysql-ephemeral/persistent template should be bumped into ded-stage-aws from origin
Product: OpenShift Online Reporter: Wenjing Zheng <wzheng>
Component: WebsiteAssignee: Dan Mace <dmace>
Status: CLOSED CURRENTRELEASE QA Contact: Yanping Zhang <yanpzhan>
Severity: medium Docs Contact:
Priority: medium    
Version: 3.xCC: abhgupta, aos-bugs, dmace, jokerman, mmccomas
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-05-23 15:10:18 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:

Description Wenjing Zheng 2016-04-01 09:30:18 UTC
Description of problem:
Pod cannot startup if create with default installed mysql-ephemeral template or mysql-persistent template on ded-stage-aws, but template from orgin repo works well:
https://raw.githubusercontent.com/openshift/origin/master/examples/db-templates/mysql-persistent-template.json
https://raw.githubusercontent.com/openshift/origin/master/examples/db-templates/mysql-ephemeral-template.json

Version-Release number of selected component (if applicable):
atomic-openshift-3.2.0.8-1.git.0.f4edaed.el7.x86_64

How reproducible:
always

Steps to Reproduce:
1. Log into web console
2. Create a project
3. Add mysql-ephemeral template to project
4. Check pod status

Actual results:
$ oc get pods
NAME             READY     STATUS    RESTARTS   AGE
mysql-1-ai8py    0/1       Running   0          <invalid>
mysql-1-deploy   1/1       Running   0          <invalid>


Expected results:
Pod should be running

Additional info:
Events:
  FirstSeen    LastSeen    Count    From                    SubobjectPath        Type        Reason        Message
  ---------    --------    -----    ----                    -------------        --------    ------        -------
  8m        8m        1    {default-scheduler }                        Normal        Scheduled    Successfully assigned mysql-1-ai8py to ip-172-31-2-203.ec2.internal
  8m        8m        1    {kubelet ip-172-31-2-203.ec2.internal}    spec.containers{mysql}    Normal        Pulled        Container image "registry.access.redhat.com/rhscl/mysql-56-rhel7:latest" already present on machine
  8m        8m        1    {kubelet ip-172-31-2-203.ec2.internal}    spec.containers{mysql}    Normal        Created        Created container with docker id 28099776d8b2
  8m        8m        1    {kubelet ip-172-31-2-203.ec2.internal}    spec.containers{mysql}    Normal        Started        Started container with docker id 28099776d8b2
  8m        8m        1    {kubelet ip-172-31-2-203.ec2.internal}    spec.containers{mysql}    Warning        Unhealthy    Readiness probe failed: sh: cannot set terminal process group (-1): Inappropriate ioctl for device
sh: no job control in this shell
ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111)

  8m    <invalid>    60    {kubelet ip-172-31-2-203.ec2.internal}    spec.containers{mysql}    Warning    Unhealthy    Readiness probe failed: sh: cannot set terminal process group (-1): Inappropriate ioctl for device
sh: no job control in this shell
ERROR 1045 (28000): Access denied for user 'userRGQ'@'127.0.0.1' (using password: YES)

Comment 1 Dan Mace 2016-04-06 00:14:27 UTC
There's a quoting issue in the readiness checks for the mysql-* templates which was fixed in https://github.com/openshift/openshift-ansible/commit/de133ff75c912c588dc05ac8b9f9b6a6d29a9f99. Ops has deployed the fix to ded-stage-aws and the template now seems to work well.

Comment 2 Wenjing Zheng 2016-04-06 10:05:25 UTC
Pod keeps in ContainerCreating in ded-stage-aws for cannot find volume, this seems an env issue:
Events:
  FirstSeen	LastSeen	Count	From					SubobjectPath	Type		Reason		Message
  ---------	--------	-----	----					-------------	--------	------		-------
  28s		28s		1	{default-scheduler }					Normal		Scheduled	Successfully assigned mysql-1-vi57b to ip-172-31-2-204.ec2.internal
  <invalid>	<invalid>	2	{kubelet ip-172-31-2-204.ec2.internal}			Warning		FailedMount	Unable to mount volumes for pod "mysql-1-vi57b_wzheng1(5bf9a1b2-fbde-11e5-89b5-0eb4d24322f9)": Could not attach EBS Disk "aws://us-east-1c/vol-a598950c": Error attaching EBS volume: InvalidVolume.NotFound: The volume 'vol-a598950c' does not exist.
		status code: 400, request id: 
  <invalid>	<invalid>	2	{kubelet ip-172-31-2-204.ec2.internal}		Warning	FailedSync	Error syncing pod, skipping: Could not attach EBS Disk "aws://us-east-1c/vol-a598950c": Error attaching EBS volume: InvalidVolume.NotFound: The volume 'vol-a598950c' does not exist.
		status code: 400, request id:

Comment 3 Dan Mace 2016-04-06 13:19:26 UTC
(In reply to Wenjing Zheng from comment #2)
> Pod keeps in ContainerCreating in ded-stage-aws for cannot find volume, this
> seems an env issue:
> Events:
>   FirstSeen	LastSeen	Count	From					SubobjectPath	Type		Reason		Message
>   ---------	--------	-----	----					-------------	--------	------		-------
>   28s		28s		1	{default-scheduler }					Normal		Scheduled	Successfully
> assigned mysql-1-vi57b to ip-172-31-2-204.ec2.internal
>   <invalid>	<invalid>	2	{kubelet ip-172-31-2-204.ec2.internal}			Warning	
> FailedMount	Unable to mount volumes for pod
> "mysql-1-vi57b_wzheng1(5bf9a1b2-fbde-11e5-89b5-0eb4d24322f9)": Could not
> attach EBS Disk "aws://us-east-1c/vol-a598950c": Error attaching EBS volume:
> InvalidVolume.NotFound: The volume 'vol-a598950c' does not exist.
> 		status code: 400, request id: 
>   <invalid>	<invalid>	2	{kubelet ip-172-31-2-204.ec2.internal}		Warning
> FailedSync	Error syncing pod, skipping: Could not attach EBS Disk
> "aws://us-east-1c/vol-a598950c": Error attaching EBS volume:
> InvalidVolume.NotFound: The volume 'vol-a598950c' does not exist.
> 		status code: 400, request id:


This is a different error than the reported bug which was a general readiness check failure affecting both the mysql-ephemeral and myst-persistent templates. I split off the new error into https://bugzilla.redhat.com/show_bug.cgi?id=1324492.

Please verify the fix for the reported bug, and let's track the new error (which is specific to mysql-persistent) in the new report. Thanks!

Comment 4 Wenjing Zheng 2016-04-07 06:30:46 UTC
Verified with latest ded-stage-aws, no such issue now.