Hide Forgot
Description of problem: Pod cannot startup if create with default installed mysql-ephemeral template or mysql-persistent template on ded-stage-aws, but template from orgin repo works well: https://raw.githubusercontent.com/openshift/origin/master/examples/db-templates/mysql-persistent-template.json https://raw.githubusercontent.com/openshift/origin/master/examples/db-templates/mysql-ephemeral-template.json Version-Release number of selected component (if applicable): atomic-openshift-3.2.0.8-1.git.0.f4edaed.el7.x86_64 How reproducible: always Steps to Reproduce: 1. Log into web console 2. Create a project 3. Add mysql-ephemeral template to project 4. Check pod status Actual results: $ oc get pods NAME READY STATUS RESTARTS AGE mysql-1-ai8py 0/1 Running 0 <invalid> mysql-1-deploy 1/1 Running 0 <invalid> Expected results: Pod should be running Additional info: Events: FirstSeen LastSeen Count From SubobjectPath Type Reason Message --------- -------- ----- ---- ------------- -------- ------ ------- 8m 8m 1 {default-scheduler } Normal Scheduled Successfully assigned mysql-1-ai8py to ip-172-31-2-203.ec2.internal 8m 8m 1 {kubelet ip-172-31-2-203.ec2.internal} spec.containers{mysql} Normal Pulled Container image "registry.access.redhat.com/rhscl/mysql-56-rhel7:latest" already present on machine 8m 8m 1 {kubelet ip-172-31-2-203.ec2.internal} spec.containers{mysql} Normal Created Created container with docker id 28099776d8b2 8m 8m 1 {kubelet ip-172-31-2-203.ec2.internal} spec.containers{mysql} Normal Started Started container with docker id 28099776d8b2 8m 8m 1 {kubelet ip-172-31-2-203.ec2.internal} spec.containers{mysql} Warning Unhealthy Readiness probe failed: sh: cannot set terminal process group (-1): Inappropriate ioctl for device sh: no job control in this shell ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) 8m <invalid> 60 {kubelet ip-172-31-2-203.ec2.internal} spec.containers{mysql} Warning Unhealthy Readiness probe failed: sh: cannot set terminal process group (-1): Inappropriate ioctl for device sh: no job control in this shell ERROR 1045 (28000): Access denied for user 'userRGQ'@'127.0.0.1' (using password: YES)
There's a quoting issue in the readiness checks for the mysql-* templates which was fixed in https://github.com/openshift/openshift-ansible/commit/de133ff75c912c588dc05ac8b9f9b6a6d29a9f99. Ops has deployed the fix to ded-stage-aws and the template now seems to work well.
Pod keeps in ContainerCreating in ded-stage-aws for cannot find volume, this seems an env issue: Events: FirstSeen LastSeen Count From SubobjectPath Type Reason Message --------- -------- ----- ---- ------------- -------- ------ ------- 28s 28s 1 {default-scheduler } Normal Scheduled Successfully assigned mysql-1-vi57b to ip-172-31-2-204.ec2.internal <invalid> <invalid> 2 {kubelet ip-172-31-2-204.ec2.internal} Warning FailedMount Unable to mount volumes for pod "mysql-1-vi57b_wzheng1(5bf9a1b2-fbde-11e5-89b5-0eb4d24322f9)": Could not attach EBS Disk "aws://us-east-1c/vol-a598950c": Error attaching EBS volume: InvalidVolume.NotFound: The volume 'vol-a598950c' does not exist. status code: 400, request id: <invalid> <invalid> 2 {kubelet ip-172-31-2-204.ec2.internal} Warning FailedSync Error syncing pod, skipping: Could not attach EBS Disk "aws://us-east-1c/vol-a598950c": Error attaching EBS volume: InvalidVolume.NotFound: The volume 'vol-a598950c' does not exist. status code: 400, request id:
(In reply to Wenjing Zheng from comment #2) > Pod keeps in ContainerCreating in ded-stage-aws for cannot find volume, this > seems an env issue: > Events: > FirstSeen LastSeen Count From SubobjectPath Type Reason Message > --------- -------- ----- ---- ------------- -------- ------ ------- > 28s 28s 1 {default-scheduler } Normal Scheduled Successfully > assigned mysql-1-vi57b to ip-172-31-2-204.ec2.internal > <invalid> <invalid> 2 {kubelet ip-172-31-2-204.ec2.internal} Warning > FailedMount Unable to mount volumes for pod > "mysql-1-vi57b_wzheng1(5bf9a1b2-fbde-11e5-89b5-0eb4d24322f9)": Could not > attach EBS Disk "aws://us-east-1c/vol-a598950c": Error attaching EBS volume: > InvalidVolume.NotFound: The volume 'vol-a598950c' does not exist. > status code: 400, request id: > <invalid> <invalid> 2 {kubelet ip-172-31-2-204.ec2.internal} Warning > FailedSync Error syncing pod, skipping: Could not attach EBS Disk > "aws://us-east-1c/vol-a598950c": Error attaching EBS volume: > InvalidVolume.NotFound: The volume 'vol-a598950c' does not exist. > status code: 400, request id: This is a different error than the reported bug which was a general readiness check failure affecting both the mysql-ephemeral and myst-persistent templates. I split off the new error into https://bugzilla.redhat.com/show_bug.cgi?id=1324492. Please verify the fix for the reported bug, and let's track the new error (which is specific to mysql-persistent) in the new report. Thanks!
Verified with latest ded-stage-aws, no such issue now.