Bug 1270441

Summary: The container in pod with user defined secret volume can't work
Product: OKD Reporter: zhou ying <yinzhou>
Component: DeploymentsAssignee: Dan Mace <dmace>
Status: CLOSED CURRENTRELEASE QA Contact: zhou ying <yinzhou>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 3.xCC: aos-bugs, bparees, pweil, yinzhou
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2015-11-23 21:15:53 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description zhou ying 2015-10-10 06:42:31 UTC
Description of problem:
Create a secret, add the secret volume to dc, after deployment, the pod can't be running.


Version-Release number of selected component (if applicable):
oc v1.0.6-328-gdf1f19e
kubernetes v1.1.0-alpha.1-653-g86b4e77

How reproducible:
always

Steps to Reproduce:
1. Create resource  with json file:
  `oc create -f https://raw.githubusercontent.com/openshift-qe/v3-testfiles/master/deployment/deployment1.json`
2. Create a secret:
   `oc secrets new my-secret ssh-privatekey=/root/.ssh/id_rsa ssh-publickey=/root/.ssh/id_rsa.pub`
3. Check the secret in project:
 [root@ip-172-18-1-90 amd64]# oc get secret
NAME                       TYPE                                  DATA      AGE
builder-dockercfg-paukz    kubernetes.io/dockercfg               1         3h
builder-token-o2e51        kubernetes.io/service-account-token   2         3h
builder-token-ywd8e        kubernetes.io/service-account-token   2         3h
default-dockercfg-eb0ve    kubernetes.io/dockercfg               1         3h
default-token-703lu        kubernetes.io/service-account-token   2         3h
default-token-kuk2f        kubernetes.io/service-account-token   2         3h
deployer-dockercfg-v06rf   kubernetes.io/dockercfg               1         3h
deployer-token-fi5z2       kubernetes.io/service-account-token   2         3h
deployer-token-ykq6u       kubernetes.io/service-account-token   2         3h
my-secret                  Opaque                                2         57m
registry-secret            Opaque                                2         3h
  
4. Add the new secret to dc
   `oc volume dc/hooks  --add --name=secret --type=secret --secret-name=my-secret --mount-path=/etc`
5. Check the new pod's status:

Actual results:
The pod does not work:
[root@ip-172-18-1-90 amd64]# oc get pod
NAME            READY     STATUS             RESTARTS   AGE
hooks-2-l4ryz   0/1       CrashLoopBackOff   1          51m
[root@ip-172-18-1-90 amd64]# oc describe pod hooks-2-l4ryz
Name:                hooks-2-l4ryz
Namespace:            zhouy
Image(s):            openshift/mysql-55-centos7:latest
Node:                ip-172-18-1-90/172.18.1.90
Start Time:            Sat, 10 Oct 2015 05:18:25 +0000
Labels:                deployment=hooks-2,deploymentconfig=hooks,name=mysql
Status:                Running
Reason:                
Message:            
IP:                172.17.0.22
Replication Controllers:    hooks-2 (1/1 replicas created)
Containers:
  mysql-55-centos7:
    Container ID:        docker://0368c644a3b87f1aec1cd3eaab62b5ef13106d606996161967eb7df7b9c6fbf1
    Image:            openshift/mysql-55-centos7:latest
    Image ID:            docker://0ca2fa46cd1776ddfad962cc647e392fb15dc5e75a838cf503b4e9744a9e960a
    State:            Waiting
      Reason:            CrashLoopBackOff
    Last Termination State:    Terminated
      Reason:            Error
      Exit Code:        127
      Started:            Sat, 10 Oct 2015 06:06:11 +0000
      Finished:            Sat, 10 Oct 2015 06:06:11 +0000
    Ready:            False
    Restart Count:        1
    Environment Variables:
      MYSQL_USER:    user8Y2
      MYSQL_PASSWORD:    Plqe5Wev
      MYSQL_DATABASE:    root
Conditions:
  Type        Status
  Ready     False 
Volumes:
  secret:
    Type:    Secret (a secret that should populate this volume)
    SecretName:    my-secret
  default-token-703lu:
    Type:    Secret (a secret that should populate this volume)
    SecretName:    default-token-703lu
Events:
  FirstSeen    LastSeen    Count    From                SubobjectPath                Reason        Message
  ─────────    ────────    ─────    ────                ─────────────                ──────        ───────
  51m        51m        1    {kubelet ip-172-18-1-90}    implicitly required container POD    Pulled        Container image "openshift/origin-pod:v1.0.6" already present on machine
  51m        51m        1    {scheduler }                                Scheduled    Successfully assigned hooks-2-l4ryz to ip-172-18-1-90
  51m        51m        1    {kubelet ip-172-18-1-90}    implicitly required container POD    Created        Created with docker id 8e197201f7f1
  19m        19m        1    {kubelet ip-172-18-1-90}    spec.containers{mysql-55-centos7}    Started        Started with docker id 9abb9ce38d7a
  13m        13m        1    {kubelet ip-172-18-1-90}    spec.containers{mysql-55-centos7}    Created        Created with docker id e9f9919b6f74
  13m        13m        1    {kubelet ip-172-18-1-90}    spec.containers{mysql-55-centos7}    Started        Started with docker id e9f9919b6f74
  8m        8m        1    {kubelet ip-172-18-1-90}    spec.containers{mysql-55-centos7}    Started        Started with docker id 8fdc8bad2257
  8m        8m        1    {kubelet ip-172-18-1-90}    spec.containers{mysql-55-centos7}    Created        Created with docker id 8fdc8bad2257
  51m        3m        15    {kubelet ip-172-18-1-90}    spec.containers{mysql-55-centos7}    Pulling        Pulling image "openshift/mysql-55-centos7:latest"
  51m        3m        15    {kubelet ip-172-18-1-90}    spec.containers{mysql-55-centos7}    Pulled        Successfully pulled image "openshift/mysql-55-centos7:latest"
  3m        3m        1    {kubelet ip-172-18-1-90}    spec.containers{mysql-55-centos7}    Started        Started with docker id 0368c644a3b8
  3m        3m        1    {kubelet ip-172-18-1-90}    spec.containers{mysql-55-centos7}    Created        Created with docker id 0368c644a3b8
  50m        2s        292    {kubelet ip-172-18-1-90}    spec.containers{mysql-55-centos7}    Backoff        Back-off restarting failed docker container

Expected results:
The pod should be running and the secret volume was added correctly.

Additional info:

Comment 1 Dan Mace 2015-10-13 20:28:52 UTC
I'm not sure that there's any specific issue with deployments here. The secret is correctly added to the deployment template, and the pod gets the correct volumes as a result of the template change. The pod container fails to start- you'll need to use `docker logs 0368c644a3b8` to see why the container process failed.

Comment 2 zhou ying 2015-10-19 10:34:34 UTC
The container was not correctly created:
[root@ip-172-18-2-48 amd64]# docker ps |grep hooks
a6ce064da063        openshift/origin-pod:v1.0.6               "/pod"                 2 minutes ago       Up 2 minutes                            k8s_POD.829cf5d9_hooks-2-7k3my_zhy_88e0fcfa-764c-11e5-a55c-0e78887570a7_f54e5a0e


oc logs hooks-2-7k3my
Pod "hooks-2-7k3my" in namespace "zhy": container "mysql-55-centos7" is in waiting state.

Comment 3 Dan Mace 2015-10-19 13:45:03 UTC
Can you attach your master logs when the container fails to be created? Also, the output of `oc get pod -o yaml`, `oc get dc -o yaml`, and `oc get rc -o yaml`.

If you could provide steps to reproduce that would work from any machine, that would also be helpful (your example uses paths which aren't readable by my cluster locally, which could be another problem).

Comment 5 Dan Mace 2015-10-21 13:33:59 UTC
From your pod output, we can see that the hooks pod is created correctly. The container 52eeb9d354c229d91da813aed6a7b028bb78ad2684d3d6c5f6994dcfa25e5aa8 is repeatedly failing to start. You'll need to look at the docker logs to see why. Please share those logs here.

Comment 6 zhou ying 2015-10-22 03:22:21 UTC
When I use the hello-pod , the secret volume can be used, please see:
http://pastebin.test.redhat.com/321731
http://pastebin.test.redhat.com/321733

But the mysql image still can use the secret volume, I use the `oc logs`:
[root@ip-172-18-0-45 amd64]# oc get pods
NAME               READY     STATUS      RESTARTS   AGE
hooks-3-deploy     1/1       Running     0          22s
hooks-3-posthook   1/1       Running     0          6s
hooks-3-prehook    0/1       Completed   0          21s
hooks-3-yqv4w      1/1       Running     0          7s
[root@ip-172-18-0-45 amd64]# oc logs -f hooks-3-yqv4w
Can't read /etc/scl/prefixes/mysql55, mysql55 is probably not installed.
/var/lib/mysql/common.sh: line 101: mysql_install_db: command not found
Running mysql_install_db ...
[root@ip-172-18-0-45 amd64]# oc logs -f hooks-3-yqv4w
Error from server: Internal error occurred: Pod "hooks-3-yqv4w" in namespace "zhouy" : pod is not in 'Running', 'Succeeded' or 'Failed' state - State: "Pending"

Maybe this bug is related the image:openshift/mysql-55-centos7:latest.

Comment 7 Ben Parees 2015-10-23 17:32:22 UTC
it appears you're configuring the secret to get mounted into "/etc":
4. Add the new secret to dc
   `oc volume dc/hooks  --add --name=secret --type=secret --secret-name=my-secret --mount-path=/etc`

that's a bad place to mount a secret because it's going to overwrite the entire contents of the /etc directory in the image, which is why mysql is failing to start with errors like:
Can't read /etc/scl/prefixes/mysql55

Can you change this test scenario to mount the secret in a location where it will not overwrite critical image files and try again?

Comment 8 zhou ying 2015-10-26 03:17:17 UTC
Confirmed on /home ,wonderful!
[root@ip-172-18-5-12 amd64]# oc volume dc/hooks  --add --name=secret --type=secret --secret-name=my-secret --mount-path=/home
deploymentconfigs/hooks

[root@ip-172-18-5-12 amd64]# oc get pods
NAME            READY     STATUS    RESTARTS   AGE
hooks-2-ejjhf   1/1       Running   0          5m
[root@ip-172-18-5-12 amd64]# oc exec hooks-2-ejjhf -- ls /home
ssh-privatekey
ssh-publickey