Bug 1346777

Summary: [dev-preview-int] Lack of default secrets after creating new project
Product: OpenShift Online Reporter: Bing Li <bingli>
Component: MasterAssignee: Jordan Liggitt <jliggitt>
Status: CLOSED CURRENTRELEASE QA Contact: weiwei jiang <wjiang>
Severity: high Docs Contact:
Priority: unspecified    
Version: 3.xCC: aos-bugs, deads, jokerman, mfojtik, mmccomas, wsun
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 1348319 (view as bug list) Environment:
Last Closed: 2016-10-04 13:07:48 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1303130, 1348319, 1387755    
Attachments:
Description Flags
logs from master of INT online env none

Description Bing Li 2016-06-15 10:41:36 UTC
Description of problem:
some secrets cannot be created after creating new project sometimes, even after a long time.

Version-Release number of selected component (if applicable):
dev-preview-int
docker-1.9.1-40.el7.x86_64
atomic-openshift-3.2.1.1-1.git.0.96f9555.el7.x86_64
kernel-3.10.0-327.18.2.el7.x86_64

How reproducible:
Sometimes

Steps to Reproduce:
1. Create new project
$ oc new-project kdslwqe

2. Check the default secrets and serviceaccount.

Actual results:
2. Only 8 secrets were created sometimes:
$ oc get sa
NAME       SECRETS   AGE
builder    2         11m
default    1         11m
deployer   2         11m
$ oc get secret
NAME                       TYPE                                  DATA      AGE
builder-dockercfg-94b0p    kubernetes.io/dockercfg               1         11m
builder-token-h8olx        kubernetes.io/service-account-token   3         11m
builder-token-hg93l        kubernetes.io/service-account-token   3         11m
default-token-1rg3l        kubernetes.io/service-account-token   3         11m
default-token-7uhis        kubernetes.io/service-account-token   3         11m
deployer-dockercfg-sn0h7   kubernetes.io/dockercfg               1         11m
deployer-token-9kpi8       kubernetes.io/service-account-token   3         11m
deployer-token-q2vbg       kubernetes.io/service-account-token   3         11m

Expected results:
2. All secrets should be created like below:
$ oc get sa
NAME       SECRETS   AGEgggGG
builder    2         7s
default    2         7s
deployer   2         7s
$ oc get secret
NAME                       TYPE                                  DATA      AGE
builder-dockercfg-jffus    kubernetes.io/dockercfg               1         1m
builder-token-2yb5s        kubernetes.io/service-account-token   3         1m
builder-token-8ecds        kubernetes.io/service-account-token   3         1m
default-dockercfg-fpx52    kubernetes.io/dockercfg               1         1m
default-token-21myo        kubernetes.io/service-account-token   3         1m
default-token-6jsnt        kubernetes.io/service-account-token   3         1m
default-token-xal7m        kubernetes.io/service-account-token   3         1m
deployer-dockercfg-m4ye3   kubernetes.io/dockercfg               1         1m
deployer-token-3tdsr       kubernetes.io/service-account-token   3         1m
deployer-token-ds2an       kubernetes.io/service-account-token   3         1m

Comment 1 Jordan Liggitt 2016-06-16 17:51:45 UTC
Looks like the api token secrets are present, but the dockercfg secret is not. That seems odd... the service-account-token secret is always created first, and the fact that it exists means quota enforcement is allowing secrets in.

Do we have higher-verbosity logs from the master from when this is happening?

Comment 2 Bing Li 2016-06-20 07:21:38 UTC
Created attachment 1169738 [details]
logs from master of INT online env

Tried to create two projects with random name "kdhlejh" and "kdnfls", and the bug was reproduced, then catch logs from master of online INT.

# oc new-project kdhlejh

# oc get sa
NAME       SECRETS   AGE
builder    1         5s
default    1         5s
deployer   2         5s
# oc get secret
NAME                       TYPE                                  DATA      AGE
builder-token-d7a58        kubernetes.io/service-account-token   3         11s
default-token-w8615        kubernetes.io/service-account-token   3         11s
default-token-z7t2a        kubernetes.io/service-account-token   3         11s
deployer-dockercfg-0fxr4   kubernetes.io/dockercfg               1         11s
deployer-token-1qk3p       kubernetes.io/service-account-token   3         11s
deployer-token-g6g0b       kubernetes.io/service-account-token   3         11s


# oc new-project kdnfls

# oc get sa
NAME       SECRETS   AGE
builder    1         14s
default    1         14s
deployer   2         14s
# oc get secret
NAME                       TYPE                                  DATA      AGE
builder-token-q681u        kubernetes.io/service-account-token   3         7s
default-token-9o4g2        kubernetes.io/service-account-token   3         7s
default-token-m40xx        kubernetes.io/service-account-token   3         7s
deployer-dockercfg-qhsic   kubernetes.io/dockercfg               1         6s
deployer-token-7nz1q       kubernetes.io/service-account-token   3         7s
deployer-token-tlylv       kubernetes.io/service-account-token   3         7s

Comment 4 Bing Li 2016-06-21 02:50:12 UTC
Created new project more than 20 times on dev-preview-stg, default api token secret and dockercfg secret for each serviceaccount could be created successfully like below:
$ oc get sa
NAME       SECRETS   AGE
builder    2         56s
default    2         56s
deployer   2         56s

But it seems the sum of the secrets differs each time, is this normal? How many secrets an serviceaccount have by default?
Below are 3 projects with 11, 9, 12 secrets separately:
$ oc new-project djfk134
$ oc get secret
NAME                       TYPE                                  DATA      AGE
builder-dockercfg-00si8    kubernetes.io/dockercfg               1         14s
builder-token-0war1        kubernetes.io/service-account-token   3         14s
builder-token-8b72y        kubernetes.io/service-account-token   3         15s
builder-token-jb0v9        kubernetes.io/service-account-token   3         14s
default-dockercfg-41m4q    kubernetes.io/dockercfg               1         15s
default-token-vmj2y        kubernetes.io/service-account-token   3         15s
default-token-wamkw        kubernetes.io/service-account-token   3         15s
default-token-yzse5        kubernetes.io/service-account-token   3         15s
deployer-dockercfg-aanky   kubernetes.io/dockercfg               1         15s
deployer-token-fagoo       kubernetes.io/service-account-token   3         15s
deployer-token-h9uj2       kubernetes.io/service-account-token   3         15s
$ oc new-project jdfak1
$ oc get secret
NAME                       TYPE                                  DATA      AGE
builder-dockercfg-7jwac    kubernetes.io/dockercfg               1         7s
builder-token-9uzs9        kubernetes.io/service-account-token   3         7s
builder-token-aihs4        kubernetes.io/service-account-token   3         7s
default-dockercfg-prf71    kubernetes.io/dockercfg               1         7s
default-token-cz68p        kubernetes.io/service-account-token   3         7s
default-token-g4dt7        kubernetes.io/service-account-token   3         7s
deployer-dockercfg-1y0qj   kubernetes.io/dockercfg               1         7s
deployer-token-gc1ap       kubernetes.io/service-account-token   3         7s
deployer-token-jagw6       kubernetes.io/service-account-token   3         7s
$ oc new-project kfe2
]$ oc get secret 
NAME                       TYPE                                  DATA      AGE
builder-dockercfg-25pn3    kubernetes.io/dockercfg               1         13s
builder-token-8yzqd        kubernetes.io/service-account-token   3         13s
builder-token-d2kmk        kubernetes.io/service-account-token   3         13s
builder-token-la07h        kubernetes.io/service-account-token   3         13s
default-dockercfg-n17lw    kubernetes.io/dockercfg               1         13s
default-token-1p415        kubernetes.io/service-account-token   3         13s
default-token-78ahs        kubernetes.io/service-account-token   3         13s
default-token-w9d3v        kubernetes.io/service-account-token   3         13s
deployer-dockercfg-4ca01   kubernetes.io/dockercfg               1         13s
deployer-token-39ah3       kubernetes.io/service-account-token   3         13s
deployer-token-jbcps       kubernetes.io/service-account-token   3         13s
deployer-token-k3ixv       kubernetes.io/service-account-token   3         13s

Comment 6 Bing Li 2016-06-21 06:39:51 UTC
Verified on dev-preview-stg:

$ oc get sa
NAME       SECRETS   AGE
builder    2         6s
default    2         6s
deployer   2         6s
$ oc describe sa builder
Name:		builder
Namespace:	fkdre2h
Labels:		<none>

Image pull secrets:	builder-dockercfg-317eb

Mountable secrets: 	builder-token-1lqxv
                   	builder-dockercfg-317eb

Tokens:            	builder-token-1lqxv
                   	builder-token-ffedv
                   	builder-token-nc2yg
$ oc describe sa default
Name:		default
Namespace:	fkdre2h
Labels:		<none>

Image pull secrets:	default-dockercfg-f9bxu

Mountable secrets: 	default-token-llds7
                   	default-dockercfg-f9bxu

Tokens:            	default-token-ewd54
                   	default-token-llds7
                   	default-token-qnojx
$ oc describe sa deployer
Name:		deployer
Namespace:	fkdre2h
Labels:		<none>

Image pull secrets:	deployer-dockercfg-u33bg

Mountable secrets: 	deployer-token-0m0bt
                   	deployer-dockercfg-u33bg

Tokens:            	deployer-token-0m0bt
                   	deployer-token-qe5nq
$ oc get secret
NAME                       TYPE                                  DATA      AGE
builder-dockercfg-317eb    kubernetes.io/dockercfg               1         18s
builder-token-1lqxv        kubernetes.io/service-account-token   3         19s
builder-token-ffedv        kubernetes.io/service-account-token   3         19s
builder-token-nc2yg        kubernetes.io/service-account-token   3         19s
default-dockercfg-f9bxu    kubernetes.io/dockercfg               1         18s
default-token-ewd54        kubernetes.io/service-account-token   3         18s
default-token-llds7        kubernetes.io/service-account-token   3         19s
default-token-qnojx        kubernetes.io/service-account-token   3         19s
deployer-dockercfg-u33bg   kubernetes.io/dockercfg               1         18s
deployer-token-0m0bt       kubernetes.io/service-account-token   3         19s
deployer-token-qe5nq       kubernetes.io/service-account-token   3         19s