Hide Forgot
Description of problem: Create a pod, we can update pod's mutable fields such as metadata.labels, spec.activeDeadlineSeconds. Then create quota or limitrange in project, we fail to update the previous pod's mutable fields. Then create new pod, we succeed updating new pod's mutable fields. Plugin control may has sth wrong. Version-Release number of selected component (if applicable): openshift v1.1.6-97-gbe0abe5 kubernetes v1.2.0-36-g4a3f9c5 etcd 2.2.5 How reproducible: Always Steps to Reproduce: 1. oc login, create project xxia-proj 2. Create pod: $ oc new-app -f origin/examples/sample-app/application-template-stibuild.json 3. After pod database-1-zr2nr is running: 1> $ oc edit pod database-1-zr2nr # Add/change label, e.g. new: abc 2> $ oc edit pod database-1-zr2nr # Under spec, add/change activeDeadlineSeconds: 21600 4. Create limitrange in project （with cluster-admin）: $ oc create -f origin/examples/project-quota/limits.yaml --config=./admin.kubeconfig -n xxia-proj 5. Repeat 3 6. Delete limitrange, then create quota $ oc delete -f origin/examples/project-quota/limits.yaml --config=./admin.kubeconfig -n xxia-proj $ oc create -f origin/examples/project-quota/quota.yaml --config=./admin.kubeconfig -n xxia-proj 7. Repeat 3 8. With limitrange or quota existing, create new pod, e.g. by scaling: $ oc scale dc database --replicas=2 $ oc get pod database-1-x3omd 1/1 Running 0 41s database-1-zr2nr 1/1 Running 0 7h 9. Repeat 3 on new pod database-1-x3omd Actual results: 3. 1> and 2> both can update successfully. 5. Both update fail with message: spec: Forbidden: pod updates may not change fields other than `containers[*].image` or `spec.activeDeadlineSeconds` 7. Both update fail with message: error: pods "database-1-zr2nr" could not be patched: pods "database-1-zr2nr" is forbidden: Failed quota: quota: must specify cpu,memory 9. Both can update successfully on new pod. Expected results: Step 5 and 7 should succeed as step 3. And old pod before plugin creation should be updated successfully as same as new pod after plugin creation. Additional info: Only resource pod has this issue. Others such as dc doesn't have.
Step 5: The LimitRanger is intercepting CREATE and UPDATE operations for pods. The code is evaluating the updated pod against any active LimitRange(s) that are now present in the project. It is merging the default resource requirements in an UPDATE operation, and since the pod did not previously have resource requirements, and we do not yet allow dynamically updating a pods resource requirements, the UPDATE operation is causing an error. There had been a fix to have LimitRange ignore UPDATE operations for pods, but that appears to have regressed: https://github.com/kubernetes/kubernetes/pull/18477/files#diff-e3f052839796781484de133fea5d5b5dR62 I opened the upstream issue to track a fix: https://github.com/kubernetes/kubernetes/issues/24351
Step 7: The quota system is intercepting the CREATE and UPDATE operations. Since we do not yet support mutation of a pod's resource requirements on UPDATE, it should be safe to ignore pod updates all together. https://github.com/kubernetes/kubernetes/blob/master/pkg/quota/evaluator/core/pods.go#L50 I opened the upstream issue to track a fix: https://github.com/kubernetes/kubernetes/issues/24352
A correction to a previous comment on LimitRange and regression: >>There had been a fix to have LimitRange ignore UPDATE operations for pods, but that appears to have regressed: https://github.com/kubernetes/kubernetes/pull/18477/files#diff-e3f052839796781484de133fea5d5b5dR62 The above PR never merged into the code base, so there is no actual regression on LimitRange behavior to my knowledge, but a fix should still be made.
I do not think this should be a release blocker as best-practice should be to create a quota prior to creating resources in the project.
Similar discussion on issues that can arise if a quota is added post project creation is here: https://github.com/kubernetes/kubernetes/issues/22509
for the error reported in step 5, we will not update a limit range. right now, the presence of limit range precludes the ability to run pods that dont make resource requirements. for the error reported in step 7, the quota system has evolved to now let you write a quota that tracks pods that are NotBestEffort via scopes. for the quota in question for this scenario, the quota would not have rejected the pod if it had the scope NotBestEffort applied.