Bug 1364403
Summary: | [platformmanagement_public_713] Should give proper message and prevent further creation when resources usage exceed cluster quota | ||||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
Product: | OpenShift Container Platform | Reporter: | Qixuan Wang <qixuan.wang> | ||||||||||||||
Component: | Master | Assignee: | David Eads <deads> | ||||||||||||||
Status: | CLOSED NOTABUG | QA Contact: | weiwei jiang <wjiang> | ||||||||||||||
Severity: | medium | Docs Contact: | |||||||||||||||
Priority: | medium | ||||||||||||||||
Version: | 3.3.0 | CC: | aos-bugs, deads, jforrest, jokerman, mmccomas, qixuan.wang, sdodson, wsun | ||||||||||||||
Target Milestone: | --- | ||||||||||||||||
Target Release: | --- | ||||||||||||||||
Hardware: | Unspecified | ||||||||||||||||
OS: | Unspecified | ||||||||||||||||
Whiteboard: | |||||||||||||||||
Fixed In Version: | Doc Type: | If docs needed, set a value | |||||||||||||||
Doc Text: | Story Points: | --- | |||||||||||||||
Clone Of: | Environment: | ||||||||||||||||
Last Closed: | 2016-08-11 12:17:34 UTC | Type: | Bug | ||||||||||||||
Regression: | --- | Mount Type: | --- | ||||||||||||||
Documentation: | --- | CRM: | |||||||||||||||
Verified Versions: | Category: | --- | |||||||||||||||
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||||||||||||
Cloudforms Team: | --- | Target Upstream Version: | |||||||||||||||
Embargoed: | |||||||||||||||||
Attachments: |
|
Description
Qixuan Wang
2016-08-05 09:43:35 UTC
Created attachment 1187825 [details]
Exceeded quota
Update: The problem is in OSE(openshift v3.3.0.14, kubernetes v1.3.0+57fb9ac, etcd 2.3.0+git) On Origin(openshift v1.3.0-alpha.2+89b7193, kubernetes v1.3.0+507d3a7, etcd 2.3.0+git), the problem can't be reproduced. Origin has the correct warning: Error from server: secrets "mysecret-2" is forbidden: Exceeded quota: crq, requested: secrets=1, used: secrets=10, limited: secrets=10 Are you running the OSE from config? If so, can you provide the config? It's possible to specify a different set of admission plugins and that can prevent new ones from taking affect. This problem can't be reproduced in non-HA environment but exist in HA (2master+2infra_node+2node+3etcd). Attached master-config.yaml Created attachment 1188618 [details]
master config
Ok, I suspect that you're using a different master-config.yaml in your HA and non-HA configuration. In the one you linked, you're specifying: ```yaml admissionConfig: pluginOrderOverride: - NamespaceLifecycle - OriginPodNodeEnvironment - LimitRanger - ServiceAccount - SecurityContextConstraint - BuildDefaults - BuildOverrides - ResourceQuota - SCCExecRestrictions - AlwaysPullImages ``` That takes control of the admission chain. You should be getting a warning like this in your log, "specified admission ordering is being phased out". Because its being specified, you don't get new admission plugins including "ClusterResourceQuota". You can add "ClusterResourceQuota", but you really shouldn't be specifying the chain at all. Did you have to do it for some reason? Was it set up that way automatically? Yes QE's testing environment is setup by jenkins. There are "openshift_master_kube_admission_plugin_order" and "openshift_master_kube_admission_plugin_config" in "openshift_ansible_vars" options of HA environment config template but not in Non-HA config template. Part of Jenkins log which is setup HA job: #The following parameters is used by openshift-ansible openshift_master_kube_admission_plugin_order=["NamespaceLifecycle","OriginPodNodeEnvironment","LimitRanger","ServiceAccount","SecurityContextConstraint","BuildDefaults","BuildOverrides","ResourceQuota","SCCExecRestrictions","AlwaysPullImages"] openshift_master_kube_admission_plugin_config={"RunOnceDuration":{"configuration":{"apiVersion":"v1","kind":"RunOnceDurationConfig","activeDeadlineSecondsOverride":"3600"}},"ClusterResourceOverride":{"configuration":{"apiVersion":"v1","kind":"ClusterResourceOverrideConfig","limitCPUToMemoryPercent":"200","cpuRequestToLimitPercent":"6","memoryRequestToLimitPercent":"60"}},"PodNodeConstraints":{"configuration":{"apiVersion":"v1","kind":"PodNodeConstraintsConfig"}},"BuildOverrides":{"configuration":{"apiVersion":"v1","kind":"BuildOverridesConfig","forcePull":True}}} Capture master log: Aug 09 02:56:16 ip-172-18-14-59.ec2.internal atomic-openshift-master-controllers[13217]: W0809 02:56:16.506089 13217 start_master.go:272] kubernetesMasterConfig.admissionConfig.pluginOrderOverride: Invalid value: ["NamespaceLifecycle","OriginPodNodeEnvironment","LimitRanger","ServiceAccount","SecurityContextConstraint","BuildDefaults","BuildOverrides","ResourceQuota","SCCExecRestrictions","AlwaysPullImages"]: specified admission ordering is being phased out. Convert to DefaultAdmissionConfig in admissionConfig.pluginConfig. I think perhaps QE wrote incomplete ansible variables. The log shows "Convert to DefaultAdmissionConfig in admissionConfig.pluginConfig". The "DefaultAdmissionConfig" should be the same with Non-HA master-config, but these above admission plugins are still added into master-config. Doesn't "convertion" happen? HA: kubernetesMasterConfig: admissionConfig: pluginOrderOverride: - NamespaceLifecycle - OriginPodNodeEnvironment - LimitRanger - ServiceAccount - SecurityContextConstraint - BuildDefaults - BuildOverrides - ResourceQuota - SCCExecRestrictions - AlwaysPullImages pluginConfig: BuildOverrides: configuration: apiVersion: v1 forcePull: true kind: BuildOverridesConfig ClusterResourceOverride: configuration: apiVersion: v1 cpuRequestToLimitPercent: '6' kind: ClusterResourceOverrideConfig limitCPUToMemoryPercent: '200' memoryRequestToLimitPercent: '60' PodNodeConstraints: configuration: apiVersion: v1 kind: PodNodeConstraintsConfig RunOnceDuration: configuration: activeDeadlineSecondsOverride: '3600' apiVersion: v1 kind: RunOnceDurationConfig Non-HA: kubernetesMasterConfig: admissionConfig: pluginConfig: {} Attached files, hope these help. Created attachment 1189189 [details]
ha-master-config.yaml
Created attachment 1189190 [details]
non-ha-master-config.yaml
Created attachment 1189192 [details]
ha-atomic-openshift-master-controllers.log
Created attachment 1189193 [details]
ha-atomic-openshift-master-api.log
> "DefaultAdmissionConfig" should be the same with Non-HA master-config, but
> these above admission plugins are still added into master-config.
Sorry, please ignore "these above admission plugins are still added into master-config". I mean since it's an invalid configuration, the behavior should be the same with "DefaultAdmissionConfig", but it seems not convert to DefaultAdmissionConfig.
@Scott: are we encouraging people to set these admission values? @Qixuan Wang: You need to either add `ClusterResourceQuota` to the bottom of your list or you need to stop specifying the values. The current configuration is saying to *NOT* run the admission plugin that enforces quota. (In reply to Qixuan Wang from comment #7) > Yes QE's testing environment is setup by jenkins. There are > "openshift_master_kube_admission_plugin_order" and > "openshift_master_kube_admission_plugin_config" in "openshift_ansible_vars" > options of HA environment config template but not in Non-HA config template. Ok, that's an installer bug we should fix. (In reply to David Eads from comment #13) > @Scott: are we encouraging people to set these admission values? Encourage no, but we enable them to set admission plugin config. If they're shooting themselves not much we can do about that. @scott: I want to remove that knob from the master-config in two releases. What does it take to get there from here in ansible? We're combining the admission chains and we're providing a different on/off mechanism. (In reply to David Eads from comment #15) > @scott: I want to remove that knob from the master-config in two releases. > What does it take to get there from here in ansible? > > We're combining the admission chains and we're providing a different on/off > mechanism. When the time comes, file an issue in openshift-ansible and link it to the origin PR that drops it from the config. The ClusterResourceQuota admission plugin needs to be enabled. This can be done by adding to the list or by not specifying the list. Not specifying is preferred. Adding "ClusterResourceQuota" instead of "ResourceQuota" can get expected result. Thanks. |