Bug 1454601
| Summary: | Provision PV in zone other than master failed with error "disk is not found" while disk exists | ||
|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | Liang Xia <lxia> |
| Component: | Storage | Assignee: | David Eads <deads> |
| Status: | CLOSED ERRATA | QA Contact: | Jianwei Hou <jhou> |
| Severity: | medium | Docs Contact: | |
| Priority: | medium | ||
| Version: | 3.6.0 | CC: | aos-bugs, eparis, hekumar, wmeng |
| Target Milestone: | --- | ||
| Target Release: | 3.7.0 | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | Enhancement | |
| Doc Text: |
Feature:
Fix static PV provisioning in multizone environments.
Reason:
Lack of Cloudprovider configuration in admission plugin that checks zone for PV provisioning can cause PV creation to fail when PV is being created in a zone other than master.
Result:
After the fix, users can statically provision PVs in zones other than master in multizone configurations.
|
Story Points: | --- |
| Clone Of: | Environment: | ||
| Last Closed: | 2017-11-28 21:56:17 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
Can you verify this with plain Kubernetes installation and see if it works? Wondering if something broke existing code. With the same pv.yaml as in #comment 0, pv can be created on k8s with below version, # kubectl version Client Version: version.Info{Major:"1", Minor:"7+", GitVersion:"v1.7.0-alpha.4.322+95a6f108bda5c4", GitCommit:"95a6f108bda5c4af79e78e4ebf613252c8c0fd5f", GitTreeState:"clean", BuildDate:"2017-05-24T01:31:11Z", GoVersion:"go1.8", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"7+", GitVersion:"v1.7.0-alpha.4.322+95a6f108bda5c4", GitCommit:"95a6f108bda5c4af79e78e4ebf613252c8c0fd5f", GitTreeState:"clean", BuildDate:"2017-05-24T01:31:11Z", GoVersion:"go1.8", Compiler:"gc", Platform:"linux/amd64"} Openshift has broken admission plugin initializer - https://github.com/openshift/origin/blob/master/pkg/cmd/server/start/start_master.go#L399 We are passing nil cloudprovider config to plugin initializer. Also, because plugin is initialized before cloudprovider config, cloudprovider config isn't available for plugin initialization. Opened https://github.com/openshift/origin/pull/14444 as a possibility, but its a problematic thing that should really fall out as the result of a more in-depth refactor. Marking this UpcomingRelease. David was not confident in his fix so lets merge it monday morning. That will leave use 3 weeks to make sure we don't make things worse. I have added a Trello card for myself to write an e2e test for this - https://trello.com/c/Ul2Ldjjn/508-write-an-e2e-test-for-multizone-fix Having said that, I did pull david's change and verified on real GCE mutlizone cluster (repeating same comment I left on PR). Verified this has been fixed on OCP 3.6.121 Once the bug is ON_QA, we can move it to verified. Move bug to verified. Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2017:3188 |
Description of problem: On a multi-zone cluster (with multizone enabled), creating PV in zone other than master failed with below error: "Error from server (Forbidden): error when creating "pv.yaml": persistentvolumes "pv2" is forbidden: error querying GCE PD volume disk1-us-central1-b: disk is not found", but actually, the volume/disk exists. Version-Release number of selected component (if applicable): openshift v3.6.74 kubernetes v1.6.1+5115d708d7 etcd 3.1.0 How reproducible: Always Steps to Reproduce: 1.Set up multi-zone cluster(1 master in us-central1-a, 2 nodes in us-central1-b) on GCE. 2.Create the volumes in us-central1-a, us-central1-b and us-central1-c. 3.Create the pv with above volumes. $ oc create -f pv.yaml apiVersion: "v1" kind: "PersistentVolume" metadata: name: "pv2" spec: capacity: storage: "1Gi" accessModes: - "ReadWriteOnce" gcePersistentDisk: fsType: "ext4" pdName: "disk1-us-central1-b" Actual results: PV with volume in us-central1-a(the same as master) can be created. PV with volume in us-central1-b/us-central1-c can _NOT_ be created. Failed as below, # oc create -f pv.yaml Error from server (Forbidden): error when creating "pv.yaml": persistentvolumes "pv2" is forbidden: error querying GCE PD volume disk1-us-central1-b: disk is not found Expected results: PV should be created successfully. Additional info: The disk really exist. $ gcloud compute disks list | grep disk disk1-us-central1-a us-central1-a 1 pd-standard READY disk1-us-central1-c us-central1-c 1 pd-standard READY disk1-us-central1-b us-central1-b 1 pd-standard READY