Bug 1371872

Can you try adding "multizone = true" to your gce.conf file?

And restarting your master

This is failing because of https://github.com/kubernetes/kubernetes/issues/27656. There is a workaround that I'm trying now.

Thanks, Andy.
The workaround works.
On GCE, if create pv in a zone different from OpenShift master's zone, zone info should be explicitly specified. (for AWS, no need to do that)

Brad, can you make sure the docs include this workaround information? We'll try to dig deeper in the upcoming release.

This is really a Kube admission wiring bug. Probably better to reassign to CI. Earliest we could get this is 3.5.

Analysis:

In https://github.com/kubernetes/kubernetes/blob/ef0c9f0c5b8efbba948a0be2c98d9d2e32e0b68c/plugin/pkg/admission/persistentvolume/label/admission.go#L180, it creates the GCE cloud provider without specifying a config file. Because of this, the admission plugin doesn't know that the admin wants multizone=true.

We currently have a couple of places in the master config where it's possible to specify the cloud provider and cloud config file. Unfortunately, due to the way that admission plugins are currently initialized, they can't reuse these same settings, so we'll have to use a different mechanism (and potentially create yet another place where you have to specify the cloud provider and cloud config file).

Looks like this is still a bug.

Looking a little further - this affects AWS and GCE, both of which the admission controller initializes without passing a config.

moving to storage component.

The PersistentVolumeLabel admission controller could fix this problem in one of two ways.

Option 1: use per admission control configuration file
https://github.com/kubernetes/kubernetes/pull/39109

Option 2: extend admission control with a "WantsCloudConfig" interface
this is probably preferred, and is a well-established pattern.
a similar PR that is doing the same for "WantsToRun" is available here for reference:
https://github.com/kubernetes/kubernetes/pull/37148

I would prefer Option 2 as I suspect the cloud configuration is generally desired across multiple plug-ins and users would not want to configure the same information in multiple places.

Yeah I like option#2 as well. I am half way there, the only mini minor problem I see is -  as plugins require more such "special things" constructor in NewPluginInitializer (https://github.com/kubernetes/kubernetes/blob/master/cmd/kube-apiserver/app/server.go#L277) will get 'crowded'. Part of me thinks, the PluginInitializer being an interface is a problem - it doesn't do much as an interface can easily be replaced with concrete implementation itself and thereby removing this construction pain.

Opened a PR to fix this upstream - https://github.com/kubernetes/kubernetes/pull/40537 It is ready for review, but I am testing it on GCE and stuff.

Verified the workaround works. Move bug to verified as we can track the code fix in trello.

# openshift version
openshift v3.5.0.20+87266c6
kubernetes v1.5.2+43a9be4
etcd 3.1.0

# oc create -f pv1.json 
Error from server (Forbidden): error when creating "pv1.json": persistentvolumes "pv1" is forbidden: error querying GCE PD volume gcepd: GCE persistent disk "gcepd" not found in managed zones (us-central1-a)

# oc create -f pv2.json
persistentvolume "pv2" created
# oc get pv
NAME           CAPACITY   ACCESSMODES   RECLAIMPOLICY   STATUS      CLAIM                 REASON    AGE
pv2         5Gi        RWO           Retain          Available                                   3s

# diff pv1.json pv2.json 
1,17c1,15
< {
<   "apiVersion": "v1",
<   "kind": "PersistentVolume",
<   "metadata": {
<     "name": "pv1"
<   },
<   "spec": {
<     "capacity": {
<         "storage": "5Gi"
<     },
<     "accessModes": [ "ReadWriteOnce" ],
<     "gcePersistentDisk": {
<         "pdName": "gcepd",
<         "fsType": "ext4"
<     }
<   }
< }
---
> apiVersion: "v1"
> kind: "PersistentVolume"
> metadata:
>   name: "pv2"
>   labels:
>     failure-domain.beta.kubernetes.io/region: "us-central1"
>     failure-domain.beta.kubernetes.io/zone: "us-central1-b"
> spec:
>   capacity:
>     storage: "5Gi"
>   accessModes:
>     - "ReadWriteOnce"
>   gcePersistentDisk:
>     fsType: "ext4"
>     pdName: "gcepd"


$ gcloud compute disks list gcepd
NAME   ZONE           SIZE_GB  TYPE         STATUS
gcepd  us-central1-b  5        pd-standard  READY

Why is this bug marked with a Target Release of 3.4.1, when the Verification was for 3.5.0.
Can we move this to a target of 3.5.1?

Hi Hemant,

Could you confirm the concern in #comment 24 ?

Thanks

I am not sure why this was marked for target of 3.4.1. This bug only fixes documentation to have proper workaround for fixing the problem in both 3.4 and 3.5. So documentation fix is applicable to both versions (3.4 and 3.5)

Also this bug has been fixed for good in 3.6 - https://trello.com/c/MskIyxux/422-enable-cloudprovider-configuration-in-admission-controller and documented workaround is not needed with 3.6.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2017:1235