Description of problem: local-storage-provisioner image v3.10 does not work in OCP 3.10 Version-Release number of selected component (if applicable): oc v3.10.0-0.22.0 openshift v3.10.0-0.22.0 kubernetes v1.10.0+b81c8f8 # oc exec local-volume-provisioner-htqmh -- rpm -qa|grep local openshift-external-storage-local-provisioner-0.0.2-2.gitd3c94f0.el7.x86_64 How reproducible: Always Steps to Reproduce: 1. Create a new project local-storage 2. Create configmap local-volume-config 3. Create serviceaccount local-storage-admin and add scc priviledged to him 4. Create template local-storage-provisioner using https://raw.githubusercontent.com/openshift/origin/master/examples/storage-examples/local-examples/local-storage-provisioner-template.yaml 5. Create a app with local-storage-provisioner v3.10 tag 6. Create dirs per configmap, and create mount point Actual results: No PV was created and get error: github.com/kubernetes-incubator/external-storage/local-volume/provisioner/pkg/populator/populator.go:87: watch of *v1.PersistentVolume ended with: The resourceVersion for the provided watch is too old. Expected results: PV can be created successfully. Master Log: Node Log (of failed PODs): PV Dump: PVC Dump: StorageClass Dump (if StorageClass used by PV/PVC): Additional info: # findmnt -o TARGET,PROPAGATION / TARGET PROPAGATION / shared # oc exec local-volume-provisioner-htqmh -- df Filesystem 1K-blocks Used Available Use% Mounted on /dev/mapper/docker-253:0-25166161-43ad8f56d49ca0d9030874c2f3f0bf72f08ea0879e92e612398255211bd4e60e 10475520 283312 10192208 3% / tmpfs 1940476 0 1940476 0% /dev tmpfs 1940476 0 1940476 0% /sys/fs/cgroup /dev/mapper/rhel-root 18864128 2915824 15948304 16% /etc/hosts shm 65536 0 65536 0% /dev/shm tmpfs 1940476 16 1940460 1% /run/secrets/kubernetes.io/serviceaccount tmpfs 1940476 0 1940476 0% /proc/scsi tmpfs 1940476 0 1940476 0% /sys/firmware vol1 1940476 0 1940476 0% /mnt/local-storage/ssd/vol1 # oc export configmap apiVersion: v1 items: - apiVersion: v1 data: local-hdd: | { "hostDir": "/mnt/local-storage/hdd", "mountDir": "/mnt/local-storage/hdd" } local-ssd: | { "hostDir": "/mnt/local-storage/ssd", "mountDir": "/mnt/local-storage/ssd" } kind: ConfigMap metadata: creationTimestamp: null name: local-volume-config kind: List metadata: {} # oc logs local-volume-provisioner-htqmh I0420 05:25:03.799492 1 main.go:42] Configuration parsing has been completed, ready to run... I0420 05:25:03.808496 1 common.go:315] Creating client using in-cluster config I0420 05:25:03.826278 1 main.go:52] Starting controller I0420 05:25:03.826291 1 controller.go:42] Initializing volume cache I0420 05:25:03.860328 1 populator.go:85] Starting Informer controller I0420 05:25:03.860340 1 populator.go:89] Waiting for Informer initial sync I0420 05:25:04.860509 1 controller.go:72] Controller started W0420 05:42:46.878967 1 reflector.go:341] github.com/kubernetes-incubator/external-storage/local-volume/provisioner/pkg/populator/populator.go:87: watch of *v1.PersistentVolume ended with: The resourceVersion for the provided watch is too old.
Upstream has changed ConfigMap format :-(. - Now they expect yaml instead of json inside the config map - And there is new top-level key New ConfigMap for local-storage looks like this: kind: ConfigMap metadata: name: local-volume-config data: storageClassMap: | local-ssd: hostDir: /mnt/local-storage/ssd mountDir: /mnt/local-storage/ssd local-hdd: hostDir: /mnt/local-storage/hdd mountDir: /mnt/local-storage/hdd This needs to be documented. I'll change https://github.com/openshift/origin/tree/master/examples/storage-examples/local-examples and OSE 3.10 version of https://docs.openshift.com/container-platform/3.9/install_config/configuring_local.html Someone else should add a release note that the format has changed. I would even add a note that since this is a technical preview feature we don't support upgrade from 3.9.
Fix in Origin example: https://github.com/openshift/origin/pull/19502
Fix in OSE docs: https://github.com/openshift/openshift-docs/pull/8899 Assigning to documentation team. 1. Please make sure the OSE PR gets reviewed and merged. 2. Add an item to release notes that the local provisioner configuration has changed.
I reviewed and suggested edits in https://github.com/openshift/openshift-docs/pull/8899. Vikram, do you want me to take ownership of this bug? Thanks. Adding @ahardin to this bug since we need to add a note to the 3.10 release notes. Ashley, please let me know if you need me to create a PR to update the release notes.
Thanks, Traci. You can capture the details of what needs to be added to the release notes in the tracker: https://github.com/openshift/openshift-docs/issues/8651 Just add a new comment with the drafted content. Thanks!
@Vikram, should this bug be assigned to Sprint 149? Thanks.
Content is updated with the release of OCP 3.10: https://access.redhat.com/documentation/en-us/openshift_container_platform/3.10/html/configuring_clusters/install-config-configuring-local