Bug 1569911

Summary: local-storage-provisioner image v3.10 does not work in OCP 3.10
Product: OpenShift Container Platform Reporter: Qin Ping <piqin>
Component: DocumentationAssignee: Vikram Goyal <vigoyal>
Status: CLOSED CURRENTRELEASE QA Contact: Vikram Goyal <vigoyal>
Severity: high Docs Contact: Vikram Goyal <vigoyal>
Priority: high    
Version: 3.10.0CC: ahardin, aos-bugs, aos-storage-staff, jokerman, mmccomas, piqin, vigoyal
Target Milestone: ---   
Target Release: 3.10.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-07-31 14:43:30 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Qin Ping 2018-04-20 08:55:50 UTC
Description of problem:
local-storage-provisioner image v3.10 does not work in OCP 3.10

Version-Release number of selected component (if applicable):
oc v3.10.0-0.22.0
openshift v3.10.0-0.22.0
kubernetes v1.10.0+b81c8f8

# oc exec local-volume-provisioner-htqmh -- rpm -qa|grep local
openshift-external-storage-local-provisioner-0.0.2-2.gitd3c94f0.el7.x86_64

How reproducible:
Always

Steps to Reproduce:
1. Create a new project local-storage
2. Create configmap local-volume-config
3. Create serviceaccount local-storage-admin and add scc priviledged to him
4. Create template local-storage-provisioner using https://raw.githubusercontent.com/openshift/origin/master/examples/storage-examples/local-examples/local-storage-provisioner-template.yaml
5. Create a app with local-storage-provisioner v3.10 tag
6. Create dirs per configmap, and create mount point

Actual results:
No PV was created and get error:
github.com/kubernetes-incubator/external-storage/local-volume/provisioner/pkg/populator/populator.go:87: watch of *v1.PersistentVolume ended with: The resourceVersion for the provided watch is too old.

Expected results:
PV can be created successfully.

Master Log:

Node Log (of failed PODs):

PV Dump:

PVC Dump:

StorageClass Dump (if StorageClass used by PV/PVC):

Additional info:
# findmnt -o TARGET,PROPAGATION /
TARGET PROPAGATION
/      shared

# oc exec local-volume-provisioner-htqmh -- df
Filesystem                                                                                         1K-blocks    Used Available Use% Mounted on
/dev/mapper/docker-253:0-25166161-43ad8f56d49ca0d9030874c2f3f0bf72f08ea0879e92e612398255211bd4e60e  10475520  283312  10192208   3% /
tmpfs                                                                                                1940476       0   1940476   0% /dev
tmpfs                                                                                                1940476       0   1940476   0% /sys/fs/cgroup
/dev/mapper/rhel-root                                                                               18864128 2915824  15948304  16% /etc/hosts
shm                                                                                                    65536       0     65536   0% /dev/shm
tmpfs                                                                                                1940476      16   1940460   1% /run/secrets/kubernetes.io/serviceaccount
tmpfs                                                                                                1940476       0   1940476   0% /proc/scsi
tmpfs                                                                                                1940476       0   1940476   0% /sys/firmware
vol1                                                                                                 1940476       0   1940476   0% /mnt/local-storage/ssd/vol1

# oc export configmap
apiVersion: v1
items:
- apiVersion: v1
  data:
    local-hdd: |
      {
        "hostDir": "/mnt/local-storage/hdd",
        "mountDir": "/mnt/local-storage/hdd"
      }
    local-ssd: |
      {
        "hostDir": "/mnt/local-storage/ssd",
        "mountDir": "/mnt/local-storage/ssd"
      }
  kind: ConfigMap
  metadata:
    creationTimestamp: null
    name: local-volume-config
kind: List
metadata: {}

# oc logs local-volume-provisioner-htqmh
I0420 05:25:03.799492       1 main.go:42] Configuration parsing has been completed, ready to run...
I0420 05:25:03.808496       1 common.go:315] Creating client using in-cluster config
I0420 05:25:03.826278       1 main.go:52] Starting controller
I0420 05:25:03.826291       1 controller.go:42] Initializing volume cache
I0420 05:25:03.860328       1 populator.go:85] Starting Informer controller
I0420 05:25:03.860340       1 populator.go:89] Waiting for Informer initial sync
I0420 05:25:04.860509       1 controller.go:72] Controller started
W0420 05:42:46.878967       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/local-volume/provisioner/pkg/populator/populator.go:87: watch of *v1.PersistentVolume ended with: The resourceVersion for the provided watch is too old.

Comment 3 Jan Safranek 2018-04-25 08:16:13 UTC
Upstream has changed ConfigMap format :-(.

- Now they expect yaml instead of json inside the config map
- And there is new top-level key

New ConfigMap for local-storage looks like this:

kind: ConfigMap
metadata:
  name: local-volume-config
data:
    storageClassMap: |
        local-ssd: 
            hostDir:  /mnt/local-storage/ssd
            mountDir: /mnt/local-storage/ssd
        local-hdd:
            hostDir: /mnt/local-storage/hdd
            mountDir: /mnt/local-storage/hdd


This needs to be documented. I'll change https://github.com/openshift/origin/tree/master/examples/storage-examples/local-examples and OSE 3.10 version of https://docs.openshift.com/container-platform/3.9/install_config/configuring_local.html

Someone else should add a release note that the format has changed. I would even add a note that since this is a technical preview feature we don't support upgrade from 3.9.

Comment 5 Jan Safranek 2018-04-25 08:59:26 UTC
Fix in Origin example: https://github.com/openshift/origin/pull/19502

Comment 6 Jan Safranek 2018-04-25 09:08:07 UTC
Fix in OSE docs: https://github.com/openshift/openshift-docs/pull/8899

Assigning to documentation team.

1. Please make sure the OSE PR gets reviewed and merged.
2. Add an item to release notes that the local provisioner configuration has changed.

Comment 7 Traci Morrison 2018-04-30 17:49:45 UTC
I reviewed and suggested edits in https://github.com/openshift/openshift-docs/pull/8899.

Vikram, do you want me to take ownership of this bug? Thanks.

Adding @ahardin to this bug since we need to add a note to the 3.10 release notes. Ashley, please let me know if you need me to create a PR to update the release notes.

Comment 8 Ashley Hardin 2018-04-30 18:01:44 UTC
Thanks, Traci.
You can capture the details of what needs to be added to the release notes in the tracker:
https://github.com/openshift/openshift-docs/issues/8651
Just add a new comment with the drafted content. Thanks!

Comment 9 Traci Morrison 2018-05-10 20:22:25 UTC
@Vikram, should this bug be assigned to Sprint 149? Thanks.