Bug 1569911 - local-storage-provisioner image v3.10 does not work in OCP 3.10
Summary: local-storage-provisioner image v3.10 does not work in OCP 3.10
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Documentation
Version: 3.10.0
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ---
: 3.10.0
Assignee: Vikram Goyal
QA Contact: Vikram Goyal
Vikram Goyal
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-04-20 08:55 UTC by Qin Ping
Modified: 2018-07-31 14:43 UTC (History)
7 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2018-07-31 14:43:30 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)

Description Qin Ping 2018-04-20 08:55:50 UTC
Description of problem:
local-storage-provisioner image v3.10 does not work in OCP 3.10

Version-Release number of selected component (if applicable):
oc v3.10.0-0.22.0
openshift v3.10.0-0.22.0
kubernetes v1.10.0+b81c8f8

# oc exec local-volume-provisioner-htqmh -- rpm -qa|grep local
openshift-external-storage-local-provisioner-0.0.2-2.gitd3c94f0.el7.x86_64

How reproducible:
Always

Steps to Reproduce:
1. Create a new project local-storage
2. Create configmap local-volume-config
3. Create serviceaccount local-storage-admin and add scc priviledged to him
4. Create template local-storage-provisioner using https://raw.githubusercontent.com/openshift/origin/master/examples/storage-examples/local-examples/local-storage-provisioner-template.yaml
5. Create a app with local-storage-provisioner v3.10 tag
6. Create dirs per configmap, and create mount point

Actual results:
No PV was created and get error:
github.com/kubernetes-incubator/external-storage/local-volume/provisioner/pkg/populator/populator.go:87: watch of *v1.PersistentVolume ended with: The resourceVersion for the provided watch is too old.

Expected results:
PV can be created successfully.

Master Log:

Node Log (of failed PODs):

PV Dump:

PVC Dump:

StorageClass Dump (if StorageClass used by PV/PVC):

Additional info:
# findmnt -o TARGET,PROPAGATION /
TARGET PROPAGATION
/      shared

# oc exec local-volume-provisioner-htqmh -- df
Filesystem                                                                                         1K-blocks    Used Available Use% Mounted on
/dev/mapper/docker-253:0-25166161-43ad8f56d49ca0d9030874c2f3f0bf72f08ea0879e92e612398255211bd4e60e  10475520  283312  10192208   3% /
tmpfs                                                                                                1940476       0   1940476   0% /dev
tmpfs                                                                                                1940476       0   1940476   0% /sys/fs/cgroup
/dev/mapper/rhel-root                                                                               18864128 2915824  15948304  16% /etc/hosts
shm                                                                                                    65536       0     65536   0% /dev/shm
tmpfs                                                                                                1940476      16   1940460   1% /run/secrets/kubernetes.io/serviceaccount
tmpfs                                                                                                1940476       0   1940476   0% /proc/scsi
tmpfs                                                                                                1940476       0   1940476   0% /sys/firmware
vol1                                                                                                 1940476       0   1940476   0% /mnt/local-storage/ssd/vol1

# oc export configmap
apiVersion: v1
items:
- apiVersion: v1
  data:
    local-hdd: |
      {
        "hostDir": "/mnt/local-storage/hdd",
        "mountDir": "/mnt/local-storage/hdd"
      }
    local-ssd: |
      {
        "hostDir": "/mnt/local-storage/ssd",
        "mountDir": "/mnt/local-storage/ssd"
      }
  kind: ConfigMap
  metadata:
    creationTimestamp: null
    name: local-volume-config
kind: List
metadata: {}

# oc logs local-volume-provisioner-htqmh
I0420 05:25:03.799492       1 main.go:42] Configuration parsing has been completed, ready to run...
I0420 05:25:03.808496       1 common.go:315] Creating client using in-cluster config
I0420 05:25:03.826278       1 main.go:52] Starting controller
I0420 05:25:03.826291       1 controller.go:42] Initializing volume cache
I0420 05:25:03.860328       1 populator.go:85] Starting Informer controller
I0420 05:25:03.860340       1 populator.go:89] Waiting for Informer initial sync
I0420 05:25:04.860509       1 controller.go:72] Controller started
W0420 05:42:46.878967       1 reflector.go:341] github.com/kubernetes-incubator/external-storage/local-volume/provisioner/pkg/populator/populator.go:87: watch of *v1.PersistentVolume ended with: The resourceVersion for the provided watch is too old.

Comment 3 Jan Safranek 2018-04-25 08:16:13 UTC
Upstream has changed ConfigMap format :-(.

- Now they expect yaml instead of json inside the config map
- And there is new top-level key

New ConfigMap for local-storage looks like this:

kind: ConfigMap
metadata:
  name: local-volume-config
data:
    storageClassMap: |
        local-ssd: 
            hostDir:  /mnt/local-storage/ssd
            mountDir: /mnt/local-storage/ssd
        local-hdd:
            hostDir: /mnt/local-storage/hdd
            mountDir: /mnt/local-storage/hdd


This needs to be documented. I'll change https://github.com/openshift/origin/tree/master/examples/storage-examples/local-examples and OSE 3.10 version of https://docs.openshift.com/container-platform/3.9/install_config/configuring_local.html

Someone else should add a release note that the format has changed. I would even add a note that since this is a technical preview feature we don't support upgrade from 3.9.

Comment 5 Jan Safranek 2018-04-25 08:59:26 UTC
Fix in Origin example: https://github.com/openshift/origin/pull/19502

Comment 6 Jan Safranek 2018-04-25 09:08:07 UTC
Fix in OSE docs: https://github.com/openshift/openshift-docs/pull/8899

Assigning to documentation team.

1. Please make sure the OSE PR gets reviewed and merged.
2. Add an item to release notes that the local provisioner configuration has changed.

Comment 7 Traci Morrison 2018-04-30 17:49:45 UTC
I reviewed and suggested edits in https://github.com/openshift/openshift-docs/pull/8899.

Vikram, do you want me to take ownership of this bug? Thanks.

Adding @ahardin to this bug since we need to add a note to the 3.10 release notes. Ashley, please let me know if you need me to create a PR to update the release notes.

Comment 8 Ashley Hardin 2018-04-30 18:01:44 UTC
Thanks, Traci.
You can capture the details of what needs to be added to the release notes in the tracker:
https://github.com/openshift/openshift-docs/issues/8651
Just add a new comment with the drafted content. Thanks!

Comment 9 Traci Morrison 2018-05-10 20:22:25 UTC
@Vikram, should this bug be assigned to Sprint 149? Thanks.


Note You need to log in before you can comment on or make changes to this bug.