Bug 1669080

Description of problem:
CNS installation failed with "Unable to add device: Device /dev/vsda not found."

Version-Release number of the following components:
openshift-ansible-3.11.72-1.git.0.7c8b4f0.el7.noarch.rpm

How reproducible:
Always

Steps to Reproduce:
1. Deploy OCP 3.11 with glusterfs group
2.
3.

Actual results:
CNS installation failed.
fatal: [*.host.com]: FAILED! => {"changed": true, "cmd": ["oc", "--config=/tmp/openshift-glusterfs-ansible-BQ4KQX/admin.kubeconfig", "rsh", "--namespace=glusterfs", "deploy-heketi-storage-1-562fd", "heketi-cli", "-s", "http://localhost:8080", "--user", "admin", "--secret", "ufPFoif6F5UP/e/w8vOIKZ5xQUfimkyvg+wunYcT6/Q=", "topology", "load", "--json=/tmp/openshift-glusterfs-ansible-BQ4KQX/topology.json", "2>&1"], "delta": "0:00:04.942068", "end": "2019-01-21 05:02:11.234561", "failed_when_result": true, "rc": 0, "start": "2019-01-21 05:02:06.292493", "stderr": "", "stderr_lines": [], "stdout": "Creating cluster ... ID: 8934e337e78817423a56e7d6cc7d0e3f\n\tAllowing file volumes on cluster.\n\tAllowing block volumes on cluster.\n\tCreating node flexy-gluster-mchomaglusterfs-node-1 ... ID: fdff635ada4832278cea78a827dc1042\n\t\tAdding device /dev/vsda ... Unable to add device: Device /dev/vsda not found.\n\tCreating node flexy-gluster-mchomaglusterfs-node-2 ... ID: 2232b6d20102586b56210a1200c2028b\n\t\tAdding device /dev/vsda ... Unable to add device: Device /dev/vsda not found.\n\tCreating node flexy-gluster-mchomaglusterfs-node-3 ... ID: 63f4b4900c1e2b8bd64d8a16fb72aa6e\n\t\tAdding device /dev/vsda ... Unable to add device: Device /dev/vsda not found.", "stdout_lines": ["Creating cluster ... ID: 8934e337e78817423a56e7d6cc7d0e3f", "\tAllowing file volumes on cluster.", "\tAllowing block volumes on cluster.", "\tCreating node flexy-gluster-mchomaglusterfs-node-1 ... ID: fdff635ada4832278cea78a827dc1042", "\t\tAdding device /dev/vsda ... Unable to add device: Device /dev/vsda not found.", "\tCreating node flexy-gluster-mchomaglusterfs-node-2 ... ID: 2232b6d20102586b56210a1200c2028b", "\t\tAdding device /dev/vsda ... Unable to add device: Device /dev/vsda not found.", "\tCreating node flexy-gluster-mchomaglusterfs-node-3 ... ID: 63f4b4900c1e2b8bd64d8a16fb72aa6e", "\t\tAdding device /dev/vsda ... Unable to add device: Device /dev/vsda not found."]}

Expected results:
CNS installation successes.

Additional info:

$ cat /tmp/openshift-glusterfs-ansible-BQ4KQX/topology.json 
{
  "clusters": [{
      "nodes": [{
          "node": {
            "hostnames": {
              "manage": ["flexy-gluster-mchomaglusterfs-node-1"],
              "storage": ["172.16.120.54"]
            },
            "zone": 1
          },
          "devices": ["/dev/vsda"]
        },{
          "node": {
            "hostnames": {
              "manage": ["flexy-gluster-mchomaglusterfs-node-2"],
              "storage": ["172.16.120.4"]
            },
            "zone": 1
          },
          "devices": ["/dev/vsda"]
        },{
          "node": {
            "hostnames": {
              "manage": ["flexy-gluster-mchomaglusterfs-node-3"],
              "storage": ["172.16.120.88"]
            },
            "zone": 1
          },
          "devices": ["/dev/vsda"]
        }]
    }]
}

device in gluster node host:
# ls /dev/vsda -l
lrwxrwxrwx. 1 root root 10 Jan 21 04:37 /dev/vsda -> /dev/loop0

device in gluster-storage pod:
# oc exec glusterfs-storage-kjmfm -- ls /dev/vsda
ls: cannot access /dev/vsda: No such file or directory
command terminated with exit code 2

# oc exec glusterfs-storage-kjmfm -- ls /mnt/host-dev/vsda -l
lrwxrwxrwx. 1 root root 10 Jan 21 09:37 /mnt/host-dev/vsda -> /dev/loop0

# oc get ds glusterfs-storage -oyaml
apiVersion: extensions/v1beta1
...
        volumeMounts:
        ...
        - mountPath: /mnt/host-dev
          name: glusterfs-dev
        ...
      volumes:
      ...
      - hostPath:
          path: /dev
          type: ""
        name: glusterfs-dev
...