Bug 1622172

Summary: Logging & metrics deployment fail on independent mode of OCS
Product: [Red Hat Storage] Red Hat Gluster Storage Reporter: krishnaram Karthick <kramdoss>
Component: CNS-deploymentAssignee: Michael Adam <madam>
Status: CLOSED WORKSFORME QA Contact:
Severity: urgent Docs Contact:
Priority: unspecified    
Version: cns-3.10CC: akhakhar, asambast, hchiramm, jarrpa, khartsoe, knarra, kramdoss, madam, musoni, pprakash, rhs-bugs, rtalur, sankarshan, sasundar
Target Milestone: ---Flags: knarra: needinfo-
knarra: needinfo-
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-04-17 06:54:53 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
ansible logs
none
inventory file none

Description krishnaram Karthick 2018-08-24 15:23:13 UTC
Description of problem:
Logging & metrics deployment fails on independent mode configuration of OCS as the PVCs for logging & metrics hasn't got created successfully.

[root@ms-master ~]# oc describe pvc/metrics-cassandra-1 -n openshift-infra
Name:          metrics-cassandra-1
Namespace:     openshift-infra
StorageClass:  
Status:        Pending
Volume:        
Labels:        metrics-infra=hawkular-cassandra
Annotations:   kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"v1","kind":"PersistentVolumeClaim","metadata":{"annotations":{},"labels":{"metrics-infra":"hawkular-cassandra"},"name":"metrics-cassandr...
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:      
Access Modes:  
Events:
  Type    Reason         Age                From                         Message
  ----    ------         ----               ----                         -------
  Normal  FailedBinding  4m (x953 over 3h)  persistentvolume-controller  no persistent volumes available for this claim and no storage class is set


oc describe pvc/logging-es-0 -n openshift-logging
Name:          logging-es-0
Namespace:     openshift-logging
StorageClass:  
Status:        Pending
Volume:        
Labels:        logging-infra=support
Annotations:   <none>
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:      
Access Modes:  
Events:
  Type    Reason         Age                From                         Message
  ----    ------         ----               ----                         -------
  Normal  FailedBinding  3m (x952 over 3h)  persistentvolume-controller  no persistent volumes available for this claim and no storage class is set

Version-Release number of selected component (if applicable):
openshift-ansible-3.10.21-1.git.0.6446011.el7.noarch

How reproducible:
always

Steps to Reproduce:
1. Setup OCS in independent mode for logging & metrics (refer attached inventory file)


Actual results:
logging & metrics configuration fails as PVCs don't get created

Expected results:
logging & metrics configuration should succeed

Additional info:
inventory & log files shall be attached

Comment 2 krishnaram Karthick 2018-08-24 15:27:47 UTC
Created attachment 1478578 [details]
ansible logs

Comment 3 krishnaram Karthick 2018-08-24 15:29:38 UTC
Created attachment 1478579 [details]
inventory file

Comment 8 Mugdha Soni 2018-08-27 18:45:59 UTC
(In reply to Jose A.Rivera from comment #5)

Jose added the parameters suggested by you in the inventory file but still PVC'S are in pending state.Though i see a storage class being assigned to these PVC's which was not assigned previously.

[root@dhcp46-111 ~]# oc describe pvc/metrics-cassandra-1 -n openshift-infra
Name:          metrics-cassandra-1
Namespace:     openshift-infra
StorageClass:  glusterfs-registry-block
Status:        Pending
Volume:        
Labels:        metrics-infra=hawkular-cassandra
Annotations:   control-plane.alpha.kubernetes.io/leader={"holderIdentity":"da248499-aa24-11e8-95ca-0a580a820005","leaseDurationSeconds":15,"acquireTime":"2018-08-27T18:17:44Z","renewTime":"2018-08-27T18:38:41Z","lea...
               kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"v1","kind":"PersistentVolumeClaim","metadata":{"annotations":{},"labels":{"metrics-infra":"hawkular-cassandra"},"name":"metrics-cassandr...
               volume.beta.kubernetes.io/storage-provisioner=gluster.org/glusterblock
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:      
Access Modes:  
Events:
  Type     Reason                Age                 From                                                           Message
  ----     ------                ----                ----                                                           -------
  Warning  ProvisioningFailed    5m (x14 over 24m)   gluster.org/glusterblock da248499-aa24-11e8-95ca-0a580a820005  Failed to provision volume with StorageClass "glusterfs-registry-block": failed to create volume: [heketi] failed to create volume:
  Normal   ExternalProvisioning  4m (x353 over 24m)  persistentvolume-controller                                    waiting for a volume to be created, either by external provisioner "gluster.org/glusterblock" or manually created by system administrator
  Normal   Provisioning          3m (x15 over 24m)   gluster.org/glusterblock da248499-aa24-11e8-95ca-0a580a820005  External provisioner is provisioning volume for claim "openshift-infra/metrics-cassandra-1"


[root@dhcp46-111 ~]# oc describe pvc/logging-es-0 -n openshift-logging
Name:          logging-es-0
Namespace:     openshift-logging
StorageClass:  glusterfs-registry-block
Status:        Pending
Volume:        
Labels:        logging-infra=support
Annotations:   control-plane.alpha.kubernetes.io/leader={"holderIdentity":"da248499-aa24-11e8-95ca-0a580a820005","leaseDurationSeconds":15,"acquireTime":"2018-08-27T18:19:45Z","renewTime":"2018-08-27T18:40:59Z","lea...
               volume.beta.kubernetes.io/storage-provisioner=gluster.org/glusterblock
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:      
Access Modes:  
Events:
  Type     Reason                Age                 From                                                           Message
  ----     ------                ----                ----                                                           -------
  Warning  ProvisioningFailed    5m (x14 over 23m)   gluster.org/glusterblock da248499-aa24-11e8-95ca-0a580a820005  Failed to provision volume with StorageClass "glusterfs-registry-block": failed to create volume: [heketi] failed to create volume:
  Normal   ExternalProvisioning  3m (x350 over 23m)  persistentvolume-controller                                    waiting for a volume to be created, either by external provisioner "gluster.org/glusterblock" or manually created by system administrator
  Normal   Provisioning          2m (x15 over 23m)   gluster.org/glusterblock da248499-aa24-11e8-95ca-0a580a820005  External provisioner is provisioning volume for claim "openshift-logging/logging-es-0"

Comment 9 Raghavendra Talur 2018-08-29 14:00:19 UTC
Please attach the heketi logs.

Comment 20 RamaKasturi 2019-03-20 11:40:56 UTC
Hello Ashmitha,

    Can you please update the bug with the latest results ?

placing need info on ashmitha to provide the details and removing it from krk.

Thanks
kasturi