Bug 1685394

Summary: Setup couchbase cluster with warning msg:"The field status.conditions is invalid" via Operator hub
Product: OpenShift Container Platform Reporter: ge liu <geliu>
Component: OLMAssignee: Evan Cordell <ecordell>
OLM sub component: OLM QA Contact: ge liu <geliu>
Status: CLOSED DUPLICATE Docs Contact:
Severity: medium    
Priority: unspecified CC: ecordell, mifiedle, scolange, scuppett
Version: 4.1.0   
Target Milestone: ---   
Target Release: 4.1.0   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-05-03 18:45:32 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
The field status.members is invalid none

Description ge liu 2019-03-05 05:59:52 UTC
Description of problem:

Setup couchbase operator in Operator hub, then setup couchbase cluster, after the operator and cluster pods(3 members) in running status,
got warning msg: "The field status.conditions is invalid" in UI, check the svc, find there is not ip assgin to the svc,

NAME             TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)               AGE
cb-example       ClusterIP   None         <none>        8091/TCP,18091/TCP    29m
cb-example-srv   ClusterIP   None         <none>        11210/TCP,11207/TCP   29m

check the log and got some err msg below:

sh-4.2$ cat error.log 
[ns_server:error,2019-03-05T02:51:56.435Z,ns_1.lgproj.svc:<0.443.0>:menelaus_web:loop:143]Server error during processing: ["web request failed",
                                 {path,"/_metakv/remoteCluster/"},
                                 {method,'GET'},
                                 {type,error},
                                 {what,function_clause},
                                 {trace,
                                  [{misc,compare_secure,
                                    [undefined,
                                     "509a925a23d1435a767edf6ec1aba1ec"],
                                    [{file,"src/misc.erl"},{line,2498}]},
                                   {ns_config_auth,authenticate,3,
                                    [{file,"src/ns_config_auth.erl"},
                                     {line,110}]},
                                   {ns_config_auth,authenticate,2,
                                    [{file,"src/ns_config_auth.erl"},
                                     {line,116}]},
                                   {menelaus_auth,authenticate,1,
                                    [{file,"src/menelaus_auth.erl"},
                                     {line,296}]},
                                   {menelaus_auth,verify_rest_auth,2,
                                    [{file,"src/menelaus_auth.erl"},
                                     {line,427}]},
                                   {menelaus_web,perform_action,2,
                                    [{file,"src/menelaus_web.erl"},
                                     {line,905}]},
                                   {request_throttler,do_request,3,
                                    [{file,"src/request_throttler.erl"},
                                     {line,59}]},
                                   {menelaus_web,loop,2,
                                    [{file,"src/menelaus_web.erl"},
                                     {line,121}]}]}]
[ns_server:error,2019-03-05T02:51:56.937Z,ns_1.lgproj.svc:<0.712.0>:menelaus_web:loop:143]Server error during processing: ["web request failed",


How reproducible:
Always

Steps to Reproduce:
1. login web ui, and go to 'Operator hub', create couchbase operator,it succeed and pod in running status
create the secret at first:
secret.yaml:

apiVersion: v1
kind: Secret
metadata:
  name: cb-example-auth
type: Opaque
data:
  username: QWRtaW5pc3RyYXRvcg==
  password: cGFzc3dvcmQ=
                 
2. create couchbase cluster,
# oc get pods
NAME                                  READY   STATUS    RESTARTS   AGE
cb-example-0000                       1/1     Running   0          10m
cb-example-0001                       1/1     Running   0          8m12s
cb-example-0002                       1/1     Running   0          5m46s
couchbase-operator-6dc8d876c5-rzfnz   1/1     Running   0          26m

3. got warning msg: "The field status.conditions is invalid" in UI

4. check the logs:

# oc logs couchbase-operator-6dc8d876c5-rzfnz
time="2019-03-05T02:34:11Z" level=info msg="couchbase-operator v1.1.0 (release)" module=main
time="2019-03-05T02:34:11Z" level=info msg="Obtaining resource lock" module=main
time="2019-03-05T02:34:11Z" level=info msg="Starting event recorder" module=main
time="2019-03-05T02:34:11Z" level=info msg="Attempting to be elected the couchbase-operator leader" module=main
time="2019-03-05T02:34:11Z" level=info msg="I'm the leader, attempt to start the operator" module=main
time="2019-03-05T02:34:11Z" level=info msg="Creating the couchbase-operator controller" module=main
time="2019-03-05T02:34:11Z" level=info msg="Event(v1.ObjectReference{Kind:\"Endpoints\", Namespace:\"lgproj\", Name:\"couchbase-operator\", UID:\"291ebbc6-3eef-11e9-829d-0639e3df255c\", APIVersion:\"v1\", ResourceVersion:\"28168\", FieldPath:\"\"}): type: 'Normal' reason: 'LeaderElection' couchbase-operator-6dc8d876c5-rzfnz became leader" module=event_recorder
time="2019-03-05T02:34:12Z" level=info msg="starting couchbaseclusters controller"
time="2019-03-05T02:34:43Z" level=info msg="Watching new cluster" cluster-name=cb-example module=cluster
time="2019-03-05T02:34:43Z" level=info msg="Janitor process starting" cluster-name=cb-example module=cluster
time="2019-03-05T02:34:43Z" level=error msg="Cluster setup failed: secrets \"cb-example-auth\" not found" cluster-name=cb-example module=cluster
time="2019-03-05T02:34:43Z" level=warning msg="Fail to handle event: ignore failed cluster (cb-example). Please delete its CR"
time="2019-03-05T02:49:14Z" level=info msg="Object (lgproj/cb-example) is deleted"
time="2019-03-05T02:49:44Z" level=info msg="Watching new cluster" cluster-name=cb-example module=cluster
time="2019-03-05T02:49:44Z" level=info msg="Janitor process starting" cluster-name=cb-example module=cluster
time="2019-03-05T02:49:44Z" level=info msg="Setting up client for operator communication with the cluster" cluster-name=cb-example module=cluster
time="2019-03-05T02:49:44Z" level=info msg="Cluster does not exist so the operator is attempting to create it" cluster-name=cb-example module=cluster
time="2019-03-05T02:49:44Z" level=info msg="Creating headless service for data nodes" cluster-name=cb-example module=cluster
time="2019-03-05T02:49:44Z" level=info msg="Creating a pod (cb-example-0000) running Couchbase 5.5.3-3" cluster-name=cb-example module=cluster
time="2019-03-05T02:51:57Z" level=info msg="Operator added member (cb-example-0000) to manage" cluster-name=cb-example module=cluster
time="2019-03-05T02:51:57Z" level=info msg="Initializing the first node in the cluster" cluster-name=cb-example module=cluster
time="2019-03-05T02:51:57Z" level=info msg="start running..." cluster-name=cb-example module=cluster
time="2019-03-05T02:52:05Z" level=info msg="server config all_services: cb-example-0000" cluster-name=cb-example module=cluster
time="2019-03-05T02:52:05Z" level=info msg="Cluster status: balanced" cluster-name=cb-example module=cluster
time="2019-03-05T02:52:05Z" level=info msg="Node status:" cluster-name=cb-example module=cluster
time="2019-03-05T02:52:05Z" level=info msg="┌─────────────────┬──────────────┬────────────────┐" cluster-name=cb-example module=cluster
time="2019-03-05T02:52:05Z" level=info msg="│ Server          │ Class        │ Status         │" cluster-name=cb-example module=cluster
time="2019-03-05T02:52:05Z" level=info msg="├─────────────────┼──────────────┼────────────────┤" cluster-name=cb-example module=cluster
time="2019-03-05T02:52:05Z" level=info msg="│ cb-example-0000 │ all_services │ managed+active │" cluster-name=cb-example module=cluster
time="2019-03-05T02:52:05Z" level=info msg="└─────────────────┴──────────────┴────────────────┘" cluster-name=cb-example module=cluster
time="2019-03-05T02:52:05Z" level=info cluster-name=cb-example module=cluster
time="2019-03-05T02:52:09Z" level=info msg="Creating a pod (cb-example-0001) running Couchbase 5.5.3-3" cluster-name=cb-example module=cluster
time="2019-03-05T02:54:35Z" level=info msg="added member (cb-example-0001)" cluster-name=cb-example module=cluster
time="2019-03-05T02:54:35Z" level=info msg="Creating a pod (cb-example-0002) running Couchbase 5.5.3-3" cluster-name=cb-example module=cluster
time="2019-03-05T02:57:00Z" level=info msg="added member (cb-example-0002)" cluster-name=cb-example module=cluster
time="2019-03-05T02:57:01Z" level=info msg="Rebalance progress: 0.000000" cluster-name=cb-example module=cluster
time="2019-03-05T02:57:05Z" level=info msg="Rebalance progress: 40.000000" cluster-name=cb-example module=cluster
time="2019-03-05T02:57:09Z" level=info msg="Rebalance progress: 40.000000" cluster-name=cb-example module=cluster
time="2019-03-05T02:57:13Z" level=info msg="Rebalance progress: 40.000000" cluster-name=cb-example module=cluster
time="2019-03-05T02:57:17Z" level=info msg="Rebalance progress: 40.000000" cluster-name=cb-example module=cluster
time="2019-03-05T02:57:21Z" level=info msg="Rebalance progress: 40.000000" cluster-name=cb-example module=cluster
time="2019-03-05T02:57:29Z" level=info msg="reconcile finished" cluster-name=cb-example module=cluster
time="2019-03-05T02:57:39Z" level=info msg="Created bucket default" cluster-name=cb-example module=cluster

$ cat error.log 
[ns_server:error,2019-03-05T02:51:56.435Z,ns_1.lgproj.svc:<0.443.0>:menelaus_web:loop:143]Server error during processing: ["web request failed",
                                 {path,"/_metakv/remoteCluster/"},
                                 {method,'GET'},

                                 .................... look at it in description above

Actual results:
As description
Expected results:
there is not warning and err msg in both ui and logs

Comment 5 Salvatore Colangelo 2019-05-03 17:15:58 UTC
Created attachment 1562617 [details]
The field status.members is invalid

Comment 6 Salvatore Colangelo 2019-05-03 17:24:11 UTC
Hi , 

  I tried to test and same error , the error in attached.

Below the version OCP

[scolange@scolange .kube]$ oc get clusterversion
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.1.0-0.nightly-2019-05-03-002055   True        False         10h     Cluster version is 4.1.0-0.nightly-2019-05-03-002055


Step 

1. Create Couchbase subscrption

2. Create Secret

apiVersion: v1
kind: Secret
metadata:
  name: cb-example-auth
type: Opaque
data:
  username: QWRtaW5pc3RyYXRvcg==
  password: cGFzc3dvcmQ=

3. Create Cluster Couchbase

apiVersion: couchbase.com/v1
kind: CouchbaseCluster
metadata:
  name: cb-example
  namespace: openshift-operators
spec:
  authSecret: cb-example-auth
  baseImage: registry.connect.redhat.com/couchbase/server
  buckets:
    - conflictResolution: seqno
      enableFlush: true
      evictionPolicy: fullEviction
      ioPriority: high
      memoryQuota: 128
      name: default
      replicas: 1
      type: couchbase
  cluster:
    analyticsServiceMemoryQuota: 1024
    autoFailoverMaxCount: 3
    autoFailoverOnDataDiskIssues: true
    autoFailoverOnDataDiskIssuesTimePeriod: 120
    autoFailoverServerGroup: false
    autoFailoverTimeout: 120
    clusterName: cb-example
    dataServiceMemoryQuota: 256
    eventingServiceMemoryQuota: 256
    indexServiceMemoryQuota: 256
    indexStorageSetting: memory_optimized
    searchServiceMemoryQuota: 256
  servers:
    - name: all_services
      services:
        - data
        - index
        - query
        - search
        - eventing
        - analytics
      size: 3
  version: 5.5.2-1

Cluster Crated with failed status

Detail from UI:
context deadline exceeded: Pod Creation error - error creating pod cb-example-0000

[scolange@scolange .kube]$ oc get pods -n openshift-operators
NAME                                 READY   STATUS             RESTARTS   AGE
cb-example-0000                      0/1     ImagePullBackOff   0          4m20s
couchbase-operator-8b9c5c469-6g94d   1/1     Running            0          13m

Comment 7 Mike Fiedler 2019-05-03 17:25:34 UTC
Moving to ASSIGNED per comment 6.

Comment 8 Stephen Cuppett 2019-05-03 18:45:32 UTC

*** This bug has been marked as a duplicate of bug 1702751 ***