Bug 1664941 - Deploy logging failed at "spec.logStore.elasticsearch.replicas in body is required"
Summary: Deploy logging failed at "spec.logStore.elasticsearch.replicas in body is req...
Keywords:
Status: CLOSED NOTABUG
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Logging
Version: 4.1.0
Hardware: Unspecified
OS: Unspecified
unspecified
high
Target Milestone: ---
: 4.1.0
Assignee: Jeff Cantrill
QA Contact: Anping Li
URL:
Whiteboard:
Depends On: 1666944
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-01-10 06:13 UTC by Qiaoling Tang
Modified: 2019-03-12 14:27 UTC (History)
2 users (show)

Fixed In Version:
Doc Type: No Doc Update
Doc Text:
undefined
Clone Of:
Environment:
Last Closed: 2019-01-25 14:49:36 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github operator-framework operator-lifecycle-manager pull 655 0 None None None 2019-01-10 14:37:31 UTC

Description Qiaoling Tang 2019-01-10 06:13:28 UTC
Description of problem:
Deploy logging via OLM, got an error when creating CR using https://raw.githubusercontent.com/openshift/cluster-logging-operator/master/hack/cr.yaml :
$ oc create -f hack/cr.yaml 
The ClusterLogging "example" is invalid: []: Invalid value: map[string]interface {}{"apiVersion":"logging.openshift.io/v1alpha1", "kind":"ClusterLogging", "metadata":map[string]interface {}{"uid":"3e0b0b1b-149b-11e9-b607-069bc7021f3c", "selfLink":"", "clusterName":"", "name":"example", "namespace":"openshift-logging", "creationTimestamp":"2019-01-10T05:47:40Z", "generation":1}, "spec":map[string]interface {}{"collection":map[string]interface {}{"logs":map[string]interface {}{"fluentd":map[string]interface {}{}, "type":"fluentd"}}, "curation":map[string]interface {}{"curator":map[string]interface {}{"schedule":"30 3 * * *"}, "type":"curator"}, "logStore":map[string]interface {}{"elasticsearch":map[string]interface {}{"nodeCount":1, "redundancyPolicy":"SingleRedundancy", "storage":map[string]interface {}{}}, "type":"elasticsearch"}, "managementState":"Managed", "visualization":map[string]interface {}{"kibana":map[string]interface {}{"replicas":1}, "type":"kibana"}}}: validation failure list:
spec.logStore.elasticsearch.replicas in body is required

Then add "spec.logStore.elasticsearch.replicas: 1" in cr.yaml, and create it again, the CRs can be created, and ES pod could start, but no resources for curator, kibana and fluentd:
$ oc get pod
NAME                                                  READY     STATUS    RESTARTS   AGE
cluster-logging-operator-7c4c4d85b-67t6l              1/1       Running   0          10m
elasticsearch-clientdatamaster-0-1-84d764899d-w9cq4   1/1       Running   0          5m
elasticsearch-operator-55d66f5d8f-56csh               1/1       Running   0          10m

logs in CLO:
$ oc logs cluster-logging-operator-7c4c4d85b-67t6l
time="2019-01-10T05:46:23Z" level=info msg="Go Version: go1.10.3"
time="2019-01-10T05:46:23Z" level=info msg="Go OS/Arch: linux/amd64"
time="2019-01-10T05:46:23Z" level=info msg="operator-sdk Version: 0.0.7"
time="2019-01-10T05:46:23Z" level=info msg="Metrics service cluster-logging-operator created"
time="2019-01-10T05:46:23Z" level=info msg="Watching logging.openshift.io/v1alpha1, ClusterLogging, openshift-logging, 5000000000"
time="2019-01-10T05:51:24Z" level=info msg="Updating status of Elasticsearch"
time="2019-01-10T05:51:24Z" level=error msg="error syncing key (openshift-logging/qitang): Unable to create or update logstore: Failed to update Cluster Logging Elasticsearch status: ClusterLogging.logging.openshift.io \"qitang\" is invalid: []: Invalid value: map[string]interface {}{\"status\":map[string]interface {}{\"logStore\":map[string]interface {}{\"elasticsearchStatus\":[]interface {}{map[string]interface {}{\"nodeCount\":1, \"pods\":map[string]interface {}{\"client\":map[string]interface {}{\"failed\":interface {}(nil), \"notReady\":interface {}(nil), \"ready\":interface {}(nil)}, \"data\":map[string]interface {}{\"failed\":interface {}(nil), \"notReady\":interface {}(nil), \"ready\":interface {}(nil)}, \"master\":map[string]interface {}{\"notReady\":interface {}(nil), \"ready\":interface {}(nil), \"failed\":interface {}(nil)}}, \"replicaSets\":[]interface {}{}, \"statefulSets\":[]interface {}{}, \"clusterHealth\":\"\", \"clusterName\":\"elasticsearch\", \"deployments\":[]interface {}{}}}}, \"visualization\":map[string]interface {}{}, \"collection\":map[string]interface {}{\"logs\":map[string]interface {}{\"fluentdStatus\":map[string]interface {}{\"daemonSet\":\"\", \"nodes\":interface {}(nil), \"pods\":interface {}(nil)}, \"rsyslogStatus\":map[string]interface {}{\"daemonSet\":\"\", \"pods\":interface {}(nil), \"Nodes\":interface {}(nil)}}}, \"curation\":map[string]interface {}{}}, \"apiVersion\":\"logging.openshift.io/v1alpha1\", \"kind\":\"ClusterLogging\", \"metadata\":map[string]interface {}{\"uid\":\"c29cc7d7-149b-11e9-b880-06c7e20fdb6a\", \"resourceVersion\":\"137716\", \"generation\":1, \"creationTimestamp\":\"2019-01-10T05:51:22Z\", \"clusterName\":\"\", \"name\":\"qitang\", \"namespace\":\"openshift-logging\", \"selfLink\":\"/apis/logging.openshift.io/v1alpha1/namespaces/openshift-logging/clusterloggings/qitang\"}, \"spec\":map[string]interface {}{\"curation\":map[string]interface {}{\"curator\":map[string]interface {}{\"resources\":map[string]interface {}{}, \"schedule\":\"30 3 * * *\"}, \"type\":\"curator\"}, \"logStore\":map[string]interface {}{\"elasticsearch\":map[string]interface {}{\"nodeCount\":1, \"redundancyPolicy\":\"SingleRedundancy\", \"resources\":map[string]interface {}{}, \"storage\":map[string]interface {}{}}, \"type\":\"elasticsearch\"}, \"managementState\":\"Managed\", \"visualization\":map[string]interface {}{\"kibana\":map[string]interface {}{\"proxy\":map[string]interface {}{\"resources\":map[string]interface {}{}}, \"replicas\":1, \"resources\":map[string]interface {}{}}, \"type\":\"kibana\"}, \"collection\":map[string]interface {}{\"logs\":map[string]interface {}{\"fluentd\":map[string]interface {}{\"resources\":map[string]interface {}{}}, \"rsyslog\":map[string]interface {}{\"resources\":map[string]interface {}{}}, \"type\":\"fluentd\"}}}}: validation failure list:\nspec.logStore.elasticsearch.replicas in body is required"
time="2019-01-10T05:51:28Z" level=info msg="Updating status of Elasticsearch"

$ oc get clusterloggings.logging.openshift.io -o yaml
apiVersion: v1
items:
- apiVersion: logging.openshift.io/v1alpha1
  kind: ClusterLogging
  metadata:
    creationTimestamp: 2019-01-10T05:51:22Z
    generation: 1
    name: qitang
    namespace: openshift-logging
    resourceVersion: "137716"
    selfLink: /apis/logging.openshift.io/v1alpha1/namespaces/openshift-logging/clusterloggings/qitang
    uid: c29cc7d7-149b-11e9-b880-06c7e20fdb6a
  spec:
    collection:
      logs:
        fluentd: {}
        type: fluentd
    curation:
      curator:
        schedule: 30 3 * * *
      type: curator
    logStore:
      elasticsearch:
        nodeCount: 1
        redundancyPolicy: SingleRedundancy
        replicas: 1
        storage: {}
      type: elasticsearch
    managementState: Managed
    visualization:
      kibana:
        replicas: 1
      type: kibana
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""


Version-Release number of selected component (if applicable):
$ oc get clusterversion
NAME      VERSION                           AVAILABLE   PROGRESSING   SINCE     STATUS
version   4.0.0-0.alpha-2019-01-10-004600   True        False         3h        Cluster version is 4.0.0-0.alpha-2019-01-10-004600

$ oc get pod cluster-logging-operator-7c4c4d85b-67t6l -o yaml |grep image
    image: quay.io/openshift/origin-cluster-logging-operator:latest
    imagePullPolicy: IfNotPresent
  imagePullSecrets:
    image: quay.io/openshift/origin-cluster-logging-operator:latest
    imageID: quay.io/openshift/origin-cluster-logging-operator@sha256:badfe7f5a14e83541f88f1495d4ce1828567aefc7703f1c092c7c522a34247d2

How reproducible:
Always

Steps to Reproduce:
1. Deploy logging via OLM, wait until CLO and EO pod start, create CRs manually
2.
3.

Actual results:


Expected results:


Additional info:

Comment 1 Qiaoling Tang 2019-01-10 06:16:51 UTC
$ oc get elasticsearches.logging.openshift.io -o yaml
apiVersion: v1
items:
- apiVersion: logging.openshift.io/v1alpha1
  kind: Elasticsearch
  metadata:
    creationTimestamp: 2019-01-10T05:51:23Z
    generation: 1
    name: elasticsearch
    namespace: openshift-logging
    ownerReferences:
    - apiVersion: logging.openshift.io/v1alpha1
      controller: true
      kind: ClusterLogging
      name: qitang
      uid: c29cc7d7-149b-11e9-b880-06c7e20fdb6a
    resourceVersion: "155882"
    selfLink: /apis/logging.openshift.io/v1alpha1/namespaces/openshift-logging/elasticsearches/elasticsearch
    uid: c33ab78c-149b-11e9-b91a-0282b3a38dc0
  spec:
    managementState: Managed
    nodeSpec:
      image: docker.io/openshift/origin-logging-elasticsearch5:latest
      resources: {}
    nodes:
    - nodeCount: 1
      nodeSpec:
        resources: {}
      roles:
      - client
      - data
      - master
      storage: {}
    redundancyPolicy: SingleRedundancy
  status:
    clusterHealth: yellow
    conditions:
    - lastTransitionTime: 2019-01-10T05:51:24Z
      status: "True"
      type: ScalingUp
    - lastTransitionTime: 2019-01-10T05:51:26Z
      message: Config Map is up to date
      reason: ConfigChange
      status: "False"
      type: UpdatingSettings
    - lastTransitionTime: 2019-01-10T05:51:26Z
      status: "False"
      type: ScalingDown
    - lastTransitionTime: 2019-01-10T05:51:26Z
      status: "False"
      type: Restarting
    nodes:
    - deploymentName: elasticsearch-clientdatamaster-0-1
      podName: elasticsearch-clientdatamaster-0-1-84d764899d-w9cq4
      replicaSetName: elasticsearch-clientdatamaster-0-1-84d764899d
      roles:
      - client
      - data
      - master
      status: Running
      upgradeStatus:
        underUpgrade: "False"
    pods:
      client:
        failed: []
        notReady: []
        ready:
        - elasticsearch-clientdatamaster-0-1-84d764899d-w9cq4
      data:
        failed: []
        notReady: []
        ready:
        - elasticsearch-clientdatamaster-0-1-84d764899d-w9cq4
      master:
        failed: []
        notReady: []
        ready:
        - elasticsearch-clientdatamaster-0-1-84d764899d-w9cq4
    shardAllocationEnabled: "True"
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""

Comment 2 Jeff Cantrill 2019-01-10 14:37:20 UTC
This should be resolved once https://github.com/operator-framework/operator-lifecycle-manager/pull/655 merges.  You are using whats checked into OLM which is not yet in sync with the changes made to clusterlogging

Comment 3 Anping Li 2019-01-11 07:14:01 UTC
That is not a test block. We can still deploy logging following following https://github.com/openshift/cluster-logging-operator

Comment 4 Jeff Cantrill 2019-01-15 18:43:27 UTC
Deployable as a community operator via https://github.com/operator-framework/community-operators/pull/16 which supersedes OLM.  Additionally, must use the current CR which can be spec'd via: https://github.com/openshift/cluster-logging-operator/blob/master/docs/configuration.md#configuring-and-tuning-cluster-logging

Comment 5 Jeff Cantrill 2019-01-25 14:49:36 UTC
Closing this issue as NOTABUG as the spec is not valid anylonger because of the change referenced in #c4.  Bug 1666944 encompasses the issue identified here.


Note You need to log in before you can comment on or make changes to this bug.