Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1765808

Summary:	elasticsearch-proxy clusterrolebinding has been overwritten as istio-system elasticsearch even though openshif-logging elasticsearch is configured that.
Product:	OpenShift Container Platform	Reporter:	Pavol Loffay <ploffay>
Component:	Logging	Assignee:	Pavol Loffay <ploffay>
Status:	CLOSED ERRATA	QA Contact:	Anping Li <anli>
Severity:	urgent	Docs Contact:
Priority:	unspecified
Version:	4.2.z	CC:	aos-bugs, rmeggins
Target Milestone:	---
Target Release:	4.2.z
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2019-12-03 22:43:11 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:	1765803
Bug Blocks:	1765807

Description Pavol Loffay 2019-10-26 07:53:27 UTC

This is a copy of https://bugzilla.redhat.com/show_bug.cgi?id=1765803 for OCP 4.2

Copied from https://issues.jboss.org/browse/OSSM-100

If openshift-logging is running on the OCP4.1, Jager which is configured by "template: production-elasticsearch" would replace namespace  "openshift-logging" with "istio-system" in elasticsearch-proxy ClusterRoleBinding subjects section.

{code:yaml}
  apiVersion: maistra.io/v1
  kind: ServiceMeshControlPlane
  spec:
    istio:
      tracing:
        jaeger:
          template: production-elasticsearch
          elasticsearch:
            nodeCount: 3
            redundancyPolicy:
            resources:
              requests:
                memory: "16Gi"
                cpu: "1"
              limits:
                memory: "16Gi"
{code}

Before installing Service Mesh,
{code:yaml}
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  creationTimestamp: "2019-10-07T05:49:31Z"
  name: elasticsearch-proxy
  ownerReferences:
  - apiVersion: logging.openshift.io/v1
    controller: true
    kind: Elasticsearch
    name: elasticsearch
...
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: elasticsearch-proxy
subjects:
- kind: ServiceAccount
  name: elasticsearch
  namespace: openshift-logging
{code}

After Installing Jager with elasticsearch
{code:yaml}
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  creationTimestamp: "2019-10-07T05:49:31Z"
  name: elasticsearch-proxy
  ownerReferences:
  - apiVersion: logging.openshift.io/v1
    controller: true
    kind: Elasticsearch
    name: elasticsearch
...
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: elasticsearch-proxy
subjects:
- kind: ServiceAccount
  name: elasticsearch
  namespace: istio-system
{code}

This issue is a root cause of following elasticsearch error in openshift-logging.
{code:shell}
2019/10/04 12:35:22 oauthproxy.go:782: 10.0.1.11:36836 tokenreviews.authentication.k8s.io is forbidden: User "system:serviceaccount:openshift-logging:elasticsearch" cannot create resource "tokenreviews" in API group "authentication.k8s.io" at the cluster scope
{code}

Comment 1 Anping Li 2019-11-12 10:51:38 UTC

The jaeger-operator can be enabled in 4.2 using registry.redhat.io/distributed-tracing/jaeger-rhel7-operator:1.13.1. But the collector pod couldn't be started.  How to setup elasicsearch cluster for jaeger? I had expected the jaeger-operator create custom resource elasticsearch for me. Could you give the detail step to deploy jaeger using elasticsearch?


[1] oc get pods -n openshift-operators
NAME                                      READY   STATUS    RESTARTS   AGE
elasticsearch-operator-5d4b85bcf8-7z9rb   1/1     Running   0          11m
jaeger-operator-98dd965f5-xvdgr           1/1     Running   0          7m55s

[2]$oc get pods
NAME                                     READY   STATUS             RESTARTS   AGE
simple-prod-collector-7cbf55cd48-n95cw   0/1     CrashLoopBackOff   5          4m18s
simple-prod-query-684bcd4777-p6cx8       2/3     Error              5          4m18s

[anli@preserve-docker-slave 42]$ oc logs simple-prod-collector-7cbf55cd48-n95cw
2019/11/12 10:46:08 maxprocs: Leaving GOMAXPROCS=4: CPU quota undefined
{"level":"info","ts":1573555568.3882132,"caller":"flags/service.go:115","msg":"Mounting metrics handler on admin server","route":"/metrics"}
{"level":"info","ts":1573555568.3884256,"caller":"flags/admin.go:108","msg":"Mounting health check on admin server","route":"/"}
{"level":"info","ts":1573555568.3884945,"caller":"flags/admin.go:114","msg":"Starting admin HTTP server","http-port":14269}
{"level":"info","ts":1573555568.388518,"caller":"flags/admin.go:100","msg":"Admin server started","http-port":14269,"health-status":"unavailable"}
{"level":"fatal","ts":1573555574.030478,"caller":"collector/main.go:89","msg":"Failed to init storage factory","error":"failed to create primary Elasticsearch client: health check timeout: Head http://elasticsearch.default.svc:9200: dial tcp: lookup elasticsearch.default.svc on 172.30.0.10:53: no such host: no Elasticsearch node available","errorVerbose":"no Elasticsearch node available\ngithub.com/jaegertracing/jaeger/vendor/gopkg.in/olivere/elastic%2ev5.init\n\t/builddir/build/BUILD/jaeger-v1.13.1.redhat5/src/github.com/jaegertracing/jaeger/vendor/gopkg.in/olivere/elastic.v5/client.go:88\ngithub.com/jaegertracing/jaeger/pkg/es.init\n\t<autogenerated>:1\ngithub.com/jaegertracing/jaeger/plugin/storage/es.init\n\t<autogenerated>:1\ngithub.com/jaegertracing/jaeger/plugin/storage.init\n\t<autogenerated>:1\ngithub.com/jaegertracing/jaeger/cmd/env.init\n\t<autogenerated>:1\nmain.init\n\t<autogenerated>:1\nruntime.main\n\t/opt/rh/go-toolset-1.11/root/usr/lib/go-toolset-1.11-golang/src/runtime/proc.go:189\nruntime.goexit\n\t/opt/rh/go-toolset-1.11/root/usr/lib/go-toolset-1.11-golang/src/runtime/asm_amd64.s:1333\nhealth check timeout: Head http://elasticsearch.default.svc:9200: dial tcp: lookup elasticsearch.default.svc on 172.30.0.10:53: no such host\ngithub.com/jaegertracing/jaeger/vendor/gopkg.in/olivere/elastic%2ev5.(*Client).startupHealthcheck\n\t/builddir/build/BUILD/jaeger-v1.13.1.redhat5/src/github.com/jaegertracing/jaeger/vendor/gopkg.in/olivere/elastic.v5/client.go:1116\ngithub.com/jaegertracing/jaeger/vendor/gopkg.in/olivere/elastic%2ev5.NewClient\n\t/builddir/build/BUILD/jaeger-v1.13.1.redhat5/src/github.com/jaegertracing/jaeger/vendor/gopkg.in/olivere/elastic.v5/client.go:244\ngithub.com/jaegertracing/jaeger/pkg/es/config.(*Configuration).NewClient\n\t/builddir/build/BUILD/jaeger-v1.13.1.redhat5/src/github.com/jaegertracing/jaeger/pkg/es/config/config.go:100\ngithub.com/jaegertracing/jaeger/plugin/storage/es.(*Factory).Initialize\n\t/builddir/build/BUILD/jaeger-v1.13.1.redhat5/src/github.com/jaegertracing/jaeger/plugin/storage/es/factory.go:80\ngithub.com/jaegertracing/jaeger/plugin/storage.(*Factory).Initialize\n\t/builddir/build/BUILD/jaeger-v1.13.1.redhat5/src/github.com/jaegertracing/jaeger/plugin/storage/factory.go:107\nmain.main.func1\n\t/builddir/build/BUILD/jaeger-v1.13.1.redhat5/src/github.com/jaegertracing/jaeger/cmd/collector/main.go:88\ngithub.com/jaegertracing/jaeger/vendor/github.com/spf13/cobra.(*Command).execute\n\t/builddir/build/BUILD/jaeger-v1.13.1.redhat5/src/github.com/jaegertracing/jaeger/vendor/github.com/spf13/cobra/command.go:762\ngithub.com/jaegertracing/jaeger/vendor/github.com/spf13/cobra.(*Command).ExecuteC\n\t/builddir/build/BUILD/jaeger-v1.13.1.redhat5/src/github.com/jaegertracing/jaeger/vendor/github.com/spf13/cobra/command.go:852\ngithub.com/jaegertracing/jaeger/vendor/github.com/spf13/cobra.(*Command).Execute\n\t/builddir/build/BUILD/jaeger-v1.13.1.redhat5/src/github.com/jaegertracing/jaeger/vendor/github.com/spf13/cobra/command.go:800\nmain.main\n\t/builddir/build/BUILD/jaeger-v1.13.1.redhat5/src/github.com/jaegertracing/jaeger/cmd/collector/main.go:180\nruntime.main\n\t/opt/rh/go-toolset-1.11/root/usr/lib/go-toolset-1.11-golang/src/runtime/proc.go:201\nruntime.goexit\n\t/opt/rh/go-toolset-1.11/root/usr/lib/go-toolset-1.11-golang/src/runtime/asm_amd64.s:1333\nfailed to create primary Elasticsearch client\ngithub.com/jaegertracing/jaeger/plugin/storage/es.(*Factory).Initialize\n\t/builddir/build/BUILD/jaeger-v1.13.1.redhat5/src/github.com/jaegertracing/jaeger/plugin/storage/es/factory.go:82\ngithub.com/jaegertracing/jaeger/plugin/storage.(*Factory).Initialize\n\t/builddir/build/BUILD/jaeger-v1.13.1.redhat5/src/github.com/jaegertracing/jaeger/plugin/storage/factory.go:107\nmain.main.func1\n\t/builddir/build/BUILD/jaeger-v1.13.1.redhat5/src/github.com/jaegertracing/jaeger/cmd/collector/main.go:88\ngithub.com/jaegertracing/jaeger/vendor/github.com/spf13/cobra.(*Command).execute\n\t/builddir/build/BUILD/jaeger-v1.13.1.redhat5/src/github.com/jaegertracing/jaeger/vendor/github.com/spf13/cobra/command.go:762\ngithub.com/jaegertracing/jaeger/vendor/github.com/spf13/cobra.(*Command).ExecuteC\n\t/builddir/build/BUILD/jaeger-v1.13.1.redhat5/src/github.com/jaegertracing/jaeger/vendor/github.com/spf13/cobra/command.go:852\ngithub.com/jaegertracing/jaeger/vendor/github.com/spf13/cobra.(*Command).Execute\n\t/builddir/build/BUILD/jaeger-v1.13.1.redhat5/src/github.com/jaegertracing/jaeger/vendor/github.com/spf13/cobra/command.go:800\nmain.main\n\t/builddir/build/BUILD/jaeger-v1.13.1.redhat5/src/github.com/jaegertracing/jaeger/cmd/collector/main.go:180\nruntime.main\n\t/opt/rh/go-toolset-1.11/root/usr/lib/go-toolset-1.11-golang/src/runtime/proc.go:201\nruntime.goexit\n\t/opt/rh/go-toolset-1.11/root/usr/lib/go-toolset-1.11-golang/src/runtime/asm_amd64.s:1333","stacktrace":"main.main.func1\n\t/builddir/build/BUILD/jaeger-v1.13.1.redhat5/src/github.com/jaegertracing/jaeger/cmd/collector/main.go:89\ngithub.com/jaegertracing/jaeger/vendor/github.com/spf13/cobra.(*Command).execute\n\t/builddir/build/BUILD/jaeger-v1.13.1.redhat5/src/github.com/jaegertracing/jaeger/vendor/github.com/spf13/cobra/command.go:762\ngithub.com/jaegertracing/jaeger/vendor/github.com/spf13/cobra.(*Command).ExecuteC\n\t/builddir/build/BUILD/jaeger-v1.13.1.redhat5/src/github.com/jaegertracing/jaeger/vendor/github.com/spf13/cobra/command.go:852\ngithub.com/jaegertracing/jaeger/vendor/github.com/spf13/cobra.(*Command).Execute\n\t/builddir/build/BUILD/jaeger-v1.13.1.redhat5/src/github.com/jaegertracing/jaeger/vendor/github.com/spf13/cobra/command.go:800\nmain.main\n\t/builddir/build/BUILD/jaeger-v1.13.1.redhat5/src/github.com/jaegertracing/jaeger/cmd/collector/main.go:180\nruntime.main\n\t/opt/rh/go-toolset-1.11/root/usr/lib/go-toolset-1.11-golang/src/runtime/proc.go:201"}

Comment 2 Pavol Loffay 2019-11-12 10:57:29 UTC

It should create Elasticsearch CR, you can verify that by getting all ES CRs with oc command. Did you deploy ES operator before deploying Jaeger?

Comment 3 Pavol Loffay 2019-11-20 05:56:54 UTC

Anping were you able to verify this issue?

Comment 5 Anping Li 2019-11-22 10:59:49 UTC

1. Deploy Jaeger operators in webconsole
2. deploy Jaeger in project jaeger1
oc new-project jaeger1
echo 'apiVersion: jaegertracing.io/v1
kind: Jaeger
metadata:
  name: simple-prod
spec:
  strategy: production
  storage:
    type: elasticsearch
    elasticsearch:
      nodeCount: 1
      resources:
        requests:
          cpu: 200m
          memory: 1Gi
        limits:
          memory: 1Gi
'| oc create -f -
3. deploy Jaeger in project jaeger2
  ....

4.  Fix the deployment bugs by 'oc adm policy  add-cluster-role-to-user system:auth-delegator system:serviceaccount:openshift-operators:jaeger-operator'

[anli@preserve-docker-slave install]$ oc logs jaeger-operator-54b947db5d-nh69s
time="2019-11-22T10:31:39Z" level=info msg=Versions arch=amd64 jaeger-operator=v1.13.1.redhat8 operator-sdk=v0.8.1 os=linux version=go1.11.5
time="2019-11-22T10:31:40Z" level=info msg="Auto-detected the platform" platform=openshift
time="2019-11-22T10:31:40Z" level=info msg="Automatically adjusted the 'es-provision' flag" es-provision=true
time="2019-11-22T10:31:40Z" level=info msg="The service account running this operator does not have the role 'system:auth-delegator', consider granting it for additional capabilities"
time="2019-11-22T10:39:58Z" level=error msg="failed to apply the changes" error="elasticsearch cluster didn't get to ready state: timed out waiting for the condition" execution="2019-11-22 10:37:57.027400968 +0000 UTC" instance=simple-prod namespace=jaeger1
time="2019-11-22T10:42:00Z" level=error msg="failed to apply the changes" error="elasticsearch cluster didn't get to ready state: timed out waiting for the condition" execution="2019-11-22 10:39:59.998064834 +0000 UTC" instance=simple-prod namespace=jaeger2

5. Check the subjects of  ClusterRoleBinding/elasticsearch-proxy 
[anli@preserve-docker-slave install]$ oc get ClusterRoleBinding  elasticsearch-proxy -o json |jq '.subjects'
[
  {
    "kind": "ServiceAccount",
    "name": "elasticsearch",
    "namespace": "jaeger1"
  },
  {
    "kind": "ServiceAccount",
    "name": "elasticsearch",
    "namespace": "jaeger2"
  },
  {
    "kind": "ServiceAccount",
    "name": "elasticsearch",
    "namespace": "openshift-logging"
  }
]

Comment 7 errata-xmlrpc 2019-12-03 22:43:11 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:3953