This is a copy of https://bugzilla.redhat.com/show_bug.cgi?id=1765803 for OCP 4.2 Copied from https://issues.jboss.org/browse/OSSM-100 If openshift-logging is running on the OCP4.1, Jager which is configured by "template: production-elasticsearch" would replace namespace "openshift-logging" with "istio-system" in elasticsearch-proxy ClusterRoleBinding subjects section. {code:yaml} apiVersion: maistra.io/v1 kind: ServiceMeshControlPlane spec: istio: tracing: jaeger: template: production-elasticsearch elasticsearch: nodeCount: 3 redundancyPolicy: resources: requests: memory: "16Gi" cpu: "1" limits: memory: "16Gi" {code} Before installing Service Mesh, {code:yaml} apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: creationTimestamp: "2019-10-07T05:49:31Z" name: elasticsearch-proxy ownerReferences: - apiVersion: logging.openshift.io/v1 controller: true kind: Elasticsearch name: elasticsearch ... roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: elasticsearch-proxy subjects: - kind: ServiceAccount name: elasticsearch namespace: openshift-logging {code} After Installing Jager with elasticsearch {code:yaml} apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: creationTimestamp: "2019-10-07T05:49:31Z" name: elasticsearch-proxy ownerReferences: - apiVersion: logging.openshift.io/v1 controller: true kind: Elasticsearch name: elasticsearch ... roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: elasticsearch-proxy subjects: - kind: ServiceAccount name: elasticsearch namespace: istio-system {code} This issue is a root cause of following elasticsearch error in openshift-logging. {code:shell} 2019/10/04 12:35:22 oauthproxy.go:782: 10.0.1.11:36836 tokenreviews.authentication.k8s.io is forbidden: User "system:serviceaccount:openshift-logging:elasticsearch" cannot create resource "tokenreviews" in API group "authentication.k8s.io" at the cluster scope {code}
The jaeger-operator can be enabled in 4.2 using registry.redhat.io/distributed-tracing/jaeger-rhel7-operator:1.13.1. But the collector pod couldn't be started. How to setup elasicsearch cluster for jaeger? I had expected the jaeger-operator create custom resource elasticsearch for me. Could you give the detail step to deploy jaeger using elasticsearch? [1] oc get pods -n openshift-operators NAME READY STATUS RESTARTS AGE elasticsearch-operator-5d4b85bcf8-7z9rb 1/1 Running 0 11m jaeger-operator-98dd965f5-xvdgr 1/1 Running 0 7m55s [2]$oc get pods NAME READY STATUS RESTARTS AGE simple-prod-collector-7cbf55cd48-n95cw 0/1 CrashLoopBackOff 5 4m18s simple-prod-query-684bcd4777-p6cx8 2/3 Error 5 4m18s [anli@preserve-docker-slave 42]$ oc logs simple-prod-collector-7cbf55cd48-n95cw 2019/11/12 10:46:08 maxprocs: Leaving GOMAXPROCS=4: CPU quota undefined {"level":"info","ts":1573555568.3882132,"caller":"flags/service.go:115","msg":"Mounting metrics handler on admin server","route":"/metrics"} {"level":"info","ts":1573555568.3884256,"caller":"flags/admin.go:108","msg":"Mounting health check on admin server","route":"/"} {"level":"info","ts":1573555568.3884945,"caller":"flags/admin.go:114","msg":"Starting admin HTTP server","http-port":14269} {"level":"info","ts":1573555568.388518,"caller":"flags/admin.go:100","msg":"Admin server started","http-port":14269,"health-status":"unavailable"} {"level":"fatal","ts":1573555574.030478,"caller":"collector/main.go:89","msg":"Failed to init storage factory","error":"failed to create primary Elasticsearch client: health check timeout: Head http://elasticsearch.default.svc:9200: dial tcp: lookup elasticsearch.default.svc on 172.30.0.10:53: no such host: no Elasticsearch node available","errorVerbose":"no Elasticsearch node available\ngithub.com/jaegertracing/jaeger/vendor/gopkg.in/olivere/elastic%2ev5.init\n\t/builddir/build/BUILD/jaeger-v1.13.1.redhat5/src/github.com/jaegertracing/jaeger/vendor/gopkg.in/olivere/elastic.v5/client.go:88\ngithub.com/jaegertracing/jaeger/pkg/es.init\n\t<autogenerated>:1\ngithub.com/jaegertracing/jaeger/plugin/storage/es.init\n\t<autogenerated>:1\ngithub.com/jaegertracing/jaeger/plugin/storage.init\n\t<autogenerated>:1\ngithub.com/jaegertracing/jaeger/cmd/env.init\n\t<autogenerated>:1\nmain.init\n\t<autogenerated>:1\nruntime.main\n\t/opt/rh/go-toolset-1.11/root/usr/lib/go-toolset-1.11-golang/src/runtime/proc.go:189\nruntime.goexit\n\t/opt/rh/go-toolset-1.11/root/usr/lib/go-toolset-1.11-golang/src/runtime/asm_amd64.s:1333\nhealth check timeout: Head http://elasticsearch.default.svc:9200: dial tcp: lookup elasticsearch.default.svc on 172.30.0.10:53: no such host\ngithub.com/jaegertracing/jaeger/vendor/gopkg.in/olivere/elastic%2ev5.(*Client).startupHealthcheck\n\t/builddir/build/BUILD/jaeger-v1.13.1.redhat5/src/github.com/jaegertracing/jaeger/vendor/gopkg.in/olivere/elastic.v5/client.go:1116\ngithub.com/jaegertracing/jaeger/vendor/gopkg.in/olivere/elastic%2ev5.NewClient\n\t/builddir/build/BUILD/jaeger-v1.13.1.redhat5/src/github.com/jaegertracing/jaeger/vendor/gopkg.in/olivere/elastic.v5/client.go:244\ngithub.com/jaegertracing/jaeger/pkg/es/config.(*Configuration).NewClient\n\t/builddir/build/BUILD/jaeger-v1.13.1.redhat5/src/github.com/jaegertracing/jaeger/pkg/es/config/config.go:100\ngithub.com/jaegertracing/jaeger/plugin/storage/es.(*Factory).Initialize\n\t/builddir/build/BUILD/jaeger-v1.13.1.redhat5/src/github.com/jaegertracing/jaeger/plugin/storage/es/factory.go:80\ngithub.com/jaegertracing/jaeger/plugin/storage.(*Factory).Initialize\n\t/builddir/build/BUILD/jaeger-v1.13.1.redhat5/src/github.com/jaegertracing/jaeger/plugin/storage/factory.go:107\nmain.main.func1\n\t/builddir/build/BUILD/jaeger-v1.13.1.redhat5/src/github.com/jaegertracing/jaeger/cmd/collector/main.go:88\ngithub.com/jaegertracing/jaeger/vendor/github.com/spf13/cobra.(*Command).execute\n\t/builddir/build/BUILD/jaeger-v1.13.1.redhat5/src/github.com/jaegertracing/jaeger/vendor/github.com/spf13/cobra/command.go:762\ngithub.com/jaegertracing/jaeger/vendor/github.com/spf13/cobra.(*Command).ExecuteC\n\t/builddir/build/BUILD/jaeger-v1.13.1.redhat5/src/github.com/jaegertracing/jaeger/vendor/github.com/spf13/cobra/command.go:852\ngithub.com/jaegertracing/jaeger/vendor/github.com/spf13/cobra.(*Command).Execute\n\t/builddir/build/BUILD/jaeger-v1.13.1.redhat5/src/github.com/jaegertracing/jaeger/vendor/github.com/spf13/cobra/command.go:800\nmain.main\n\t/builddir/build/BUILD/jaeger-v1.13.1.redhat5/src/github.com/jaegertracing/jaeger/cmd/collector/main.go:180\nruntime.main\n\t/opt/rh/go-toolset-1.11/root/usr/lib/go-toolset-1.11-golang/src/runtime/proc.go:201\nruntime.goexit\n\t/opt/rh/go-toolset-1.11/root/usr/lib/go-toolset-1.11-golang/src/runtime/asm_amd64.s:1333\nfailed to create primary Elasticsearch client\ngithub.com/jaegertracing/jaeger/plugin/storage/es.(*Factory).Initialize\n\t/builddir/build/BUILD/jaeger-v1.13.1.redhat5/src/github.com/jaegertracing/jaeger/plugin/storage/es/factory.go:82\ngithub.com/jaegertracing/jaeger/plugin/storage.(*Factory).Initialize\n\t/builddir/build/BUILD/jaeger-v1.13.1.redhat5/src/github.com/jaegertracing/jaeger/plugin/storage/factory.go:107\nmain.main.func1\n\t/builddir/build/BUILD/jaeger-v1.13.1.redhat5/src/github.com/jaegertracing/jaeger/cmd/collector/main.go:88\ngithub.com/jaegertracing/jaeger/vendor/github.com/spf13/cobra.(*Command).execute\n\t/builddir/build/BUILD/jaeger-v1.13.1.redhat5/src/github.com/jaegertracing/jaeger/vendor/github.com/spf13/cobra/command.go:762\ngithub.com/jaegertracing/jaeger/vendor/github.com/spf13/cobra.(*Command).ExecuteC\n\t/builddir/build/BUILD/jaeger-v1.13.1.redhat5/src/github.com/jaegertracing/jaeger/vendor/github.com/spf13/cobra/command.go:852\ngithub.com/jaegertracing/jaeger/vendor/github.com/spf13/cobra.(*Command).Execute\n\t/builddir/build/BUILD/jaeger-v1.13.1.redhat5/src/github.com/jaegertracing/jaeger/vendor/github.com/spf13/cobra/command.go:800\nmain.main\n\t/builddir/build/BUILD/jaeger-v1.13.1.redhat5/src/github.com/jaegertracing/jaeger/cmd/collector/main.go:180\nruntime.main\n\t/opt/rh/go-toolset-1.11/root/usr/lib/go-toolset-1.11-golang/src/runtime/proc.go:201\nruntime.goexit\n\t/opt/rh/go-toolset-1.11/root/usr/lib/go-toolset-1.11-golang/src/runtime/asm_amd64.s:1333","stacktrace":"main.main.func1\n\t/builddir/build/BUILD/jaeger-v1.13.1.redhat5/src/github.com/jaegertracing/jaeger/cmd/collector/main.go:89\ngithub.com/jaegertracing/jaeger/vendor/github.com/spf13/cobra.(*Command).execute\n\t/builddir/build/BUILD/jaeger-v1.13.1.redhat5/src/github.com/jaegertracing/jaeger/vendor/github.com/spf13/cobra/command.go:762\ngithub.com/jaegertracing/jaeger/vendor/github.com/spf13/cobra.(*Command).ExecuteC\n\t/builddir/build/BUILD/jaeger-v1.13.1.redhat5/src/github.com/jaegertracing/jaeger/vendor/github.com/spf13/cobra/command.go:852\ngithub.com/jaegertracing/jaeger/vendor/github.com/spf13/cobra.(*Command).Execute\n\t/builddir/build/BUILD/jaeger-v1.13.1.redhat5/src/github.com/jaegertracing/jaeger/vendor/github.com/spf13/cobra/command.go:800\nmain.main\n\t/builddir/build/BUILD/jaeger-v1.13.1.redhat5/src/github.com/jaegertracing/jaeger/cmd/collector/main.go:180\nruntime.main\n\t/opt/rh/go-toolset-1.11/root/usr/lib/go-toolset-1.11-golang/src/runtime/proc.go:201"}
It should create Elasticsearch CR, you can verify that by getting all ES CRs with oc command. Did you deploy ES operator before deploying Jaeger?
Anping were you able to verify this issue?
1. Deploy Jaeger operators in webconsole 2. deploy Jaeger in project jaeger1 oc new-project jaeger1 echo 'apiVersion: jaegertracing.io/v1 kind: Jaeger metadata: name: simple-prod spec: strategy: production storage: type: elasticsearch elasticsearch: nodeCount: 1 resources: requests: cpu: 200m memory: 1Gi limits: memory: 1Gi '| oc create -f - 3. deploy Jaeger in project jaeger2 .... 4. Fix the deployment bugs by 'oc adm policy add-cluster-role-to-user system:auth-delegator system:serviceaccount:openshift-operators:jaeger-operator' [anli@preserve-docker-slave install]$ oc logs jaeger-operator-54b947db5d-nh69s time="2019-11-22T10:31:39Z" level=info msg=Versions arch=amd64 jaeger-operator=v1.13.1.redhat8 operator-sdk=v0.8.1 os=linux version=go1.11.5 time="2019-11-22T10:31:40Z" level=info msg="Auto-detected the platform" platform=openshift time="2019-11-22T10:31:40Z" level=info msg="Automatically adjusted the 'es-provision' flag" es-provision=true time="2019-11-22T10:31:40Z" level=info msg="The service account running this operator does not have the role 'system:auth-delegator', consider granting it for additional capabilities" time="2019-11-22T10:39:58Z" level=error msg="failed to apply the changes" error="elasticsearch cluster didn't get to ready state: timed out waiting for the condition" execution="2019-11-22 10:37:57.027400968 +0000 UTC" instance=simple-prod namespace=jaeger1 time="2019-11-22T10:42:00Z" level=error msg="failed to apply the changes" error="elasticsearch cluster didn't get to ready state: timed out waiting for the condition" execution="2019-11-22 10:39:59.998064834 +0000 UTC" instance=simple-prod namespace=jaeger2 5. Check the subjects of ClusterRoleBinding/elasticsearch-proxy [anli@preserve-docker-slave install]$ oc get ClusterRoleBinding elasticsearch-proxy -o json |jq '.subjects' [ { "kind": "ServiceAccount", "name": "elasticsearch", "namespace": "jaeger1" }, { "kind": "ServiceAccount", "name": "elasticsearch", "namespace": "jaeger2" }, { "kind": "ServiceAccount", "name": "elasticsearch", "namespace": "openshift-logging" } ]
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2019:3953