Created attachment 1313944 [details] kibana UI Description of problem: Created one user project, such as testing, after a few minutes, checked logs on kibana, kibana throws out error: "Discover: [exception] The index 'project..empty-project.*' was not found. This could mean data has not yet been collected." See the attached file. After a few hours, same error still happened, and can not find user's project log entries. $ oc get project NAME DISPLAY NAME STATUS testing Active All pods were running well Command ***** oc get pod -n logging ***** result as below: NAME READY STATUS RESTARTS AGE logging-curator-1-g33xq 1/1 Running 0 5d logging-es-data-master-2hh0ejvf-1-4bd78 1/1 Running 0 5d logging-es-data-master-a2v15tbo-1-p0xcj 1/1 Running 0 5d logging-fluentd-1fnp9 1/1 Running 0 5d logging-fluentd-hq91m 1/1 Running 0 5d logging-fluentd-jt352 1/1 Running 0 5d logging-fluentd-mjbxh 1/1 Running 0 5d logging-fluentd-q449p 1/1 Running 0 5d logging-fluentd-rdv7b 1/1 Running 0 5d logging-kibana-1-dkh78 2/2 Running 24 5d logging-kibana-1-x937x 2/2 Running 24 5d logging images: logging-curator:v3.6.171 logging-elasticsearch:v3.6.171 logging-kibana:v3.6.171 logging-auth-proxy:v3.6.171 Version-Release number of selected component (if applicable): OpenShift Master:v3.6.171 (online version 3.5.1.79) Kubernetes Master:v1.6.1+5115d708d7 How reproducible: Always Steps to Reproduce: 1. Create one user project and populate logs. 2. Check log entries on kibana 3. Actual results: kibana always took user's project name as empty-project Expected results: log entries could be found on kibana Additional info:
Does your newly created project 'testing' have any pods that log anything? If yes, then could you check if the user you use to login to kibana has privileges to view the project 'testing'? Otherwise, you may simply login to kibana with any user that is cluster-admin and at least operational logs should always show up in kibana.
Yes, I have one pod and it generates logs, but our administrator can not login web UI since he does not know the username/password, he only could login command console, queried from command console, it showed all the project log indicies were not generated. for example, my project is 'testig' this time, it should contains index "project.testig.${uinique_id}.${datetime}", so the logging has problem, it can not collect logs # oc exec logging-curator-1-g33xq -n logging -- curator --host logging-es --use_ssl --certificate /etc/curator/keys/ca --client-cert /etc/curator/keys/cert --client-key /etc/curator/keys/key --loglevel ERROR show indices --all-indices .kibana .kibana.57f610b93bbc2b0fd61d01a35c3d91feb7150578 .kibana.f6e5cc1fd4ceeeecb7395f44db888f9601959dec .operations.2017.08.15 .operations.2017.08.16 .operations.2017.08.17 .operations.2017.08.18 .operations.2017.08.19 .operations.2017.08.20 .operations.2017.08.21 .searchguard.logging-es-data-master-2hh0ejvf .searchguard.logging-es-data-master-a2v15tbo $ oc get project NAME DISPLAY NAME STATUS testig Active $ oc get po NAME READY STATUS RESTARTS AGE java-mainclass-1-m7zdq 1/1 Running 0 1h $ oc logs java-mainclass-1-m7zdq Executing /deployments/bin/run ... Launching application in folder: /deployments Running java -javaagent:/opt/jolokia/jolokia.jar=config=/opt/jolokia/jolokia.properties,useSslClientAuthentication=true,extraClientCheck=true,protocol=https,caCert=/var/run/secrets/kubernetes.io/serviceaccount/ca.crt,clientPrincipal=cn=system:master-proxy -classpath .:/deployments/lib/java-mainclass-2.2.94-SNAPSHOT.jar:/deployments/lib/commons-lang3-3.4.jar io.fabric8.quickstarts.java.simple.Main I> No access restrictor found, access to all MBean is allowed Jolokia: Agent started with URL https://10.130.1.226:8778/jolokia/ Hello Fabric8! Here's your random string: lGuJV Hello Fabric8! Here's your random string: Xkq70 Hello Fabric8! Here's your random string: nkr1t Hello Fabric8! Here's your random string: vnd71 Hello Fabric8! Here's your random string: sqAPm Hello Fabric8! Here's your random string: IuhrZ Hello Fabric8! Here's your random string: quvOM Hello Fabric8! Here's your random string: 9rlLo Hello Fabric8! Here's your random string: Jr5uQ **************snipped***********************
Your previous comment leads me to think you may have created the project with system:admin account. This means that you would not be able to view logs for that project as a different user even if they were collected. In order to be able to view logs for a project you created with a different user, you need to add a certain policy to that user: #check you are logged in as a user with administrator rights, 'system:admin' for instance $ oc whoami #if not, then login as one, system:admin for example $ oc login -u system:admin #verify, you are on your 'testing' project $ oc project testing #list your users and verify that the one you use to login to kibana is there, for example 'test-user-01' $ oc list users #add 'view' policy to the user you use to login to kibana, for example 'test-user-01' $ oadm policy add-role-to-user view test-user-01 Now if you login to kibana with your user, for example 'test-user-01', you would be normally able to see the logs. But it appears, that fluentd may be collecting only ops logs for some reason. In order to view the ops logs in kibana, you need to login to kibana as a user with cluster administrator role. To add cluster-admin role to your user, you need to: #check you are logged in as a user with administrator rights, 'system:admin' for instance $ oc whoami #if not, then login as one $ oc login -u system:admin #add appropriate policy to the user $ oadm policy add-cluster-role-to-user cluster-admin test-user-01 Now if you login to kibana as test-user-01, you should see at least ops logs.
Moving to ON_QA based on comment above.
(In reply to Jan Wozniak from comment #3) > Your previous comment leads me to think you may have created the project > with system:admin account. This means that you would not be able to view > logs for that project as a different user even if they were collected. > I think I did not explain clearly, for online environment, we are ordinary users, and we can only view our project logs, we don't have cluster-admin permissions to view ops logs and other users' project logs, and we can not login as administrator. The root cause is something is wrong with logging, it can not collect project logs, so we can not check logs on Kibana. The following output was executed by our administrator, if logging works well, we would see index like "project.${project_name}.${uinique_id}.${datetime}", but we did not see such index, so we get the conclusion that the logging can not collect logs. I would change the summary later. # oc exec logging-curator-1-g33xq -n logging -- curator --host logging-es --use_ssl --certificate /etc/curator/keys/ca --client-cert /etc/curator/keys/cert --client-key /etc/curator/keys/key --loglevel ERROR show indices --all-indices .kibana .kibana.57f610b93bbc2b0fd61d01a35c3d91feb7150578 .kibana.f6e5cc1fd4ceeeecb7395f44db888f9601959dec .operations.2017.08.15 .operations.2017.08.16 .operations.2017.08.17 .operations.2017.08.18 .operations.2017.08.19 .operations.2017.08.20 .operations.2017.08.21 .searchguard.logging-es-data-master-2hh0ejvf .searchguard.logging-es-data-master-a2v15tbo
(In reply to Jan Wozniak from comment #3) > > Now if you login to kibana with your user, for example 'test-user-01', you > > would be normally able to see the logs. But it appears, that fluentd may be > > collecting only ops logs for some reason. (In reply to Junqi Zhao from comment #5) > I think I did not explain clearly, for online environment, we are ordinary > users, and we can only view our project logs, we don't have cluster-admin > permissions to view ops logs and other users' project logs, and we can not > login as administrator. > > The root cause is something is wrong with logging, it can not collect > project logs, so we can not check logs on Kibana. I never disagreed, I was trying to check if at least the ops logs are visible in kibana with adding an admin user who can login to kibana, as ops logs appear to be present in Elasticsearch. Then try to decide how to debug further. Could your administrator attach logs from fluentd pods please? Also, what docker logging driver are you using?
Is this related to or a duplicate of https://bugzilla.redhat.com/show_bug.cgi?id=1478821 Isn't the online environment using json file driver. Can you attach the fluentd logs to see if there is any information of value
(In reply to Jeff Cantrill from comment #7) > Is this related to or a duplicate of > https://bugzilla.redhat.com/show_bug.cgi?id=1478821 Isn't the online > environment using json file driver. Can you attach the fluentd logs to see > if there is any information of value Our administrator send me the fluentd pod logs today, there are a lot of warn info [warn]: emit transaction failed: error_class=Encoding::UndefinedConversionError error="\"\\x92\" from ASCII-8BIT to UTF-8" tag="system.var.log.messages" This caused logging can not collect logs, same issue with https://bugzilla.redhat.com/show_bug.cgi?id=1482002
Created attachment 1317343 [details] fluentd log
Logging Driveris json-file, this defect is also same issue with https://bugzilla.redhat.com/show_bug.cgi?id=1478821
Same issue on online-stg, log driver is also json-file. OpenShift Master:v3.6.173.0.7 (online version 3.5.1.76) Kubernetes Master: v1.6.1+5115d708d7
Was this issue actually reproduced in online-stg? The docker version in online-stg is older and should not be affected by this bug. Can you please test this in online-stg?
I downgraded logging to version v3.6.171 and it appears to be working in online-int and online-stg now.
Tested on online-int and online-stg, user project logs could be viewed on Kibana. Environment: online-int oc v3.6.173.0.7 Kubernetes v1.6.1+5115d708d7 online-stg oc v3.6.173.0.7 kubernetes v1.6.1+5115d708d7