Bug 2037891
| Summary: | 403 Forbidden error shows for all the graphs in each grafana dashboard after upgrade from 4.9 to 4.10 | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | Junqi Zhao <juzhao> | ||||||||
| Component: | Monitoring | Assignee: | Prashant Balachandran <pnair> | ||||||||
| Status: | CLOSED ERRATA | QA Contact: | Junqi Zhao <juzhao> | ||||||||
| Severity: | high | Docs Contact: | |||||||||
| Priority: | high | ||||||||||
| Version: | 4.10 | CC: | alisauxbpn53, amuller, anpicker, aos-bugs, dslavens, erooth, evelynmurphy886, farid.jamili4118, fornoairfryer, jfajersk, jhusta, jwakely, kenna178015crook, markspencer943, mellisajow, mjulie, noweye1216, oppa7845, robertjace512, sherronmira, snadywindyam, spasquie, stefficosenza0864, stellajonnes675, totogorae1, tracyberge69, Weelmedia, wh7094079 | ||||||||
| Target Milestone: | --- | Keywords: | Regression | ||||||||
| Target Release: | 4.10.0 | Flags: | juzhao:
needinfo-
juzhao: needinfo- juzhao: needinfo- |
||||||||
| Hardware: | Unspecified | ||||||||||
| OS: | Unspecified | ||||||||||
| Whiteboard: | |||||||||||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |||||||||
| Doc Text: | Story Points: | --- | |||||||||
| Clone Of: | Environment: | ||||||||||
| Last Closed: | 2022-03-10 16:37:33 UTC | Type: | Bug | ||||||||
| Regression: | --- | Mount Type: | --- | ||||||||
| Documentation: | --- | CRM: | |||||||||
| Verified Versions: | Category: | --- | |||||||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||||||
| Embargoed: | |||||||||||
| Attachments: |
|
||||||||||
|
Description
Junqi Zhao
2022-01-06 18:45:28 UTC
Marked as blocker+ since it's a regression and the Grafana service isn't available anymore. @Junqi have you tried to refresh the page. It might be because the grafana pod has restarted during the upgrade and has lost its local data? Template variable service failed <!DOCTYPE html> <html lang="en" charset="utf-8"> <head> <title>Log In</title> <meta name="viewport" content="width=device-width, initial-scale=1, maximum-scale=1, user-scalable=no"> <style> @font-face { font-family: "Open Sans"; src: url(data:application/x-font-woff;charset=utf-
Based on this log, it seems like an error from the oauth proxy. I will try and reproduce the error.
(In reply to Simon Pasquier from comment #6) > @Junqi have you tried to refresh the page. It might be because the grafana > pod has restarted during the upgrade and has lost its local data? refresh doesn't help, still 403 error The error is reproducible and there are error logs in the prometheus oauth-proxy. Working on figuring out the root cause. upgrade from 4.9.13 to 4.10.0-0.nightly-2022-01-11-014938, after upgrade, grafana dashboards can show data for the graphs This problem is reproduced on Power platform with build: https://mirror.openshift.com/pub/openshift-v4/ppc64le/clients/ocp-dev-preview/4.10.0-fc.1/, on upgrading from OCP 4.9.15. garafana UI does not show any data, shows "Forbidden". grafana pod did NOT restart root@zsnxt-2760-bastion-0 ~]# oc get pods -n openshift-monitoring NAME READY STATUS RESTARTS AGE alertmanager-main-0 6/6 Running 0 4h51m alertmanager-main-1 6/6 Running 0 4h44m cluster-monitoring-operator-96d8ffc66-dn28x 2/2 Running 0 4h31m grafana-85896bbc5d-cc7mz 3/3 Running 0 4h44m kube-state-metrics-84f498c4d5-gf54c 3/3 Running 1 (4h44m ago) 4h44m node-exporter-8zrzz 2/2 Running 2 5h12m node-exporter-b6lvf 2/2 Running 2 5h13m node-exporter-kvxw5 2/2 Running 2 5h12m node-exporter-spmgq 2/2 Running 2 5h11m node-exporter-x5wxz 2/2 Running 2 5h12m openshift-state-metrics-58d99989b4-bl28b 3/3 Running 0 4h44m prometheus-adapter-f8848d5cc-m9fjp 1/1 Running 0 30m prometheus-adapter-f8848d5cc-v6qv2 1/1 Running 0 30m prometheus-k8s-0 6/6 Running 0 4h51m prometheus-k8s-1 6/6 Running 0 4h44m prometheus-operator-7c7dc7d876-rftgl 2/2 Running 1 (4h29m ago) 4h31m telemeter-client-7bd665c9dc-t456b 3/3 Running 0 4h44m thanos-querier-8485d999d4-b42d2 6/6 Running 0 4h44m thanos-querier-8485d999d4-m4g5j 6/6 Running 0 4h51m Hello Julie, would you be able to provide logs from the prometheus oauth-proxy? upgraded from 4.9.15 to 4.10.0-fc.1, no error for grafana, I suggest we close this bug and open one new bug for ppcle64 cluster
# oc get clusterversion version -oyaml
...
history:
- completionTime: "2022-01-18T08:04:33Z"
image: registry.ci.openshift.org/ocp/release@sha256:9f3ac86ba907abba3ffbae580433218eef3f1934c3353caf331587ac7c450ff0
startedTime: "2022-01-18T07:02:38Z"
state: Completed
verified: true
version: 4.10.0-fc.1
- completionTime: "2022-01-18T06:38:53Z"
image: quay.io/openshift-release-dev/ocp-release@sha256:bb1987fb718f81fb30bec4e0e1cd5772945269b77006576b02546cf84c77498e
startedTime: "2022-01-18T06:20:17Z"
state: Completed
verified: false
version: 4.9.15
# oc -n openshift-monitoring logs -c grafana-proxy grafana-6857495cf4-nk4m7
2022/01/18 07:49:23 provider.go:128: Defaulting client-id to system:serviceaccount:openshift-monitoring:grafana
2022/01/18 07:49:23 provider.go:133: Defaulting client-secret to service account token /var/run/secrets/kubernetes.io/serviceaccount/token
2022/01/18 07:49:23 provider.go:351: Delegation of authentication and authorization to OpenShift is enabled for bearer tokens and client certificates.
2022/01/18 07:49:30 oauthproxy.go:203: mapping path "/" => upstream "http://localhost:3001/"
2022/01/18 07:49:30 oauthproxy.go:230: OAuthProxy configured for Client ID: system:serviceaccount:openshift-monitoring:grafana
2022/01/18 07:49:30 oauthproxy.go:240: Cookie settings: name:_oauth_proxy secure(https):true httponly:true expiry:168h0m0s domain:<default> samesite: refresh:disabled
2022/01/18 07:49:30 http.go:107: HTTPS: listening on [::]:3000
I0118 07:49:30.606074 1 dynamic_serving_content.go:130] Starting serving::/etc/tls/private/tls.crt::/etc/tls/private/tls.key
(In reply to Junqi Zhao from comment #15) > upgraded from 4.9.15 to 4.10.0-fc.1, no error for grafana upgraded from 4.9.15 to 4.10.0-fc.1 in AWS cluster, no error for grafana (In reply to Prashant Balachandran from comment #14) > Hello Julie, would you be able to provide logs from the prometheus > oauth-proxy? Logs from grafana-proxy is attached here. Created attachment 1851548 [details]
grafana-proxy-logs-on-power
Can you provide the must gather for this cluster? I tried on AWS and it is not reproducible. (In reply to Prashant Balachandran from comment #20) > Can you provide the must gather for this cluster? I tried on AWS and it is > not reproducible. We lost that cluster unfortunately. Deployed a fresh new 4.9.15 cluster on the same Power test environment, and got it upgraded to 4.10.fc1 build. Grafana dashboard is showing data, and graphs are visible now. NOT able to reproduce the issue. Anyway, I am attaching all the relevant data here (in case you want to compare the pod logs on this new cluster with that of old one). must-gather logs: https://drive.google.com/drive/folders/1L-zmdZ0Pq-GOEjO6tRiTKaR-WjaEblIJ?usp=sharing [root@varad-9826-bastion-0 e2e_tests_results]# oc version Client Version: 4.9.15 Server Version: 4.10.0-fc.1 Kubernetes Version: v1.23.0+50f645e [root@varad-9826-bastion-0 ~]# oc get pods -n openshift-monitoring NAME READY STATUS RESTARTS AGE alertmanager-main-0 6/6 Running 0 12h alertmanager-main-1 6/6 Running 0 12h cluster-monitoring-operator-96d8ffc66-p85lg 2/2 Running 0 12h grafana-d588df7db-jbmgf 3/3 Running 0 12h kube-state-metrics-84f498c4d5-vrqlf 3/3 Running 0 12h node-exporter-5b8l2 2/2 Running 2 12h node-exporter-p4lp2 2/2 Running 2 12h node-exporter-qtbhb 2/2 Running 2 12h node-exporter-txzm6 2/2 Running 2 12h node-exporter-zmvs4 2/2 Running 2 12h openshift-state-metrics-58d99989b4-q8bjw 3/3 Running 0 12h prometheus-adapter-b5b84b88f-hjs7m 1/1 Running 0 117m prometheus-adapter-b5b84b88f-tw57t 1/1 Running 0 117m prometheus-k8s-0 6/6 Running 0 12h prometheus-k8s-1 6/6 Running 0 12h prometheus-operator-7c7dc7d876-plz9n 2/2 Running 0 12h telemeter-client-7d849bcff4-xs589 3/3 Running 0 12h thanos-querier-7bd4d5f698-6zdwt 6/6 Running 0 12h thanos-querier-7bd4d5f698-rt9qf 6/6 Running 0 12h grafana-pod-logs-on-new-cluster is attached here. Created attachment 1851829 [details]
grafana-pod-logs-on-new-cluster
the fix is in 4.10.0-0.nightly-2022-01-19-212639 and later builds, upgrade from 4.9.15 to 4.10.0-0.nightly-2022-01-19-212639, no error for grafana dashboard
also checked in a fresh 4.10.0-0.nightly-2022-01-19-212639 cluster, no error for grafana dashboard either
# oc get clusterversion -oyaml
...
history:
- completionTime: "2022-01-20T02:15:49Z"
image: registry.ci.openshift.org/ocp/release@sha256:9633ec18f1ab43dd3c02d391db0f178deb698b5e708222089d063b181eb7add4
startedTime: "2022-01-20T01:11:04Z"
state: Completed
verified: false
version: 4.10.0-0.nightly-2022-01-19-212639
- completionTime: "2022-01-20T00:56:49Z"
image: quay.io/openshift-release-dev/ocp-release@sha256:bb1987fb718f81fb30bec4e0e1cd5772945269b77006576b02546cf84c77498e
startedTime: "2022-01-20T00:31:32Z"
state: Completed
verified: false
version: 4.9.15
# oc -n openshift-monitoring get secret grafana-datasources -o jsonpath="{.data.datasources\.yaml}" | base64 -d
{
"apiVersion": 1,
"datasources": [
{
"access": "proxy",
"basicAuth": true,
"basicAuthPassword": "",
"basicAuthUser": "internal",
"editable": false,
"jsonData": {
"tlsSkipVerify": true
},
"name": "prometheus",
"orgId": 1,
"type": "prometheus",
"url": "https://prometheus-k8s.openshift-monitoring.svc:9091",
"version": 1
}
]
}
*** Bug 2043098 has been marked as a duplicate of this bug. *** Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: OpenShift Container Platform 4.10.3 security update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2022:0056 I also got the same error, it didn't show up even though I tried to reload the page https://vengeio.online This comment was flagged a spam, view the edit history to see the original text if required. People on https://geometrydashunblocked.io need to know about this. Thanks for sharing É provável que você já tenha tido o prazer de ouvir sobre as air fryer a ar antes. O eletrodoméstico já existe há bastante tempo e agora está ganhando destaque em todos os lares. É uma excelente ferramenta para fazer batatas fritas, nuggets de legumes de frango e costeletas que quase não requerem óleo. https://fornoairfryer.com/ it seems there is issue for the authentication. Version-Release number of selected component (if applicable): 4.9.13 upgrade to 4.10.0-0.nightly-2022-01-05-181126, https://mcdvoice.me/ This comment was flagged a spam, view the edit history to see the original text if required. This comment was flagged a spam, view the edit history to see the original text if required. Thanks for the update and quick reply. I'll be sure to keep an eye on this thread. https://www.mymilestonecard.top/ Created attachment 1934551 This comment was flagged a spam, view the edit history to see the original text if required. Comment on attachment 1934551 This comment was flagged a spam, view the edit history to see the original text if required. Comment on attachment 1934551 This comment was flagged a spam, view the edit history to see the original text if required. This comment was flagged a spam, view the edit history to see the original text if required. Comment on attachment 1934551 Thanks for that. https://www.telltims.net/ Comment on attachment 1851829 [details] grafana-pod-logs-on-new-cluster I have same question. https://www.telltims.net/ This comment was flagged a spam, view the edit history to see the original text if required. Sonuç olarak, Traffic Racer APK, yüksek kaliteli grafikler, akıcı oyun ve çeşitli oyun modları ve zorluklar sunan mobil cihazlar için heyecan verici bir trafik yarış oyunudur. https://trafficracer.app/ Best games free online on site https://vex7.io They must be emotionally and physically strong, and able to be unaffected by what they see, whether in the past or in the future. https://wheelspinner.tools In reality, I had no idea what was being debated in this forum, which I now know a little bit about. If you want to wear something cool, I recommend adding this nike tiffany jacket(https://www.paragonjackets.com/product/tiffany-and-co-nike-jacket/) to your inventory of newest outfits. great I have same question.Thank for your writting. https://flappy-bird.io This is a great tip particularly to those fresh to the blogosphere. <a href="https://naver.com/" target="_blank">메이저놀이터</a> I love it when people get together and share ideas. Great blog, keep it up! https://totofist.com/ Fastidious response in return of this matter with firm arguments and telling all concerning that. https://totoward.com/ Fastidious response in return of this matter with firm arguments and telling all concerning that. https://totogorae.com/ Excellent way of describing, and pleasant post to obtain information concerning my presentation topic, which i am going to present in institution of higher https://mtnamsan.com/ You really make it appear really easy together with your presentation however I find this matter to be actually something which I feel I’d by no means understand. https://totomeoktwiblog.com/ This is really interesting, You’re a very professional blogger. https://totosoda.com/ I came to this site with the introduction of a friend around me and I was very impressed when I found your writing.https://meoktwi.com/ I blog often and I truly appreciate your content. Your article has really peaked my interest. https://totonoliteo.com/ I am totally agreed with this article and i just want say that this article is very nice and very informative article.I will make sure to be reading your blog more. https://totovera.com/ hello. The content of your site will be emailed to you. It's cheap and almost all of it's better information, so I see a lot of posts like this and reach for it.https://totoghost.com/ Attractive section of content. I simply stumbled upon your web site and in accession capital to say that I acquire actually loved account your weblog posts.https://totomusa.com/ Good post. I study one thing more challenging on totally different blogs every day. It is going to all the time be stimulating to learn content material from different writers and observe a bit of one thing from their store. I prefer to make use https://www.assignmentuk.co.uk/write-my-assignment of some of the content material on my blog whether you don’t mind. Natually I provide you with a hyperlink in your web blog. Thanks for sharing. Your blog is really cool and great. I really <a href="https://lavishjackets.com/product/beth-dutton-white-poncho-coat/">Yellowstone Kelly Reilly White Poncho Cloak Coat</a> appreciate your blog. I am glad to read your blog. Thanks for sharing the nice and cool post. Keep it up. I am waiting for your next blog. Beautiful area of the website. I recently came across your website and wanted to let you know how much I've enjoyed reading through your blog posts.Season 3 Star Trek Picard Leather Jacket https://www.victoriajacket.com/product/star-trek-picard-season-3-leather-jacket/ Fans of the Bleach anime series who want to dress up like the character Zangetsu can use Bleach Zangetsu Cosplay Black Cloak Coat (https://www.texasjackets.com/product/taylor-swift-the-eras-tour-washed-blue-hoodie/) for cosplay purposes. It is a wonderful addition to any cosplay collection and is guaranteed to dazzle other series fans. The article is really well written. My writing is fluent, my rhetoric is appropriate, and my perspective is unique. I am really happy to have read such a good article today. https://totovera.com/ I wanted to extend my gratitude for the informative articles on your website. visit my website at gap pink hoodie: http://jacketoria.com/product/project-gap-pink-hoodie/ I wanted to extend my gratitude for the informative articles on your website. visit my website at gap pink hoodie: https://jacketoria.com/product/project-gap-pink-hoodie/ |