Bug 1919690

Summary:	Issues with Thanos Observability Install on Bare Metal OCS
Product:	Red Hat Advanced Cluster Management for Kubernetes	Reporter:	James Young <jayoung>
Component:	Core Services / Observability	Assignee:	Chunlin Yang <chuyang>
Status:	CLOSED INSUFFICIENT_DATA	QA Contact:
Severity:	unspecified	Docs Contact:
Priority:	unspecified
Version:	rhacm-2.1.z	CC:	gghezzo
Target Milestone:	---	Flags:	ming: needinfo+
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2021-02-26 19:21:24 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description James Young 2021-01-24 18:15:40 UTC

Description of problem:
I am trying to install RedHat Advanced Cluster Management on an OCP 4.6 with Openshift Container Storage 4.6 in place.  I am having trouble with the Observability portion. I am not sure what values to put in for the thanos-object-storage.yaml.

The OCS deployment was setup following the instructions  at: 
https://access.redhat.com/documentation/en-us/red_hat_openshift_container_storage/4.6/html/deploying_openshift_container_storage_using_bare_metal_infrastructure/index

Using the following provided thanos-object-storage.yaml syntax we see thanos-store-shard pods failing to run due to: 
level=info ts=2021-01-19T02:39:11.128223396Z caller=http.go:64 service=http/server component=store msg="internal server is shutting down" err="bucket store initial sync: sync block: BaseFetcher: iter bucket: The specified bucket does not exist."

apiVersion: v1
kind: Secret
metadata:
  name: thanos-object-storage
type: Opaque
stringData:
  thanos.yaml: |
    type: s3
    config:
      bucket: observability-<random-string>
      endpoint: s3.openshift-storage.svc:443
      insecure: false
      access_key: <key value>
      secret_ket: <key value>
      http_config:
        insecure_skip_verify: true


Version-Release number of selected component (if applicable): 2.1.x


How reproducible:


Steps to Reproduce:
1. 
2.
3.

Actual results:


Expected results:


Additional info:
Steps for thanos configuration on OCS do not appear to be publicly available at this time.

Comment 2 Chunlin Yang 2021-01-28 10:03:05 UTC

From the receiver and store, we can see the clear message is `stat s3 object: The specified bucket does not exist.`

Firstly, I would like the client to have a try to list buckets

If AWS S3 CLI is the application, the following command will list buckets in OCS:

AWS_ACCESS_KEY_ID=5x0dCVopWD6B0pOGnR6i
AWS_SECRET_ACCESS_KEY=bxcJi19PFOgYx5JbK6tSxRVZINscOgP7Bl75jH4m
aws --endpoint https://noobaa-mgmt-openshift-storage.apps.chefist.cp.fyre.ibm.com --no-verify-ssl s3 ls

Secondly, I found some errors in noobaa-operator-7fb8694476-cb5n4

```
./logs/openshift-storage/noobaa-core-0.log:Jan-23 18:52:43.505 [WebServer/36] [ERROR] CONSOLE:: RPC._on_request: ERROR srv bucket_api.read_bucket_sdk_info reqid 7871@wss://noobaa-mgmt.openshift-storage.svc:443(xwo3qp.zzzzm) connid ws://[::ffff:10.254.12.21]:39616/(14v0h84c) Error: No such bucket: obc-open-cluster-management-observability-observability-thanos
./logs/openshift-storage/noobaa-core-0.log:Jan-23 18:52:58.033 [WebServer/36] [ERROR] CONSOLE:: RPC._on_request: ERROR srv bucket_api.read_bucket_sdk_info reqid 7875@wss://noobaa-mgmt.openshift-storage.svc:443(xwo3qp.zzzzm) connid ws://[::ffff:10.254.12.21]:39616/(14v0h84c) Error: No such bucket: obc-open-cluster-management-observability-observability-thanos
./logs/openshift-storage/noobaa-core-0.log:Jan-23 18:52:58.084 [WebServer/36] [ERROR] CONSOLE:: RPC._on_request: ERROR srv bucket_api.read_bucket_sdk_info reqid 7876@wss://noobaa-mgmt.openshift-storage.svc:443(xwo3qp.zzzzm) connid ws://[::ffff:10.254.12.21]:39616/(14v0h84c) Error: No such bucket: obc-open-cluster-management-observability-observability-thanos
./logs/openshift-storage/noobaa-core-0.log:Jan-23 18:52:59.588 [WebServer/36] [ERROR] CONSOLE:: RPC._on_request: ERROR srv bucket_api.read_bucket_sdk_info reqid 7877@wss://noobaa-mgmt.openshift-storage.svc:443(xwo3qp.zzzzm) connid ws://[::ffff:10.254.12.21]:39616/(14v0h84c) Error: No such bucket: obc-open-cluster-management-observability-observability-thanos
```

Comment 3 Chunlin Yang 2021-01-28 10:05:16 UTC

BTW, if `observability-deadbeef` exists, may have a try with the endpoint `noobaa-mgmt-openshift-storage.apps.chefist.cp.fyre.ibm.com`

Comment 5 Chunlin Yang 2021-02-18 09:28:17 UTC

@jayoung Can we close this issue?

Comment 6 Red Hat Bugzilla 2023-09-15 00:59:05 UTC

The needinfo request[s] on this closed bug have been removed as they have been unresolved for 500 days