Bug 1416956

Summary: Ceph pools created via CLI aren't properly showing in Console GUI.
Product: [Red Hat Storage] Red Hat Storage Console Reporter: Mike Hackett <mhackett>
Component: CephAssignee: Shubhendu Tripathi <shtripat>
Ceph sub component: configuration QA Contact: sds-qe-bugs
Status: CLOSED EOL Docs Contact:
Severity: high    
Priority: high CC: branto, gmeno, nthomas, sankarshan, vumrao
Version: 2   
Target Milestone: ---   
Target Release: 3   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Mike Hackett 2017-01-26 22:37:19 UTC
Description of problem:
Ceph pools created via CLI aren't properly showing in Console GUI.

Not able to reproduce in our test environment.

Review of bigfin.log for issue:

Errors seen on pool 42 "test"

{"name":"test","id":"42","size_bytes":"0","size_kb":"0","num_objects":"0","num_object_clones":"0","num_object_copies":"0","num_objects_missing_on_primary":"0","num_objects_unfound":"0","num_objects_degraded":"0","read_ops":"0","read_bytes":"0","write_ops":"0","write_bytes":"0"}],"total_objects":"980","total_used":"18347036","total_avail":"1124749820452","total_space":"1124768167488"}'}
2017-01-26T14:15:56.801Z ERROR    task.go:141 UpdateStatusList] Error updating status list for task: d3c84fec-ce2f-491d-a155-06839103d19e. error: not found
2017-01-26T14:16:04+0000 INFO     saltwrapper.py:49 saltwrapper.wrapper] args=(<salt.client.LocalClient object at 0x7fb5b80731d0>, 'bspv0313.os.amadeus.net', 'mon_remote.rbd_eventer', ['ceph']), kwargs={}
2017-01-26T14:16:05+0000 INFO     saltwrapper.py:51 saltwrapper.wrapper] rv={'bspv0313.os.amadeus.net': None}
2017-01-26T14:16:34.484Z ERROR    events.go:1491 ProcessEvent] skyring:ff781618-995d-4d30-9a6b-88e6fa49d59b - Event Handling Failed for event: calamari/ceph/pool/added. error: 404 NOT FOUND
2017-01-26T14:16:36.469Z ERROR    events.go:1491 ProcessEvent] skyring:464c9f19-1f0b-453b-a19a-d3d8c4f4f6e1 - Event Handling Failed for event: calamari/ceph/pool/added. error: 404 NOT FOUND
2017-01-26T14:17:09.386Z ERROR    events.go:1305 ceph_pool_delete_handler] skyring:c498610b-5e40-407e-82b2-60c5735b3c17-Error finding the storage: 42(id) from cluster: 5c5461a1-209f-4cdc-95bf-64e2bc4cc61e. error: not found
2017-01-26T14:17:09.386Z ERROR    events.go:1491 ProcessEvent] skyring:c498610b-5e40-407e-82b2-60c5735b3c17 - Event Handling Failed for event: calamari/ceph/pool/deleted. error: not found
2017-01-26T14:17:09.933Z ERROR    events.go:1305 ceph_pool_delete_handler] skyring:02071545-c0c9-4c8a-806a-70002ad1422f-Error finding the storage: 42(id) from cluster: 5c5461a1-209f-4cdc-95bf-64e2bc4cc61e. error: not found
2017-01-26T14:17:09.933Z ERROR    events.go:1491 ProcessEvent] skyring:02071545-c0c9-4c8a-806a-70002ad1422f - Event Handling Failed for event: calamari/ceph/pool/deleted. error: not found

Pool default.rgw.buskets.data was created twice and this caused issues in the Cluster. PG's went to unclean state, tracked here https://bugzilla.redhat.com/show_bug.cgi?id=1416923.

2017-01-26T14:18:35.044Z ERROR    audit_logger.go:46 AuditLog] skyring:1f9c6250-eb80-48b2-a29f-171a3c4f4497-Could not send mail for event: Cluster health changed
2017-01-26T14:18:35.492Z ERROR    utils.go:124 FailTask] Create pool default.rgw.buckets.data failed on cluster: ceph: admin:b2088f5e-ec72-402f-b9e9-f3cce2961c3a - Failed to create pool: default.rgw.buckets.data for cluster: ceph. error: Failed409 CONFLICT
2017-01-26T14:18:43.039Z ERROR    mailnotifier.go:33 MailNotifier] skyring:03cf5a0c-8532-436d-9572-f2069b3919d2-Unable to read MailNotifier from DB: <nil>
2017-01-26T14:18:43.039Z WARNING  mail_notifier.go:119 getNotifier] skyring:03cf5a0c-8532-436d-9572-f2069b3919d2-Unable to read MailNotifier from DB: can't find Mail Notifier
2017-01-26T14:18:43.039Z WARNING  mail_notifier.go:159 MailNotify] skyring:03cf5a0c-8532-436d-9572-f2069b3919d2-Could not Get the notifier. Error: can't find Mail Notifier
2017-01-26T14:18:43.039Z ERROR    audit_logger.go:46 AuditLog] skyring:03cf5a0c-8532-436d-9572-f2069b3919d2-Could not send mail for event: Cluster health changed


2017-01-26T14:18:35.492Z ERROR    utils.go:124 FailTask] Create pool default.rgw.buckets.data failed on cluster: ceph: admin:b2088f5e-ec72-402f-b9e9-f3cce2961c3a - Failed to create pool: default.rgw.buckets.data for cluster: ceph. error: Failed409 CONFLICT

Version-Release number of selected component (if applicable):
RHCS 2.1

How reproducible:
Unable to reproduce in test environment.

Comment 28 Shubhendu Tripathi 2018-11-19 05:43:43 UTC
This product is EOL now