2060626 – [GSS] noobaa endpoint throwing 'relation "idx_btree_*" already exists

Bug 2060626 - [GSS] noobaa endpoint throwing 'relation "idx_btree_*" already exists

Summary: [GSS] noobaa endpoint throwing 'relation "idx_btree_*" already exists

Keywords:
Status:	CLOSED CURRENTRELEASE
Alias:	None
Product:	Red Hat OpenShift Data Foundation
Classification:	Red Hat Storage
Component:	Multi-Cloud Object Gateway
Sub Component:
Version:	4.8
Hardware:	All
OS:	All
Priority:	unspecified
Severity:	medium
Target Milestone:	---
Target Release:	---
Assignee:	Nimrod Becker
QA Contact:	Elad
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+	depends on / blocked

Reported:	2022-03-03 22:04 UTC by kelwhite
Modified:	2023-08-09 16:49 UTC (History)
CC List:	6 users (show)
Fixed In Version:
Doc Type:	If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:	2022-03-22 08:39:02 UTC
Embargoed:

Attachments	(Terms of Use)

Description kelwhite 2022-03-03 22:04:06 UTC

Description of problem (please be detailed as possible and provide log
snippests):
Initially, when trying to access certain objects within an object storage bucket we get Gateway Time-out errors. However, other objects seem to work but the response time is slow.  I'm not sure if it's related or not but I also see errors in the noobaa-endpoint pod like "core.util.postgres_client:: INVALID_SCHEMA_DB objectstats ERRORS".

The above was resolved when the noobaa pod were deleted in the following order:

1 - noobaa-db
2 - nooba-core
3 - nooba-endpoint 
4 - nooba-operator

However, now the noobaa endpoint is complaining a bunch of 'idx_btree*' relations already exist:

$  cat 0040-noobaa-endpoint-6846bb76dd-cxmfk-endpoint_\(1\).log|grep exists
Mar-1 15:41:48.789 [Endpoint/11] [ERROR] core.util.postgres_client:: postgres_client: T00000000|Q00000013: failed with error: error: relation "idx_btree_clusters__id_index" already exists
Mar-1 15:41:48.791 [Endpoint/11] [ERROR] core.util.postgres_client:: postgres_client: T00000000|Q00000014: failed with error: error: relation "idx_btree_clusters_owner_secret" already exists
Mar-1 15:41:48.792 [Endpoint/11] [ERROR] core.util.postgres_client:: postgres_client: T00000000|Q00000015: failed with error: error: relation "idx_btree_chunk_configs__id_index" already exists
Mar-1 15:41:48.793 [Endpoint/11] [ERROR] core.util.postgres_client:: postgres_client: T00000000|Q00000016: failed with error: error: relation "idx_btree_namespace_resources__id_index" already exists
Mar-1 15:41:48.794 [Endpoint/11] [ERROR] core.util.postgres_client:: postgres_client: T00000000|Q00000017: failed with error: error: relation "idx_btree_namespace_resources_name_deleted" already exists
Mar-1 15:41:48.860 [Endpoint/11] [ERROR] core.util.postgres_client:: postgres_client: T00000000|Q00000018: failed with error: error: relation "idx_btree_systems__id_index" already exists
Mar-1 15:41:48.862 [Endpoint/11] [ERROR] core.util.postgres_client:: postgres_client: T00000000|Q00000024: failed with error: error: relation "idx_btree_tieringpolicies_system_name_deleted" already exists
Mar-1 15:41:48.863 [Endpoint/11] [ERROR] core.util.postgres_client:: postgres_client: T00000000|Q00000025: failed with error: error: relation "idx_btree_master_keys__id_index" already exists
Mar-1 15:41:48.864 [Endpoint/11] [ERROR] core.util.postgres_client:: postgres_client: T00000000|Q00000026: failed with error: error: relation "idx_btree_agent_configs__id_index" already exists
Mar-1 15:41:48.865 [Endpoint/11] [ERROR] core.util.postgres_client:: postgres_client: T00000000|Q00000027: failed with error: error: relation "idx_btree_agent_configs_system_name" already exists
Mar-1 15:41:48.878 [Endpoint/11] [ERROR] core.util.postgres_client:: postgres_client: T00000000|Q00000020: failed with error: error: relation "idx_btree_accounts__id_index" already exists
Mar-1 15:41:48.879 [Endpoint/11] [ERROR] core.util.postgres_client:: postgres_client: T00000000|Q00000031: failed with error: error: relation "idx_btree_buckets_system_name_deleted" already exists
Mar-1 15:41:48.880 [Endpoint/11] [ERROR] core.util.postgres_client:: postgres_client: T00000000|Q00000019: failed with error: error: relation "idx_btree_systems_name_deleted" already exists
Mar-1 15:41:48.881 [Endpoint/11] [ERROR] core.util.postgres_client:: postgres_client: T00000000|Q00000021: failed with error: error: relation "idx_btree_accounts_email_deleted" already exists
Mar-1 15:41:48.882 [Endpoint/11] [ERROR] core.util.postgres_client:: postgres_client: T00000000|Q00000023: failed with error: error: relation "idx_btree_tieringpolicies__id_index" already exists
Mar-1 15:41:48.884 [Endpoint/11] [ERROR] core.util.postgres_client:: postgres_client: T00000000|Q00000022: failed with error: error: relation "idx_btree_roles__id_index" already exists
Mar-1 15:41:48.884 [Endpoint/11] [ERROR] core.util.postgres_client:: postgres_client: T00000000|Q00000029: failed with error: error: relation "idx_btree_pools_system_name_deleted" already exists
Mar-1 15:41:48.957 [Endpoint/11] [ERROR] core.util.postgres_client:: postgres_client: T00000000|Q00000032: failed with error: error: relation "idx_btree_tiers__id_index" already exists
Mar-1 15:41:48.958 [Endpoint/11] [ERROR] core.util.postgres_client:: postgres_client: T00000000|Q00000028: failed with error: error: relation "idx_btree_pools__id_index" already exists
Mar-1 15:41:48.959 [Endpoint/11] [ERROR] core.util.postgres_client:: postgres_client: T00000000|Q00000030: failed with error: error: relation "idx_btree_buckets__id_index" already exists
Mar-1 15:41:48.968 [Endpoint/11] [ERROR] core.util.postgres_client:: postgres_client: T00000000|Q00000033: failed with error: error: relation "idx_btree_tiers_system_name_deleted" already exists
Mar-1 15:41:57.528 [Endpoint/11] [ERROR] core.util.postgres_client:: postgres_client: T00000000|Q00000088: failed with error: error: relation "idx_btree_objectmds__id_index" already exists
Mar-1 15:41:57.529 [Endpoint/11] [ERROR] core.util.postgres_client:: postgres_client: T00000000|Q00000093: failed with error: error: relation "idx_btree_objectmds_create_time" already exists
Mar-1 15:41:57.530 [Endpoint/11] [ERROR] core.util.postgres_client:: postgres_client: T00000000|Q00000094: failed with error: error: relation "idx_btree_objectmds_deleted" already exists
Mar-1 15:41:57.534 [Endpoint/11] [ERROR] core.util.postgres_client:: postgres_client: T00000000|Q00000089: failed with error: error: relation "idx_btree_objectmds_latest_version_index" already exists
Mar-1 15:41:57.535 [Endpoint/11] [ERROR] core.util.postgres_client:: postgres_client: T00000000|Q00000090: failed with error: error: relation "idx_btree_objectmds_null_version_index" already exists
Mar-1 15:41:57.536 [Endpoint/11] [ERROR] core.util.postgres_client:: postgres_client: T00000000|Q00000091: failed with error: error: relation "idx_btree_objectmds_version_seq_index" already exists
Mar-1 15:41:57.537 [Endpoint/11] [ERROR] core.util.postgres_client:: postgres_client: T00000000|Q00000092: failed with error: error: relation "idx_btree_objectmds_upload_index" already exists
Mar-1 15:41:57.538 [Endpoint/11] [ERROR] core.util.postgres_client:: postgres_client: T00000000|Q00000098: failed with error: error: relation "idx_btree_objectmultiparts_obj_num" already exists
Mar-1 15:41:57.539 [Endpoint/11] [ERROR] core.util.postgres_client:: postgres_client: T00000000|Q00000099: failed with error: error: relation "idx_btree_objectparts__id_index" already exists
Mar-1 15:41:57.540 [Endpoint/11] [ERROR] core.util.postgres_client:: postgres_client: T00000000|Q00000100: failed with error: error: relation "idx_btree_objectparts_obj_1_start_1" already exists
Mar-1 15:41:57.547 [Endpoint/11] [ERROR] core.util.postgres_client:: postgres_client: T00000000|Q00000097: failed with error: error: relation "idx_btree_objectmultiparts__id_index" already exists
Mar-1 15:41:57.549 [Endpoint/11] [ERROR] core.util.postgres_client:: postgres_client: T00000000|Q00000102: failed with error: error: relation "idx_btree_datachunks__id_index" already exists
Mar-1 15:41:57.557 [Endpoint/11] [ERROR] core.util.postgres_client:: postgres_client: T00000000|Q00000104: failed with error: error: relation "idx_btree_datachunks_dedup_key" already exists
Mar-1 15:41:57.558 [Endpoint/11] [ERROR] core.util.postgres_client:: postgres_client: T00000000|Q00000103: failed with error: error: relation "idx_btree_datachunks_id_desc" already exists
Mar-1 15:41:57.559 [Endpoint/11] [ERROR] core.util.postgres_client:: postgres_client: T00000000|Q00000101: failed with error: error: relation "idx_btree_objectparts_chunk" already exists
Mar-1 15:41:57.559 [Endpoint/11] [ERROR] core.util.postgres_client:: postgres_client: T00000000|Q00000095: failed with error: error: relation "idx_btree_objectmds_bucket" already exists
Mar-1 15:41:57.561 [Endpoint/11] [ERROR] core.util.postgres_client:: postgres_client: T00000000|Q00000096: failed with error: error: relation "idx_btree_objectmds_deleted_unreclaimed_index" already exists
Mar-1 15:41:57.563 [Endpoint/11] [ERROR] core.util.postgres_client:: postgres_client: T00000000|Q00000106: failed with error: error: relation "idx_btree_datachunks_deleted" already exists
Mar-1 15:41:57.563 [Endpoint/11] [ERROR] core.util.postgres_client:: postgres_client: T00000000|Q00000107: failed with error: error: relation "idx_btree_mdsequences__id_index" already exists
Mar-1 15:41:57.566 [Endpoint/11] [ERROR] core.util.postgres_client:: postgres_client: T00000000|Q00000108: failed with error: error: relation "idx_btree_datablocks__id_index" already exists
Mar-1 15:41:57.657 [Endpoint/11] [ERROR] core.util.postgres_client:: postgres_client: T00000000|Q00000105: failed with error: error: relation "idx_btree_datachunks_tiering_index" already exists
Mar-1 15:41:57.661 [Endpoint/11] [ERROR] core.util.postgres_client:: postgres_client: T00000000|Q00000109: failed with error: error: relation "idx_btree_datablocks_chunk" already exists
Mar-1 15:41:57.663 [Endpoint/11] [ERROR] core.util.postgres_client:: postgres_client: T00000000|Q00000110: failed with error: error: relation "idx_btree_datablocks_node__id" already exists
Mar-1 15:41:57.664 [Endpoint/11] [ERROR] core.util.postgres_client:: postgres_client: T00000000|Q00000111: failed with error: error: relation "idx_btree_datablocks_deleted" already exists
Mar-1 15:45:43.002 [Endpoint/11] [ERROR] core.util.postgres_client:: postgres_client: T00000000|Q00000456: failed with error: error: relation "idx_btree_bucketstats__id_index" already exists
Mar-1 15:45:43.008 [Endpoint/11] [ERROR] core.util.postgres_client:: postgres_client: T00000000|Q00000457: failed with error: error: relation "idx_btree_bucketstats_bucket" already exists
Mar-1 15:51:48.777 [Endpoint/11] [ERROR] core.util.postgres_client:: postgres_client: T00000000|Q00001048: failed with error: error: relation "idx_btree_objectstats__id_index" already exists
Mar-1 15:51:48.778 [Endpoint/11] [ERROR] core.util.postgres_client:: postgres_client: T00000000|Q00001050: failed with error: error: relation "idx_btree_usagereports__id_index" already exists
Mar-1 15:51:48.779 [Endpoint/11] [ERROR] core.util.postgres_client:: postgres_client: T00000000|Q00001049: failed with error: error: relation "idx_btree_objectstats_system" already exists
Mar-1 15:51:48.780 [Endpoint/11] [ERROR] core.util.postgres_client:: postgres_client: T00000000|Q00001051: failed with error: error: relation "idx_btree_usagereports_start_time_aggregated_time_aggregated_ti" already exists
Mar-1 15:51:48.780 [Endpoint/11] [ERROR] core.util.postgres_client:: postgres_client: T00000000|Q00001053: failed with error: error: relation "idx_btree_endpointgroupreports_start_time_aggregated_time_aggre" already exists
Mar-1 15:51:48.781 [Endpoint/11] [ERROR] core.util.postgres_client:: postgres_client: T00000000|Q00001052: failed with error: error: relation "idx_btree_endpointgroupreports__id_index" already exists
Mar-1 15:53:55.972 [Endpoint/11] [ERROR] core.util.postgres_client:: postgres_client: T00000000|Q00001259: failed with error: error: relation "idx_btree_activitylogs__id_index" already exists
Mar-1 15:53:55.973 [Endpoint/11] [ERROR] core.util.postgres_client:: postgres_client: T00000000|Q00001260: failed with error: error: relation "idx_btree_activitylogs_time" already exists

When attempting to insert a duplicate row?:

Mar-1 15:53:55.965^[[35m [Endpoint/11] ^[[36m   [L0]^[[39m core.server.notifications.dispatcher:: Adding ActivityLog entry { system: 61239c386877a4002411853e, level: ^[[32m'info'^[[39m, event: ^[[32m'obj.deleted'^[[39m, obj: 6217ed8bf99307000c897c22, actor: 61b12bffb1a6f300245b10d6, desc: SENSITIVE-99d8e76a4919b7b1 }

Since the customer isn't noticing any issues with any noobaa* pods but these errors in the endpoint logs, do we need to worry about these? If we don't have to worry, why are these being thrown? What would be the cause of this?  

Version of all relevant components (if applicable):
OCS 4.8

Does this issue impact your ability to continue to work with the product
(please explain in detail what is the user impact)?
No, everything seems to be working fine.

Note You need to log in before you can comment on or make changes to this bug.