Bug 2060626
| Summary: | [GSS] noobaa endpoint throwing 'relation "idx_btree_*" already exists | ||
|---|---|---|---|
| Product: | [Red Hat Storage] Red Hat OpenShift Data Foundation | Reporter: | kelwhite |
| Component: | Multi-Cloud Object Gateway | Assignee: | Nimrod Becker <nbecker> |
| Status: | CLOSED CURRENTRELEASE | QA Contact: | Elad <ebenahar> |
| Severity: | medium | Docs Contact: | |
| Priority: | unspecified | ||
| Version: | 4.8 | CC: | etamir, mmuench, nbecker, ocs-bugs, odf-bz-bot, tdesala |
| Target Milestone: | --- | ||
| Target Release: | --- | ||
| Hardware: | All | ||
| OS: | All | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2022-03-22 08:39:02 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
Description of problem (please be detailed as possible and provide log snippests): Initially, when trying to access certain objects within an object storage bucket we get Gateway Time-out errors. However, other objects seem to work but the response time is slow. I'm not sure if it's related or not but I also see errors in the noobaa-endpoint pod like "core.util.postgres_client:: INVALID_SCHEMA_DB objectstats ERRORS". The above was resolved when the noobaa pod were deleted in the following order: 1 - noobaa-db 2 - nooba-core 3 - nooba-endpoint 4 - nooba-operator However, now the noobaa endpoint is complaining a bunch of 'idx_btree*' relations already exist: $ cat 0040-noobaa-endpoint-6846bb76dd-cxmfk-endpoint_\(1\).log|grep exists Mar-1 15:41:48.789 [Endpoint/11] [ERROR] core.util.postgres_client:: postgres_client: T00000000|Q00000013: failed with error: error: relation "idx_btree_clusters__id_index" already exists Mar-1 15:41:48.791 [Endpoint/11] [ERROR] core.util.postgres_client:: postgres_client: T00000000|Q00000014: failed with error: error: relation "idx_btree_clusters_owner_secret" already exists Mar-1 15:41:48.792 [Endpoint/11] [ERROR] core.util.postgres_client:: postgres_client: T00000000|Q00000015: failed with error: error: relation "idx_btree_chunk_configs__id_index" already exists Mar-1 15:41:48.793 [Endpoint/11] [ERROR] core.util.postgres_client:: postgres_client: T00000000|Q00000016: failed with error: error: relation "idx_btree_namespace_resources__id_index" already exists Mar-1 15:41:48.794 [Endpoint/11] [ERROR] core.util.postgres_client:: postgres_client: T00000000|Q00000017: failed with error: error: relation "idx_btree_namespace_resources_name_deleted" already exists Mar-1 15:41:48.860 [Endpoint/11] [ERROR] core.util.postgres_client:: postgres_client: T00000000|Q00000018: failed with error: error: relation "idx_btree_systems__id_index" already exists Mar-1 15:41:48.862 [Endpoint/11] [ERROR] core.util.postgres_client:: postgres_client: T00000000|Q00000024: failed with error: error: relation "idx_btree_tieringpolicies_system_name_deleted" already exists Mar-1 15:41:48.863 [Endpoint/11] [ERROR] core.util.postgres_client:: postgres_client: T00000000|Q00000025: failed with error: error: relation "idx_btree_master_keys__id_index" already exists Mar-1 15:41:48.864 [Endpoint/11] [ERROR] core.util.postgres_client:: postgres_client: T00000000|Q00000026: failed with error: error: relation "idx_btree_agent_configs__id_index" already exists Mar-1 15:41:48.865 [Endpoint/11] [ERROR] core.util.postgres_client:: postgres_client: T00000000|Q00000027: failed with error: error: relation "idx_btree_agent_configs_system_name" already exists Mar-1 15:41:48.878 [Endpoint/11] [ERROR] core.util.postgres_client:: postgres_client: T00000000|Q00000020: failed with error: error: relation "idx_btree_accounts__id_index" already exists Mar-1 15:41:48.879 [Endpoint/11] [ERROR] core.util.postgres_client:: postgres_client: T00000000|Q00000031: failed with error: error: relation "idx_btree_buckets_system_name_deleted" already exists Mar-1 15:41:48.880 [Endpoint/11] [ERROR] core.util.postgres_client:: postgres_client: T00000000|Q00000019: failed with error: error: relation "idx_btree_systems_name_deleted" already exists Mar-1 15:41:48.881 [Endpoint/11] [ERROR] core.util.postgres_client:: postgres_client: T00000000|Q00000021: failed with error: error: relation "idx_btree_accounts_email_deleted" already exists Mar-1 15:41:48.882 [Endpoint/11] [ERROR] core.util.postgres_client:: postgres_client: T00000000|Q00000023: failed with error: error: relation "idx_btree_tieringpolicies__id_index" already exists Mar-1 15:41:48.884 [Endpoint/11] [ERROR] core.util.postgres_client:: postgres_client: T00000000|Q00000022: failed with error: error: relation "idx_btree_roles__id_index" already exists Mar-1 15:41:48.884 [Endpoint/11] [ERROR] core.util.postgres_client:: postgres_client: T00000000|Q00000029: failed with error: error: relation "idx_btree_pools_system_name_deleted" already exists Mar-1 15:41:48.957 [Endpoint/11] [ERROR] core.util.postgres_client:: postgres_client: T00000000|Q00000032: failed with error: error: relation "idx_btree_tiers__id_index" already exists Mar-1 15:41:48.958 [Endpoint/11] [ERROR] core.util.postgres_client:: postgres_client: T00000000|Q00000028: failed with error: error: relation "idx_btree_pools__id_index" already exists Mar-1 15:41:48.959 [Endpoint/11] [ERROR] core.util.postgres_client:: postgres_client: T00000000|Q00000030: failed with error: error: relation "idx_btree_buckets__id_index" already exists Mar-1 15:41:48.968 [Endpoint/11] [ERROR] core.util.postgres_client:: postgres_client: T00000000|Q00000033: failed with error: error: relation "idx_btree_tiers_system_name_deleted" already exists Mar-1 15:41:57.528 [Endpoint/11] [ERROR] core.util.postgres_client:: postgres_client: T00000000|Q00000088: failed with error: error: relation "idx_btree_objectmds__id_index" already exists Mar-1 15:41:57.529 [Endpoint/11] [ERROR] core.util.postgres_client:: postgres_client: T00000000|Q00000093: failed with error: error: relation "idx_btree_objectmds_create_time" already exists Mar-1 15:41:57.530 [Endpoint/11] [ERROR] core.util.postgres_client:: postgres_client: T00000000|Q00000094: failed with error: error: relation "idx_btree_objectmds_deleted" already exists Mar-1 15:41:57.534 [Endpoint/11] [ERROR] core.util.postgres_client:: postgres_client: T00000000|Q00000089: failed with error: error: relation "idx_btree_objectmds_latest_version_index" already exists Mar-1 15:41:57.535 [Endpoint/11] [ERROR] core.util.postgres_client:: postgres_client: T00000000|Q00000090: failed with error: error: relation "idx_btree_objectmds_null_version_index" already exists Mar-1 15:41:57.536 [Endpoint/11] [ERROR] core.util.postgres_client:: postgres_client: T00000000|Q00000091: failed with error: error: relation "idx_btree_objectmds_version_seq_index" already exists Mar-1 15:41:57.537 [Endpoint/11] [ERROR] core.util.postgres_client:: postgres_client: T00000000|Q00000092: failed with error: error: relation "idx_btree_objectmds_upload_index" already exists Mar-1 15:41:57.538 [Endpoint/11] [ERROR] core.util.postgres_client:: postgres_client: T00000000|Q00000098: failed with error: error: relation "idx_btree_objectmultiparts_obj_num" already exists Mar-1 15:41:57.539 [Endpoint/11] [ERROR] core.util.postgres_client:: postgres_client: T00000000|Q00000099: failed with error: error: relation "idx_btree_objectparts__id_index" already exists Mar-1 15:41:57.540 [Endpoint/11] [ERROR] core.util.postgres_client:: postgres_client: T00000000|Q00000100: failed with error: error: relation "idx_btree_objectparts_obj_1_start_1" already exists Mar-1 15:41:57.547 [Endpoint/11] [ERROR] core.util.postgres_client:: postgres_client: T00000000|Q00000097: failed with error: error: relation "idx_btree_objectmultiparts__id_index" already exists Mar-1 15:41:57.549 [Endpoint/11] [ERROR] core.util.postgres_client:: postgres_client: T00000000|Q00000102: failed with error: error: relation "idx_btree_datachunks__id_index" already exists Mar-1 15:41:57.557 [Endpoint/11] [ERROR] core.util.postgres_client:: postgres_client: T00000000|Q00000104: failed with error: error: relation "idx_btree_datachunks_dedup_key" already exists Mar-1 15:41:57.558 [Endpoint/11] [ERROR] core.util.postgres_client:: postgres_client: T00000000|Q00000103: failed with error: error: relation "idx_btree_datachunks_id_desc" already exists Mar-1 15:41:57.559 [Endpoint/11] [ERROR] core.util.postgres_client:: postgres_client: T00000000|Q00000101: failed with error: error: relation "idx_btree_objectparts_chunk" already exists Mar-1 15:41:57.559 [Endpoint/11] [ERROR] core.util.postgres_client:: postgres_client: T00000000|Q00000095: failed with error: error: relation "idx_btree_objectmds_bucket" already exists Mar-1 15:41:57.561 [Endpoint/11] [ERROR] core.util.postgres_client:: postgres_client: T00000000|Q00000096: failed with error: error: relation "idx_btree_objectmds_deleted_unreclaimed_index" already exists Mar-1 15:41:57.563 [Endpoint/11] [ERROR] core.util.postgres_client:: postgres_client: T00000000|Q00000106: failed with error: error: relation "idx_btree_datachunks_deleted" already exists Mar-1 15:41:57.563 [Endpoint/11] [ERROR] core.util.postgres_client:: postgres_client: T00000000|Q00000107: failed with error: error: relation "idx_btree_mdsequences__id_index" already exists Mar-1 15:41:57.566 [Endpoint/11] [ERROR] core.util.postgres_client:: postgres_client: T00000000|Q00000108: failed with error: error: relation "idx_btree_datablocks__id_index" already exists Mar-1 15:41:57.657 [Endpoint/11] [ERROR] core.util.postgres_client:: postgres_client: T00000000|Q00000105: failed with error: error: relation "idx_btree_datachunks_tiering_index" already exists Mar-1 15:41:57.661 [Endpoint/11] [ERROR] core.util.postgres_client:: postgres_client: T00000000|Q00000109: failed with error: error: relation "idx_btree_datablocks_chunk" already exists Mar-1 15:41:57.663 [Endpoint/11] [ERROR] core.util.postgres_client:: postgres_client: T00000000|Q00000110: failed with error: error: relation "idx_btree_datablocks_node__id" already exists Mar-1 15:41:57.664 [Endpoint/11] [ERROR] core.util.postgres_client:: postgres_client: T00000000|Q00000111: failed with error: error: relation "idx_btree_datablocks_deleted" already exists Mar-1 15:45:43.002 [Endpoint/11] [ERROR] core.util.postgres_client:: postgres_client: T00000000|Q00000456: failed with error: error: relation "idx_btree_bucketstats__id_index" already exists Mar-1 15:45:43.008 [Endpoint/11] [ERROR] core.util.postgres_client:: postgres_client: T00000000|Q00000457: failed with error: error: relation "idx_btree_bucketstats_bucket" already exists Mar-1 15:51:48.777 [Endpoint/11] [ERROR] core.util.postgres_client:: postgres_client: T00000000|Q00001048: failed with error: error: relation "idx_btree_objectstats__id_index" already exists Mar-1 15:51:48.778 [Endpoint/11] [ERROR] core.util.postgres_client:: postgres_client: T00000000|Q00001050: failed with error: error: relation "idx_btree_usagereports__id_index" already exists Mar-1 15:51:48.779 [Endpoint/11] [ERROR] core.util.postgres_client:: postgres_client: T00000000|Q00001049: failed with error: error: relation "idx_btree_objectstats_system" already exists Mar-1 15:51:48.780 [Endpoint/11] [ERROR] core.util.postgres_client:: postgres_client: T00000000|Q00001051: failed with error: error: relation "idx_btree_usagereports_start_time_aggregated_time_aggregated_ti" already exists Mar-1 15:51:48.780 [Endpoint/11] [ERROR] core.util.postgres_client:: postgres_client: T00000000|Q00001053: failed with error: error: relation "idx_btree_endpointgroupreports_start_time_aggregated_time_aggre" already exists Mar-1 15:51:48.781 [Endpoint/11] [ERROR] core.util.postgres_client:: postgres_client: T00000000|Q00001052: failed with error: error: relation "idx_btree_endpointgroupreports__id_index" already exists Mar-1 15:53:55.972 [Endpoint/11] [ERROR] core.util.postgres_client:: postgres_client: T00000000|Q00001259: failed with error: error: relation "idx_btree_activitylogs__id_index" already exists Mar-1 15:53:55.973 [Endpoint/11] [ERROR] core.util.postgres_client:: postgres_client: T00000000|Q00001260: failed with error: error: relation "idx_btree_activitylogs_time" already exists When attempting to insert a duplicate row?: Mar-1 15:53:55.965^[[35m [Endpoint/11] ^[[36m [L0]^[[39m core.server.notifications.dispatcher:: Adding ActivityLog entry { system: 61239c386877a4002411853e, level: ^[[32m'info'^[[39m, event: ^[[32m'obj.deleted'^[[39m, obj: 6217ed8bf99307000c897c22, actor: 61b12bffb1a6f300245b10d6, desc: SENSITIVE-99d8e76a4919b7b1 } Since the customer isn't noticing any issues with any noobaa* pods but these errors in the endpoint logs, do we need to worry about these? If we don't have to worry, why are these being thrown? What would be the cause of this? Version of all relevant components (if applicable): OCS 4.8 Does this issue impact your ability to continue to work with the product (please explain in detail what is the user impact)? No, everything seems to be working fine.