Bug 2264480 - db-noobaa-db-pg-0 PVC continues to grow extensively, activitylogs already truncated need to either fix or workaround
Summary: db-noobaa-db-pg-0 PVC continues to grow extensively, activitylogs already tru...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenShift Data Foundation
Classification: Red Hat Storage
Component: Multi-Cloud Object Gateway
Version: 4.12
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ---
: ODF 4.16.0
Assignee: Liran Mauda
QA Contact: Tiffany Nguyen
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2024-02-15 22:18 UTC by lema
Modified: 2024-09-25 23:19 UTC (History)
6 users (show)

Fixed In Version: 4.16.0-39
Doc Type: No Doc Update
Doc Text:
Clone Of:
Environment:
Last Closed: 2024-07-17 13:13:59 UTC
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github noobaa noobaa-core pull 7804 0 None Merged convert activity logs in the object server to dbg1 2024-02-18 07:34:44 UTC
Github red-hat-storage ocs-ci pull 9720 0 None Merged Test noobaa db activitylogs table for unnecessary logs when in default log level 2024-06-26 06:14:04 UTC
Red Hat Product Errata RHSA-2024:4591 0 None None None 2024-07-17 13:14:03 UTC

Description lema 2024-02-15 22:18:36 UTC
Description of problem (please be detailed as possible and provide log
snippets):

The customer has an environment from GCP Loki Storage setting LogStore=>Noobaa(MCG)=>GCS(S3). (development) 

Since last year, customers noticed the db-noobaa-db-pg-0 PVC continues to grow extensively, the trend like this previous ratio is 0.201,  from 0.611 (Dec 30th) to 0.812 (Feb 15th) for the last 7 weeks.


Version of all relevant components (if applicable):
4.12


Does this issue impact your ability to continue to work with the product
(please explain in detail what is the user impact)?

Customers would like to roll out applications very soon, but this type of DB is increasing concern for system stability.  


Is there any workaround available to the best of your knowledge?

- we do list everything inside DB and also TRUNCATE activitylogs already but still, customers don't want to expand the DB at this stage.

nbcore=# \d+
                              List of relations
 Schema |         Name         |   Type   | Owner  |    Size    | Description
--------+----------------------+----------+--------+------------+-------------
 public | accounts             | table    | noobaa | 96 kB      |
 public | activitylogs         | table    | noobaa | 7883 MB    |
 public | agent_configs        | table    | noobaa | 8192 bytes |
 public | alertslogs           | table    | noobaa | 48 kB      |
 public | buckets              | table    | noobaa | 88 kB      |
 public | bucketstats          | table    | noobaa | 64 kB      |
 public | chunk_configs        | table    | noobaa | 48 kB      |
 public | clusters             | table    | noobaa | 144 kB     |
 public | datablocks           | table    | noobaa | 2734 MB    |
 public | datachunks           | table    | noobaa | 4701 MB    |
 public | endpointgroupreports | table    | noobaa | 8512 kB    |
 public | funcs                | table    | noobaa | 8192 bytes |
 public | iostats              | table    | noobaa | 144 kB     |
 public | master_keys          | table    | noobaa | 48 kB      |
 public | mdsequencesnative    | sequence | noobaa | 8192 bytes |
 public | namespace_resources  | table    | noobaa | 8192 bytes |
 public | nodes                | table    | noobaa | 136 kB     |
 public | objectmds            | table    | noobaa | 4707 MB    |
 public | objectmultiparts     | table    | noobaa | 8192 bytes |
 public | objectparts          | table    | noobaa | 2543 MB    |
 public | objectstats          | table    | noobaa | 2624 kB    |
 public | pools                | table    | noobaa | 216 kB     |
 public | replicationconfigs   | table    | noobaa | 8192 bytes |
 public | roles                | table    | noobaa | 48 kB      |
 public | system_history       | table    | noobaa | 856 kB     |
 public | systems              | table    | noobaa | 88 kB      |
 public | tieringpolicies      | table    | noobaa | 48 kB      |
 public | tiers                | table    | noobaa | 48 kB      |
 public | usagereports         | table    | noobaa | 5544 kB    |
(29 rows)


Rate from 1 - 5 the complexity of the scenario you performed that caused this
bug (1 - very simple, 5 - very complex)?

-3 

Can this issue reproducible?

It is very hard to reproduce since the customer has GCP and Loki involved.  


Can this issue reproduce from the UI?

NO

If this is a regression, please provide more details to justify this:

I am not able to reproduce since I don't have any GCP and Loki.

Steps to Reproduce:
1.
2.
3.


Actual results:


Expected results:


Additional info:

We did review this - https://bugzilla.redhat.com/show_bug.cgi?id=1932846 , this type of activitylog may somehow related to each other.

CU would like to confirm if these are the same or similar then will go upgrade, but if these are fresh new bugs, they would like to get any recommendation or fix before they consider rolling out applications.

Comment 9 krishnaram Karthick 2024-03-19 13:10:33 UTC
What would be the steps to verify this bug?

Comment 13 Tiffany Nguyen 2024-04-17 00:22:18 UTC
Verified using ODF 4.16.0-76. Ran MCG test to upload objects and DB activitylogs collection didn't grow extensively.

Before running MCG object tests:
nbcore-# \d+
                                             List of relations
 Schema |         Name         |   Type   | Owner  | Persistence | Access method |    Size    | Description 
--------+----------------------+----------+--------+-------------+---------------+------------+-------------
 public | accounts             | table    | noobaa | permanent   | heap          | 56 kB      | 
 public | activitylogs         | table    | noobaa | permanent   | heap          | 88 kB      | 


After running MCG object tests:
nbcore-# \d+
                                             List of relations
 Schema |         Name         |   Type   | Owner  | Persistence | Access method |    Size    | Description 
--------+----------------------+----------+--------+-------------+---------------+------------+-------------
 public | accounts             | table    | noobaa | permanent   | heap          | 120 kB     | 
 public | activitylogs         | table    | noobaa | permanent   | heap          | 104 kB     |

Comment 17 errata-xmlrpc 2024-07-17 13:13:59 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Important: Red Hat OpenShift Data Foundation 4.16.0 security, enhancement & bug fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2024:4591


Note You need to log in before you can comment on or make changes to this bug.