Back to bug 2093106

Who When What Removed Added
Vikhyat Umrao 2022-06-02 23:00:23 UTC Assignee nojha nmordech
Vikhyat Umrao 2022-06-02 23:01:42 UTC Link ID Github ceph/ceph/pull/46251
Vikhyat Umrao 2022-06-02 23:01:57 UTC Link ID Ceph Project Bug Tracker 55632
Red Hat One Jira (issues.redhat.com) 2022-06-02 23:05:18 UTC Link ID Red Hat Issue Tracker RHCEPH-4453
Veera Raghava Reddy 2022-06-05 16:42:37 UTC CC vereddy
Vikhyat Umrao 2022-06-15 19:53:56 UTC Summary [RHCS 6] osd/PGLog.cc: Trim duplicates by number of entries [RHCS 6] dups.size logging + COT dups trim command + online dups trimming fix
Assignee nmordech rzarzyns
Neha Ojha 2022-08-19 04:17:52 UTC Doc Type If docs needed, set a value Bug Fix
Link ID Github ceph/ceph/pull/47688
Radoslaw Zarzynski 2022-09-08 16:50:00 UTC Status ASSIGNED POST
errata-xmlrpc 2022-09-08 19:16:35 UTC Fixed In Version ceph-17.2.3-23.el9cp
Status POST MODIFIED
CC tserlin
Status MODIFIED ON_QA
Eliska 2022-09-13 09:06:03 UTC CC ekristov
Flags needinfo?(rzarzyns)
Radoslaw Zarzynski 2022-09-13 09:31:51 UTC Flags needinfo?(rzarzyns)
Doc Text Corrupted dups entries of a PG Log can be removed by off-line and on-line trimming

Previously, trimming of PG log dups entries could be prevented during the low-level PG split operation, which is used by the PG autoscaler with far higher frequency than by a human operator. Stalling the trimming of dups resulted in significant memory growth of PG log, leading to OSD crashes as it ran out of memory. Restarting an OSD did not solve the problem as the PG log is stored on disk and reloaded to RAM on startup.

With this fix, both off-line (using the `ceph-objectstore-tool` command) and on-line (within OSD) trimming is able to remove corrupted dups entries of a PG Log that jammed the on-line trimming machinery and were responsible for the memory growth. A debug improvement is implemented that prints the number of dups entries to the OSD's log to help future investigations.
Eliska 2022-09-13 09:36:53 UTC Blocks 2126050
Eliska 2022-09-20 14:40:55 UTC Doc Text Corrupted dups entries of a PG Log can be removed by off-line and on-line trimming

Previously, trimming of PG log dups entries could be prevented during the low-level PG split operation, which is used by the PG autoscaler with far higher frequency than by a human operator. Stalling the trimming of dups resulted in significant memory growth of PG log, leading to OSD crashes as it ran out of memory. Restarting an OSD did not solve the problem as the PG log is stored on disk and reloaded to RAM on startup.

With this fix, both off-line (using the `ceph-objectstore-tool` command) and on-line (within OSD) trimming is able to remove corrupted dups entries of a PG Log that jammed the on-line trimming machinery and were responsible for the memory growth. A debug improvement is implemented that prints the number of dups entries to the OSD's log to help future investigations.
.Corrupted dups entries of a PG Log can be removed by off-line and on-line trimming

Previously, trimming of PG log dups entries could be prevented during the low-level PG split operation, which is used by the PG autoscaler with far higher frequency than by a human operator. Stalling the trimming of dups resulted in significant memory growth of PG log, leading to OSD crashes as it ran out of memory. Restarting an OSD did not solve the problem as the PG log is stored on disk and reloaded to RAM on startup.

With this fix, both off-line, using the `ceph-objectstore-tool` command, and on-line, within OSD, trimming can remove corrupted dups entries of a PG log that jammed the on-line trimming machinery and were responsible for the memory growth. A debug improvement is implemented that prints the number of dups entries to the OSD's log to help future investigations.
Eliska 2022-09-21 14:48:39 UTC Docs Contact ekristov
Pawan 2022-11-04 03:57:54 UTC Status ON_QA VERIFIED
Red Hat Bugzilla 2022-12-31 19:13:31 UTC CC amathuri
Red Hat Bugzilla 2022-12-31 19:32:38 UTC CC pdhiran
QA Contact pdhiran
Red Hat Bugzilla 2022-12-31 20:00:03 UTC CC sseshasa
Red Hat Bugzilla 2022-12-31 22:32:23 UTC CC nmordech
Red Hat Bugzilla 2022-12-31 22:43:33 UTC CC rfriedma
Red Hat Bugzilla 2022-12-31 23:43:36 UTC Assignee rzarzyns nojha
CC rzarzyns
Red Hat Bugzilla 2022-12-31 23:45:56 UTC CC akupczyk
Red Hat Bugzilla 2023-01-01 05:35:24 UTC CC ksirivad
Red Hat Bugzilla 2023-01-01 05:39:38 UTC CC tserlin
Red Hat Bugzilla 2023-01-01 06:27:11 UTC CC lflores
Red Hat Bugzilla 2023-01-01 06:29:05 UTC CC choffman
Red Hat Bugzilla 2023-01-01 08:38:33 UTC CC nojha
Assignee nojha nobody
Red Hat Bugzilla 2023-01-01 08:39:41 UTC CC pdhange
Red Hat Bugzilla 2023-01-01 08:47:20 UTC CC vereddy
Red Hat Bugzilla 2023-01-01 08:49:55 UTC CC vumrao
Alasdair Kergon 2023-01-04 04:40:45 UTC CC akupczyk
Alasdair Kergon 2023-01-04 04:43:34 UTC CC amathuri
Alasdair Kergon 2023-01-04 04:51:21 UTC Assignee nobody rzarzyns
Alasdair Kergon 2023-01-04 04:56:54 UTC QA Contact pdhiran
Alasdair Kergon 2023-01-04 05:08:58 UTC CC ksirivad
Alasdair Kergon 2023-01-04 05:10:58 UTC CC lflores
Alasdair Kergon 2023-01-04 05:21:30 UTC CC nmordech
Alasdair Kergon 2023-01-04 05:21:38 UTC CC nojha
Alasdair Kergon 2023-01-04 05:28:18 UTC CC pdhange
Alasdair Kergon 2023-01-04 05:30:13 UTC CC pdhiran
Alasdair Kergon 2023-01-04 05:34:52 UTC CC rfriedma
Alasdair Kergon 2023-01-04 05:37:37 UTC CC rzarzyns
Alasdair Kergon 2023-01-04 05:59:30 UTC CC vumrao
Alasdair Kergon 2023-01-04 06:13:47 UTC CC choffman
Alasdair Kergon 2023-01-04 06:25:53 UTC CC tserlin
Alasdair Kergon 2023-01-04 06:56:31 UTC CC sseshasa
Alasdair Kergon 2023-01-04 06:59:12 UTC CC vereddy
Red Hat Bugzilla 2023-01-09 08:29:52 UTC CC ceph-eng-bugs
Alasdair Kergon 2023-01-09 19:43:36 UTC CC ceph-eng-bugs
errata-xmlrpc 2023-03-20 18:37:49 UTC Status VERIFIED RELEASE_PENDING
errata-xmlrpc 2023-03-20 18:56:39 UTC Resolution --- ERRATA
Status RELEASE_PENDING CLOSED
Last Closed 2023-03-20 18:56:39 UTC
errata-xmlrpc 2023-03-20 18:57:06 UTC Link ID Red Hat Product Errata RHBA-2023:1360

Back to bug 2093106