Back to bug 2119853

Who When What Removed Added
Vikhyat Umrao 2022-08-19 16:05:04 UTC Target Release Backlog 5.3
Red Hat One Jira (issues.redhat.com) 2022-08-19 16:31:33 UTC Link ID Red Hat Issue Tracker RHCEPH-5121
Neha Ojha 2022-08-22 04:28:34 UTC Assignee nojha rzarzyns
Radoslaw Zarzynski 2022-08-26 12:35:16 UTC Flags needinfo?(tserlin)
Radoslaw Zarzynski 2022-08-30 22:33:24 UTC Flags needinfo?(tserlin)
Flags needinfo?(rzarzyns)
Flags needinfo?(rzarzyns)
Vikhyat Umrao 2022-08-30 23:45:10 UTC Status ASSIGNED POST
errata-xmlrpc 2022-08-31 01:42:01 UTC Status POST MODIFIED
Fixed In Version ceph-16.2.10-28.el8cp
Status MODIFIED ON_QA
Pawan 2022-09-19 13:03:24 UTC Assignee rzarzyns skanta
CC skanta
QA Contact pdhiran skanta
Assignee skanta rzarzyns
QA Contact skanta pdhiran
Status ON_QA VERIFIED
Radoslaw Zarzynski 2022-09-22 12:57:45 UTC CC anarnold
Doc Text Corrupted dups entries of a PG Log can be removed by off-line and on-line trimming

Previously, trimming of PG log dups entries could be prevented during the low-level PG split operation, which is used by the PG autoscaler with far higher frequency than by a human operator. Stalling the trimming of dups resulted in significant memory growth of PG log, leading to OSD crashes as it ran out of memory. Restarting an OSD did not solve the problem as the PG log is stored on disk and reloaded to RAM on startup.

With this fix, both off-line, using the `ceph-objectstore-tool` command, and on-line, within OSD, trimming can remove corrupted dups entries of a PG log that jammed the on-line trimming machinery and were responsible for the memory growth. A debug improvement is implemented that prints the number of dups entries to the OSD's log to help future investigations.
Doc Type If docs needed, set a value Bug Fix
Akash Raj 2022-10-04 09:08:43 UTC Flags needinfo?(rzarzyns)
Docs Contact akraj
Doc Text Corrupted dups entries of a PG Log can be removed by off-line and on-line trimming

Previously, trimming of PG log dups entries could be prevented during the low-level PG split operation, which is used by the PG autoscaler with far higher frequency than by a human operator. Stalling the trimming of dups resulted in significant memory growth of PG log, leading to OSD crashes as it ran out of memory. Restarting an OSD did not solve the problem as the PG log is stored on disk and reloaded to RAM on startup.

With this fix, both off-line, using the `ceph-objectstore-tool` command, and on-line, within OSD, trimming can remove corrupted dups entries of a PG log that jammed the on-line trimming machinery and were responsible for the memory growth. A debug improvement is implemented that prints the number of dups entries to the OSD's log to help future investigations.
.Corrupted dups entries of a PG Log can be removed by off-line and on-line trimming

Previously, trimming of PG log dups entries could be prevented during the low-level PG split operation, which is used by the PG autoscaler with far higher frequency than by a human operator. Stalling the trimming of dups resulted in significant memory growth of PG log, leading to OSD crashes as it ran out of memory. Restarting an OSD did not solve the problem as the PG log is stored on disk and reloaded to RAM on startup.

With this fix, both off-line, using the `ceph-objectstore-tool` command, and on-line, within OSD, trimming can remove corrupted dups entries of a PG log that jammed the on-line trimming machinery and were responsible for the memory growth. A debug improvement is implemented that prints the number of dups entries to the OSD's log to help future investigations.
Radoslaw Zarzynski 2022-10-04 10:01:17 UTC Flags needinfo?(rzarzyns)
Ranjini M N 2022-10-04 10:34:45 UTC Blocks 2126049
Red Hat Bugzilla 2022-12-31 19:09:43 UTC CC skanta
Red Hat Bugzilla 2022-12-31 19:13:21 UTC CC amathuri
Red Hat Bugzilla 2022-12-31 19:32:28 UTC QA Contact pdhiran
CC pdhiran
Red Hat Bugzilla 2022-12-31 19:59:54 UTC CC sseshasa
Red Hat Bugzilla 2022-12-31 22:32:24 UTC CC nmordech
Red Hat Bugzilla 2022-12-31 22:43:27 UTC CC rfriedma
Red Hat Bugzilla 2022-12-31 23:43:28 UTC CC rzarzyns
Assignee rzarzyns nojha
Red Hat Bugzilla 2022-12-31 23:45:46 UTC CC akupczyk
Red Hat Bugzilla 2023-01-01 05:35:14 UTC CC ksirivad
Red Hat Bugzilla 2023-01-01 05:39:32 UTC CC tserlin
Red Hat Bugzilla 2023-01-01 06:27:04 UTC CC lflores
Red Hat Bugzilla 2023-01-01 06:28:55 UTC CC choffman
Red Hat Bugzilla 2023-01-01 08:38:24 UTC Assignee nojha nobody
CC nojha
Red Hat Bugzilla 2023-01-01 08:39:23 UTC CC pdhange
Red Hat Bugzilla 2023-01-01 08:48:02 UTC CC vereddy
Red Hat Bugzilla 2023-01-01 08:49:45 UTC CC vumrao
Alasdair Kergon 2023-01-04 04:40:45 UTC CC akupczyk
Alasdair Kergon 2023-01-04 04:43:34 UTC CC amathuri
Alasdair Kergon 2023-01-04 04:51:21 UTC Assignee nobody rzarzyns
Alasdair Kergon 2023-01-04 04:56:54 UTC QA Contact pdhiran
Alasdair Kergon 2023-01-04 05:08:58 UTC CC ksirivad
Alasdair Kergon 2023-01-04 05:10:58 UTC CC lflores
Alasdair Kergon 2023-01-04 05:21:30 UTC CC nmordech
Alasdair Kergon 2023-01-04 05:21:38 UTC CC nojha
Alasdair Kergon 2023-01-04 05:28:18 UTC CC pdhange
Alasdair Kergon 2023-01-04 05:30:13 UTC CC pdhiran
Alasdair Kergon 2023-01-04 05:34:52 UTC CC rfriedma
Alasdair Kergon 2023-01-04 05:37:37 UTC CC rzarzyns
Alasdair Kergon 2023-01-04 05:41:45 UTC CC skanta
Alasdair Kergon 2023-01-04 05:59:30 UTC CC vumrao
Alasdair Kergon 2023-01-04 06:13:47 UTC CC choffman
Alasdair Kergon 2023-01-04 06:56:31 UTC CC sseshasa
Alasdair Kergon 2023-01-04 06:59:12 UTC CC vereddy
Red Hat Bugzilla 2023-01-09 08:30:41 UTC CC ceph-eng-bugs
Alasdair Kergon 2023-01-09 19:43:36 UTC CC ceph-eng-bugs
errata-xmlrpc 2023-01-11 17:41:19 UTC Status VERIFIED CLOSED
Resolution --- ERRATA
Last Closed 2023-01-11 17:41:19 UTC
errata-xmlrpc 2023-01-11 17:42:03 UTC Link ID Red Hat Product Errata RHSA-2023:0076

Back to bug 2119853