2130081 – ceph-mds issues during upgrade from 5.1 to 5.2

Bug 2130081 - ceph-mds issues during upgrade from 5.1 to 5.2

Summary: ceph-mds issues during upgrade from 5.1 to 5.2

Keywords:
Status:	CLOSED ERRATA
Alias:	None
Product:	Red Hat Ceph Storage
Classification:	Red Hat Storage
Component:	CephFS
Sub Component:
Version:	5.2
Hardware:	x86_64
OS:	Linux
Priority:	unspecified
Severity:	medium
Target Milestone:	---
Target Release:	6.0
Assignee:	Venky Shankar
QA Contact:	Amarnath
Docs Contact:	Eliska
URL:
Whiteboard:
Depends On:	2126163
Blocks:	2126050
TreeView+	depends on / blocked

Reported:	2022-09-27 06:37 UTC by Venky Shankar
Modified:	2023-03-20 18:58 UTC (History)
CC List:	11 users (show)
Fixed In Version:	ceph-17.2.3-46.el9cp
Doc Type:	Bug Fix
Doc Text:	.The `ceph-mds` daemon no longer crashes during the upgrade Previously, the Ceph Metadata Server daemons (`ceph-mds`) would crash during an upgrade due to an incorrect assumption in the Metadata Servers when recovering inodes. It caused `ceph-mds` to hit an assert during an upgrade. With this fix, the `ceph-mds` makes correct assumptions during inode recovery and the `ceph-mds` no longer crashes during an upgrade.
Clone Of:	2126163
Environment:
Last Closed:	2023-03-20 18:58:17 UTC
Embargoed:
Dependent Products:

Attachments	(Terms of Use)

Links
System	ID	Priority	Status	Summary	Last Updated
Ceph Project Bug Tracker	56016	None	None	None	2022-09-27 06:37:00 UTC
Red Hat Issue Tracker	RHCEPH-5355	None	None	None	2022-09-27 07:09:26 UTC
Red Hat Product Errata	RHBA-2023:1360	None	None	None	2023-03-20 18:58:54 UTC

Comment 2 Venky Shankar 2022-09-27 06:43:11 UTC

This issues is could be hit when upgrading to 6.0 -- so we need the fix.

Comment 11 Amarnath 2022-10-20 10:59:35 UTC

Upgraded From ceph version 16.2.7-126.el8cp (fe0af61d104d48cb9d116cde6e593b5fc8c197e4) pacific (stable) to 
ceph version 17.2.3-55.el9cp (e57fd6f8008c472ddf2115482308a726e8f4fc0b) quincy (stable)

Did not observe any crashes 

[root@ceph-amk-up-2-1numcu-node7 cephfs_fuseuse]# ceph orch ps
NAME                                             HOST                                  PORTS   STATUS         REFRESHED  AGE  MEM USE  MEM LIM  VERSION           IMAGE ID      CONTAINER ID  
mds.cephfs.ceph-amk-up-2-1numcu-node5.nbxaaa     ceph-amk-up-2-1numcu-node5                    running (43m)     9m ago  43m    27.3M        -  16.2.7-126.el8cp  ceb76cd098f2  67f7cd468197  
mds.cephfs.ceph-amk-up-2-1numcu-node6.tzvnxq     ceph-amk-up-2-1numcu-node6                    running (43m)    15s ago  43m    14.9M        -  16.2.7-126.el8cp  ceb76cd098f2  c866e5c53b46  
mgr.ceph-amk-up-2-1numcu-node1-installer.uqatnl  ceph-amk-up-2-1numcu-node1-installer  *:9283  running (53m)    77s ago  53m     454M        -  16.2.7-126.el8cp  ceb76cd098f2  c3ccab90d6d5  
mgr.ceph-amk-up-2-1numcu-node2.zbuvya            ceph-amk-up-2-1numcu-node2            *:8443  running (49m)     6m ago  49m     394M        -  16.2.7-126.el8cp  ceb76cd098f2  aa578f525e77  
mon.ceph-amk-up-2-1numcu-node1-installer         ceph-amk-up-2-1numcu-node1-installer          running (53m)    77s ago  54m     222M    2048M  16.2.7-126.el8cp  ceb76cd098f2  85c86f998013  
mon.ceph-amk-up-2-1numcu-node2                   ceph-amk-up-2-1numcu-node2                    running (48m)     6m ago  48m     161M    2048M  16.2.7-126.el8cp  ceb76cd098f2  a75140af2626  
mon.ceph-amk-up-2-1numcu-node3                   ceph-amk-up-2-1numcu-node3                    running (48m)     6m ago  48m     165M    2048M  16.2.7-126.el8cp  ceb76cd098f2  2960fa30fab5  
osd.0                                            ceph-amk-up-2-1numcu-node5                    running (43m)     9m ago  43m    60.4M    4096M  16.2.7-126.el8cp  ceb76cd098f2  24f11ea96e6f  
osd.1                                            ceph-amk-up-2-1numcu-node3                    running (43m)     6m ago  43m    58.0M    4096M  16.2.7-126.el8cp  ceb76cd098f2  11181dae889c  
osd.10                                           ceph-amk-up-2-1numcu-node3                    running (43m)     6m ago  43m    64.0M    4096M  16.2.7-126.el8cp  ceb76cd098f2  9675ff282473  
osd.11                                           ceph-amk-up-2-1numcu-node2                    running (43m)     6m ago  43m    58.9M    4096M  16.2.7-126.el8cp  ceb76cd098f2  51c629f58083  
osd.2                                            ceph-amk-up-2-1numcu-node2                    running (43m)     6m ago  43m    61.5M    4096M  16.2.7-126.el8cp  ceb76cd098f2  8897042b23cf  
osd.3                                            ceph-amk-up-2-1numcu-node5                    running (43m)     9m ago  43m    57.5M    4096M  16.2.7-126.el8cp  ceb76cd098f2  fcf8eab1ee2d  
osd.4                                            ceph-amk-up-2-1numcu-node3                    running (43m)     6m ago  43m    60.8M    4096M  16.2.7-126.el8cp  ceb76cd098f2  5583d3bc9beb  
osd.5                                            ceph-amk-up-2-1numcu-node2                    running (43m)     6m ago  43m    58.4M    4096M  16.2.7-126.el8cp  ceb76cd098f2  118127f8dd0d  
osd.6                                            ceph-amk-up-2-1numcu-node5                    running (43m)     9m ago  43m    58.9M    4096M  16.2.7-126.el8cp  ceb76cd098f2  4ece78ec3631  
osd.7                                            ceph-amk-up-2-1numcu-node3                    running (43m)     6m ago  43m    62.6M    4096M  16.2.7-126.el8cp  ceb76cd098f2  7b1a50760003  
osd.8                                            ceph-amk-up-2-1numcu-node2                    running (43m)     6m ago  43m    59.9M    4096M  16.2.7-126.el8cp  ceb76cd098f2  99a5bd12d2df  
osd.9                                            ceph-amk-up-2-1numcu-node5                    running (43m)     9m ago  43m    57.5M    4096M  16.2.7-126.el8cp  ceb76cd098f2  1f818ff1ff95  
[root@ceph-amk-up-2-1numcu-node7 cephfs_fuseuse]# ceph crash ls
[root@ceph-amk-up-2-1numcu-node7 cephfs_fuseuse]# 
[root@ceph-amk-up-2-1numcu-node7 cephfs_fuseuse]# 
[root@ceph-amk-up-2-1numcu-node7 cephfs_fuseuse]# ceph orch host ls
HOST                                  ADDR          LABELS                    STATUS  
ceph-amk-up-2-1numcu-node1-installer  10.0.210.14   _admin installer mgr mon          
ceph-amk-up-2-1numcu-node2            10.0.210.173  osd mgr mon                       
ceph-amk-up-2-1numcu-node3            10.0.209.130  mon osd                           
ceph-amk-up-2-1numcu-node4            10.0.208.162  nfs                               
ceph-amk-up-2-1numcu-node5            10.0.208.218  osd mds                           
ceph-amk-up-2-1numcu-node6            10.0.209.124  nfs mds  


After Upgrade : 

[ceph: root@ceph-amk-up-2-1numcu-node1-installer /]# ceph orch ps
NAME                                             HOST                                  PORTS   STATUS        REFRESHED  AGE  MEM USE  MEM LIM  VERSION          IMAGE ID      CONTAINER ID  
mds.cephfs.ceph-amk-up-2-1numcu-node5.nbxaaa     ceph-amk-up-2-1numcu-node5                    running (4m)     2m ago  66m    34.6M        -  17.2.3-55.el9cp  4b5fde9e28ca  57ef21b1889e  
mds.cephfs.ceph-amk-up-2-1numcu-node6.tzvnxq     ceph-amk-up-2-1numcu-node6                    running (2m)     2m ago  66m    13.7M        -  17.2.3-55.el9cp  4b5fde9e28ca  a2295fc6cfe3  
mgr.ceph-amk-up-2-1numcu-node1-installer.uqatnl  ceph-amk-up-2-1numcu-node1-installer  *:8443  running (8m)     7m ago  77m     426M        -  17.2.3-55.el9cp  4b5fde9e28ca  daa665c63ed4  
mgr.ceph-amk-up-2-1numcu-node2.zbuvya            ceph-amk-up-2-1numcu-node2            *:8443  running (8m)     4m ago  73m     395M        -  17.2.3-55.el9cp  4b5fde9e28ca  14a2673124cf  
mon.ceph-amk-up-2-1numcu-node1-installer         ceph-amk-up-2-1numcu-node1-installer          running (8m)     7m ago  77m    30.9M    2048M  17.2.3-55.el9cp  4b5fde9e28ca  7d02e85f93f7  
mon.ceph-amk-up-2-1numcu-node2                   ceph-amk-up-2-1numcu-node2                    running (7m)     4m ago  72m    86.2M    2048M  17.2.3-55.el9cp  4b5fde9e28ca  616bdc72c008  
mon.ceph-amk-up-2-1numcu-node3                   ceph-amk-up-2-1numcu-node3                    running (7m)     4m ago  71m    80.3M    2048M  17.2.3-55.el9cp  4b5fde9e28ca  96093698beac  
osd.0                                            ceph-amk-up-2-1numcu-node5                    running (4m)     2m ago  67m     378M    4096M  17.2.3-55.el9cp  4b5fde9e28ca  277b246330e8  
osd.1                                            ceph-amk-up-2-1numcu-node3                    running (5m)     4m ago  67m     227M    4096M  17.2.3-55.el9cp  4b5fde9e28ca  0cc352dd0aa7  
osd.2                                            ceph-amk-up-2-1numcu-node2                    running (6m)     4m ago  67m     293M    4096M  17.2.3-55.el9cp  4b5fde9e28ca  8bf0beeaf6b1  
osd.3                                            ceph-amk-up-2-1numcu-node5                    running (4m)     2m ago  67m     284M    4096M  17.2.3-55.el9cp  4b5fde9e28ca  a0cf9f09b29b  
osd.4                                            ceph-amk-up-2-1numcu-node3                    running (5m)     4m ago  67m     241M    4096M  17.2.3-55.el9cp  4b5fde9e28ca  6e397c9253a1  
osd.5                                            ceph-amk-up-2-1numcu-node2                    running (6m)     4m ago  67m     304M    4096M  17.2.3-55.el9cp  4b5fde9e28ca  51dbaeeeb89f  
osd.6                                            ceph-amk-up-2-1numcu-node5                    running (4m)     2m ago  67m     249M    4096M  17.2.3-55.el9cp  4b5fde9e28ca  bf70ff1c63bf  
osd.7                                            ceph-amk-up-2-1numcu-node3                    running (5m)     4m ago  67m     312M    4096M  17.2.3-55.el9cp  4b5fde9e28ca  df0492ef47c2  
osd.8                                            ceph-amk-up-2-1numcu-node2                    running (6m)     4m ago  67m     295M    4096M  17.2.3-55.el9cp  4b5fde9e28ca  19b06d57d22c  
osd.9                                            ceph-amk-up-2-1numcu-node5                    running (4m)     2m ago  67m     186M    4096M  17.2.3-55.el9cp  4b5fde9e28ca  b01c0cc34232  
osd.10                                           ceph-amk-up-2-1numcu-node3                    running (5m)     4m ago  67m     234M    4096M  17.2.3-55.el9cp  4b5fde9e28ca  1f296352c40c  
osd.11                                           ceph-amk-up-2-1numcu-node2                    running (6m)     4m ago  67m     279M    4096M  17.2.3-55.el9cp  4b5fde9e28ca  ae2028d35d9d  
[ceph: root@ceph-amk-up-2-1numcu-node1-installer /]# ceph orch host ls
HOST                                  ADDR          LABELS                    STATUS  
ceph-amk-up-2-1numcu-node1-installer  10.0.210.14   _admin installer mgr mon          
ceph-amk-up-2-1numcu-node2            10.0.210.173  osd mgr mon                       
ceph-amk-up-2-1numcu-node3            10.0.209.130  mon osd                           
ceph-amk-up-2-1numcu-node4            10.0.208.162  nfs                               
ceph-amk-up-2-1numcu-node5            10.0.208.218  osd mds                           
ceph-amk-up-2-1numcu-node6            10.0.209.124  nfs mds                           
6 hosts in cluster
[ceph: root@ceph-amk-up-2-1numcu-node1-installer /]# 

[ceph: root@ceph-amk-up-2-1numcu-node1-installer /]# ceph crash ls

Comment 21 errata-xmlrpc 2023-03-20 18:58:17 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Red Hat Ceph Storage 6.0 Bug Fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2023:1360

Note You need to log in before you can comment on or make changes to this bug.