Created attachment 1965611 [details]
mgr pod log prior restart

Description of problem:

We noticed that mgr pod will restart while running tests.

Version-Release number of selected component (if applicable):

ceph versions
{
"mon": {
"ceph version 17.2.6-47.el9cp (6add4f24d1eff88e1db808ecdc16fd5b2db96dd4) quincy (stable)": 3
},
"mgr": {
"ceph version 17.2.6-47.el9cp (6add4f24d1eff88e1db808ecdc16fd5b2db96dd4) quincy (stable)": 1
},
"osd": {
"ceph version 17.2.6-47.el9cp (6add4f24d1eff88e1db808ecdc16fd5b2db96dd4) quincy (stable)": 21
},
"mds": {
"ceph version 17.2.6-47.el9cp (6add4f24d1eff88e1db808ecdc16fd5b2db96dd4) quincy (stable)": 2
},
"rbd-mirror": {
"ceph version 17.2.6-47.el9cp (6add4f24d1eff88e1db808ecdc16fd5b2db96dd4) quincy (stable)": 1
},
"rgw": {
"ceph version 17.2.6-47.el9cp (6add4f24d1eff88e1db808ecdc16fd5b2db96dd4) quincy (stable)": 1
},
"overall": {
"ceph version 17.2.6-47.el9cp (6add4f24d1eff88e1db808ecdc16fd5b2db96dd4) quincy (stable)": 29
}
}



How reproducible:
The issue was visible while running test with mixed rw ( fio randrw ) 70% write, 30% read


Steps to Reproduce:
Our environment was ODF DR ( rbd mirroring ), below are steps we followed
1. create ODF DR with above ceph version
2. on primary cluster issue test ( randrw , 70% write, 30% read ) 
3. let test to run for 12h or so

Note: This issue was visible once up to now ( but we also test this version for couple days ) , so it might happen that reproducer does not work. 

Actual results:
mgr pods restarts. Check logs

Expected results:
mgr pod not to restart 

Additional info:

Attached logs