Bug 2022133

Must gather located @

/cases/03078164

drwxrwxrwx+ 3 yank yank 58 Nov 10 17:45 0010-must-gather-ocs-10_Nov_2021.tar.gz

Hello,

Is there a workaround or a way for the cu to clear the warning ?

customer has been patient thus far, but i need assistance getting them out of this state please.

(In reply to khover from comment #6)
> Hello,
> 
> Is there a workaround or a way for the cu to clear the warning ?
> 
> customer has been patient thus far, but i need assistance getting them out
> of this state please.

Mhack, brought this one to our attention, this could be a transient thing when the cluster had this pg deep-scrubbed the number of open files exceeded the currently configured threshold for option `osd_deep_scrub_large_omap_object_key_threshold` which defaults to 200K. You can handle this situation with the following:

1. Run `ceph health detail` find the pg which has large omap object, deep-scrub the pg and if the open files are less than during the deep-scrub this pg will go away from large omap object warning
2. If you think this cluster is suspected to have these many openfiles like step-1 did not help and want more permanent solution change the default which is higher than 200K for this optiom.

Hi Vikhyat,

Thanks for taking a look at this.

The problem here is a couple of things.

First, the steps in KCS created for this issue in OCS is not working in this particular case.

https://access.redhat.com/solutions/5510261

$ grep -Hrni "Large omap object found" current.log
grep: current.log: No such file or directory

----------
grep against entire must gather 

$ grep -Hrni "Large omap object found" 
ceph/must_gather_commands/ceph_health_detail:4:    Search the cluster log for 'Large omap object found' for more details.
ceph/must_gather_commands/ceph_report:19:                        "message": "Search the cluster log for 'Large omap object found' for more details."
ceph/must_gather_commands_json_output/ceph_health_detail_--format_json-pretty:14:                    "message": "Search the cluster log for 'Large omap object found' for more details."
ceph/must_gather_commands_json_output/ceph_report_--format_json-pretty:20:                        "message": "Search the cluster log for 'Large omap object found' for more details."




Second, the health detail does not point to any pg.

HEALTH_WARN 1 large omap objects
LARGE_OMAP_OBJECTS 1 large omap objects
    1 large objects found in pool 'ocs-storagecluster-cephfilesystem-metadata'
    Search the cluster log for 'Large omap object found' for more details.

(In reply to khover from comment #10)
> Hi Vikhyat,
> 
> Thanks for taking a look at this.
> 
> The problem here is a couple of things.
> 
> First, the steps in KCS created for this issue in OCS is not working in this
> particular case.
> 
> https://access.redhat.com/solutions/5510261
> 
> $ grep -Hrni "Large omap object found" current.log
> grep: current.log: No such file or directory
> 
> ----------
> grep against entire must gather 
> 
> $ grep -Hrni "Large omap object found" 
> ceph/must_gather_commands/ceph_health_detail:4:    Search the cluster log
> for 'Large omap object found' for more details.
> ceph/must_gather_commands/ceph_report:19:                        "message":
> "Search the cluster log for 'Large omap object found' for more details."
> ceph/must_gather_commands_json_output/ceph_health_detail_--format_json-
> pretty:14:                    "message": "Search the cluster log for 'Large
> omap object found' for more details."
> ceph/must_gather_commands_json_output/ceph_report_--format_json-pretty:20:  
> "message": "Search the cluster log for 'Large omap object found' for more
> details."
> 
> 
> 
> 
> Second, the health detail does not point to any pg.
> 
> HEALTH_WARN 1 large omap objects
> LARGE_OMAP_OBJECTS 1 large omap objects
>     1 large objects found in pool
> 'ocs-storagecluster-cephfilesystem-metadata'
>     Search the cluster log for 'Large omap object found' for more details.

This looks interesting - I am not sure how log rotation works in ODF and why cluster logs of ceph(ceph.log) are named here as current.log but if you are not able to find the large omap warning and you want to first go with step 1 the best thing is to scrub all the pgs in this pool[1] 'ocs-storagecluster-cephfilesystem-metadata'.

[1] # ceph osd pool deep-scrub ocs-storagecluster-cephfilesystem-metadata

Wait for all deep-scrub's to complete and then you can check the logs. If the openfiles would be less than the threshold the warning will go away and nothing would be logged in the logs. But if the log will have this warning then it means that Warning will remain in the cluster and you need to increase the threshold and then after increasing the threshold you can deep-scrub the individual pg[2] and the warning should go away.

[2] # ceph pg deep-scrub <pg-id>

Customer env has been deep scrubbing since 11/18

Customer update concerning the same 11/29

----------------------------------------

We are not able to identify the process is stuck or in-progress.


Need more way to check the progress of the pgs which are in scrubbing.


Since the OCP/OCS 4.6 is declared as EOL, we need to upgrade the cluster to 4.7/4.8 at the earliest.


$ oc rsh -n openshift-storage oc get pods -n openshift-storage |grep ceph-tools |awk '{print $1}'


sh-4.4# ceph health detail
HEALTH_WARN 1 large omap objects; 1 pools have many more objects per pg than average
LARGE_OMAP_OBJECTS 1 large omap objects
  1 large objects found in pool 'ocs-storagecluster-cephfilesystem-metadata'
  Search the cluster log for 'Large omap object found' for more details.
MANY_OBJECTS_PER_PG 1 pools have many more objects per pg than average
  pool ocs-storagecluster-cephfilesystem-metadata objects per pg (145027) is more than 10.0455 times cluster average (14437)


sh-4.4# ceph -s
 cluster:
  id:   bddc3cb4-1e60-4eba-8c8e-ebd286952888
  health: HEALTH_WARN
      1 large omap objects
      1 pools have many more objects per pg than average
 services:
  mon: 3 daemons, quorum a,c,d (age 3w)
  mgr: a(active, since 3d)
  mds: ocs-storagecluster-cephfilesystem:1 {0=ocs-storagecluster-cephfilesystem-b=up:active} 1 up:standby-replay
  osd: 12 osds: 12 up (since 3w), 12 in (since 12M)
  rgw: 2 daemons active (ocs.storagecluster.cephobjectstore.a, ocs.storagecluster.cephobjectstore.b)
 data:
  pools:  10 pools, 752 pgs
  objects: 10.86M objects, 4.6 TiB
  usage:  14 TiB used, 10 TiB / 24 TiB avail
  pgs:   750 active+clean
       2  active+clean+scrubbing+deep
 io:
  client:  124 MiB/s rd, 20 MiB/s wr, 4.36k op/s rd, 466 op/s wr

(In reply to khover from comment #13)
> Customer env has been deep scrubbing since 11/18
> 
> Customer update concerning the same 11/29
> 
> ----------------------------------------
> 
> We are not able to identify the process is stuck or in-progress.
> 
> 
> Need more way to check the progress of the pgs which are in scrubbing.
> 
> 
> Since the OCP/OCS 4.6 is declared as EOL, we need to upgrade the cluster to
> 4.7/4.8 at the earliest.
> 
> 
> $ oc rsh -n openshift-storage oc get pods -n openshift-storage |grep
> ceph-tools |awk '{print $1}'
> 
> 
> sh-4.4# ceph health detail
> HEALTH_WARN 1 large omap objects; 1 pools have many more objects per pg than
> average
> LARGE_OMAP_OBJECTS 1 large omap objects
>   1 large objects found in pool 'ocs-storagecluster-cephfilesystem-metadata'
>   Search the cluster log for 'Large omap object found' for more details.
> MANY_OBJECTS_PER_PG 1 pools have many more objects per pg than average
>   pool ocs-storagecluster-cephfilesystem-metadata objects per pg (145027) is
> more than 10.0455 times cluster average (14437)
> 
> 
> sh-4.4# ceph -s
>  cluster:
>   id:   bddc3cb4-1e60-4eba-8c8e-ebd286952888
>   health: HEALTH_WARN
>       1 large omap objects
>       1 pools have many more objects per pg than average
>  services:
>   mon: 3 daemons, quorum a,c,d (age 3w)
>   mgr: a(active, since 3d)
>   mds: ocs-storagecluster-cephfilesystem:1
> {0=ocs-storagecluster-cephfilesystem-b=up:active} 1 up:standby-replay
>   osd: 12 osds: 12 up (since 3w), 12 in (since 12M)
>   rgw: 2 daemons active (ocs.storagecluster.cephobjectstore.a,
> ocs.storagecluster.cephobjectstore.b)
>  data:
>   pools:  10 pools, 752 pgs
>   objects: 10.86M objects, 4.6 TiB
>   usage:  14 TiB used, 10 TiB / 24 TiB avail
>   pgs:   750 active+clean
>        2  active+clean+scrubbing+deep
>  io:
>   client:  124 MiB/s rd, 20 MiB/s wr, 4.36k op/s rd, 466 op/s wr

If you check comment#11 the next step was already mentioned in the comment and it is checking cluster logs(ceph.log) in ODF they could be in monitor logs or/and current.log please verify that and if they are not log-rotated by now you can catch this warning in the logs. 

Coming to checking the pgs when they were last deep-scrub simple run `ceph pg dump` command output. So the next steps should be:

1. ceph df <--- identify the pool id for this pool
2. ceph pg dump <--- identify the last deep-scrub time stamp for the given pool pgs
3. checking the cluster logs for large omap warning 

Please also provide the `ceph versions` command output.

(In reply to Vikhyat Umrao from comment #14)
> (In reply to khover from comment #13)
> > Customer env has been deep scrubbing since 11/18
> > 
> > Customer update concerning the same 11/29
> > 
> > ----------------------------------------
> > 
> > We are not able to identify the process is stuck or in-progress.
> > 
> > 
> > Need more way to check the progress of the pgs which are in scrubbing.
> > 
> > 
> > Since the OCP/OCS 4.6 is declared as EOL, we need to upgrade the cluster to
> > 4.7/4.8 at the earliest.
> > 
> > 
> > $ oc rsh -n openshift-storage oc get pods -n openshift-storage |grep
> > ceph-tools |awk '{print $1}'
> > 
> > 
> > sh-4.4# ceph health detail
> > HEALTH_WARN 1 large omap objects; 1 pools have many more objects per pg than
> > average
> > LARGE_OMAP_OBJECTS 1 large omap objects
> >   1 large objects found in pool 'ocs-storagecluster-cephfilesystem-metadata'
> >   Search the cluster log for 'Large omap object found' for more details.
> > MANY_OBJECTS_PER_PG 1 pools have many more objects per pg than average
> >   pool ocs-storagecluster-cephfilesystem-metadata objects per pg (145027) is
> > more than 10.0455 times cluster average (14437)
> > 
> > 
> > sh-4.4# ceph -s
> >  cluster:
> >   id:   bddc3cb4-1e60-4eba-8c8e-ebd286952888
> >   health: HEALTH_WARN
> >       1 large omap objects
> >       1 pools have many more objects per pg than average
> >  services:
> >   mon: 3 daemons, quorum a,c,d (age 3w)
> >   mgr: a(active, since 3d)
> >   mds: ocs-storagecluster-cephfilesystem:1
> > {0=ocs-storagecluster-cephfilesystem-b=up:active} 1 up:standby-replay
> >   osd: 12 osds: 12 up (since 3w), 12 in (since 12M)
> >   rgw: 2 daemons active (ocs.storagecluster.cephobjectstore.a,
> > ocs.storagecluster.cephobjectstore.b)
> >  data:
> >   pools:  10 pools, 752 pgs
> >   objects: 10.86M objects, 4.6 TiB
> >   usage:  14 TiB used, 10 TiB / 24 TiB avail
> >   pgs:   750 active+clean
> >        2  active+clean+scrubbing+deep
> >  io:
> >   client:  124 MiB/s rd, 20 MiB/s wr, 4.36k op/s rd, 466 op/s wr
> 
> If you check comment#11 the next step was already mentioned in the comment
> and it is checking cluster logs(ceph.log) in ODF they could be in monitor
> logs or/and current.log please verify that and if they are not log-rotated
> by now you can catch this warning in the logs. 
> 
> Coming to checking the pgs when they were last deep-scrub simple run `ceph
> pg dump` command output. So the next steps should be:
> 
> 1. ceph df <--- identify the pool id for this pool
> 2. ceph pg dump <--- identify the last deep-scrub time stamp for the given
> pool pgs
> 3. checking the cluster logs for large omap warning 
> 
> Please also provide the `ceph versions` command output.

This seems odd, is there some sort of deep-scrub loop happening here ? 
Deep scrub kicked in on 11/18 

2.13d      2236                  0        0         0       0 8863154194         606         17 3033     3033                active+clean 2021-12-05 06:59:14.373380  22432'44409844 22432:212873637  [4,2,0]          4  [4,2,0]              4  22136'44337021 2021-12-04 07:46:33.977877  20429'43189019 2021-11-27 << LAST_DEEP_SCRUB DEEP_SCRUB_

2.104      2269                  0        0         0       0 9138610194         248         11 3083     3083 active+clean+scrubbing+deep 2021-12-05 07:47:21.721343  22432'41569927 22432:198740412  [5,2,9]          5  [5,2,9]              5  21795'41479245 2021-12-03 21:10:21.134492  20661'40682279 2021-11-28

no info in the current.log or mon logs uploaded identifying the pg related to the large omap object

0060-rook-ceph-mon-a-8ffdcdd55-8q4xb_logs.txt

cluster 2021-12-04 16:43:17.905167 mon.a (mon.0) 9 : cluster [WRN] Health detail: HEALTH_WARN 1 large omap objects; 1 pools have many more objects per pg than average
cluster 2021-12-04 16:43:17.905208 mon.a (mon.0) 10 : cluster [WRN] LARGE_OMAP_OBJECTS 1 large omap objects
cluster 2021-12-04 16:43:17.905219 mon.a (mon.0) 11 : cluster [WRN]     1 large objects found in pool 'ocs-storagecluster-cephfilesystem-metadata'
cluster 2021-12-04 16:43:17.905229 mon.a (mon.0) 12 : cluster [WRN]     Search the cluster log for 'Large omap object found' for more details.
cluster 2021-12-04 16:43:17.905240 mon.a (mon.0) 13 : cluster [WRN] MANY_OBJECTS_PER_PG 1 pools have many more objects per pg than average
cluster 2021-12-04 16:43:17.905249 mon.a (mon.0) 14 : cluster [WRN]     pool ocs-storagecluster-cephfilesystem-metadata objects per pg (148094) is more than 10.0382 times cluster average (14753)


debug 2021-12-05 07:50:00.003 7f3989ac2700  0 log_channel(cluster) log [WRN] : overall HEALTH_WARN 1 large omap objects; 1 pools have many more objects per pg than average
cluster 2021-12-05 07:50:00.005261 mon.a (mon.0) 17977 : cluster [WRN] overall HEALTH_WARN 1 large omap objects; 1 pools have many more objects per pg than average
audit 2021-12-05 07:50:00.415242 mon.d (mon.2) 47453 : audit [DBG] from='admin socket' entity='admin socket' cmd='mon_status' args=[]: dispatch
audit 2021-12-05 07:50:00.415549 mon.d (mon.2) 47454 : audit [DBG] from='admin socket' entity='admin socket' cmd=mon_status args=[]: finished




sh-4.4# ceph versions
{
    "mon": {
        "ceph version 14.2.11-199.el8cp (f5470cbfb5a4dac5925284cef1215f3e4e191a38) nautilus (stable)": 3
    },
    "mgr": {
        "ceph version 14.2.11-199.el8cp (f5470cbfb5a4dac5925284cef1215f3e4e191a38) nautilus (stable)": 1
    },
    "osd": {
        "ceph version 14.2.11-199.el8cp (f5470cbfb5a4dac5925284cef1215f3e4e191a38) nautilus (stable)": 12
    },
    "mds": {
        "ceph version 14.2.11-199.el8cp (f5470cbfb5a4dac5925284cef1215f3e4e191a38) nautilus (stable)": 2
    },
    "rgw": {
        "ceph version 14.2.11-199.el8cp (f5470cbfb5a4dac5925284cef1215f3e4e191a38) nautilus (stable)": 2
    },
    "overall": {
        "ceph version 14.2.11-199.el8cp (f5470cbfb5a4dac5925284cef1215f3e4e191a38) nautilus (stable)": 20


statement from cu:

This is running for weeks now and we have an urgent upgrade activity is pending for OCP & OCS platforms to be completed.