Description of problem: PGs not distributed evenly by Balancer in a heterogeneous cluster resulting in OSD full. We have OSDs based of ~7.3TB HDD OSDs and ~900G SSD OSDs. The weights on these OSDs is also assigned accordingly on the OSDs. During normal writes, ( Without recovery/ backfill involved ), the average usage on the HDD disks was 25 - 35 %, the SSD disks barely had any data written onto them. Again this is due to the relative weight being so low. But during recovery/ backfill scenarios, We saw that all data (PGs) were placed on One OSD only, instead of all the OSDs on the hosts that node. It looks like Only 1 OSD from each host received all the data. All the hosts in the cluster had 1 OSD with 0.8 weight, whereas other OSDs had 7.2. And since the cluster was rebalancing, the balancer module was out of commission as well, and it was waiting for PGs to get clean. And since due to uneven distrubution of PGs on the OSDs, the Cluster balancer never ran and eventually ending with only 1 OSD from hosts got filled. In the below cluster, we saw that on hosts: ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS TYPE NAME -7 88.19592 - 88 TiB 36 TiB 36 TiB 53 GiB 638 GiB 52 TiB 41.14 1.00 - host mero006 4 hdd 7.27689 1.00000 7.3 TiB 6.9 TiB 6.8 TiB 2 KiB 62 GiB 428 GiB 94.26 2.29 112 up osd.4 -> Only OSD 4 got filled up on mero006 -10 88.19592 - 88 TiB 35 TiB 34 TiB 53 GiB 612 GiB 54 TiB 39.25 0.95 - host mero008 45 hdd 7.27689 1.00000 7.3 TiB 6.7 TiB 6.6 TiB 3 KiB 59 GiB 625 GiB 91.61 2.23 81 up osd.45 -> Only OSD 45 got filled up on mero008 -3 88.19794 - 88 TiB 36 TiB 36 TiB 50 GiB 639 GiB 52 TiB 41.18 1.00 - host mero010 13 hdd 7.27689 1.00000 7.3 TiB 5.3 TiB 5.3 TiB 1 KiB 35 GiB 1.9 TiB 73.34 1.78 56 up osd.13 -> Only OSD 13 got filled up on mero010 -13 88.19794 - 81 TiB 35 TiB 34 TiB 49 KiB 614 GiB 46 TiB 43.22 1.05 - host mero012 38 hdd 7.27739 1.00000 7.3 TiB 6.7 TiB 6.6 TiB 1 KiB 59 GiB 603 GiB 91.90 2.23 79 up osd.38 -> Only OSD 38 got filled up on mero012 # ceph balancer status { "active": true, "last_optimize_duration": "0:00:00.000307", "last_optimize_started": "Wed Aug 16 16:03:22 2023", "mode": "upmap", "no_optimization_needed": false, "optimize_result": "Some objects (0.001673) are degraded; try again later", "plans": [] } # ceph -s cluster: id: 08416350-2b97-11ee-a9a2-ac1f6b40d3fc health: HEALTH_ERR 1 failed cephadm daemon(s) failed to probe daemons or devices 2 backfillfull osd(s) 1 full osd(s) Low space hindering backfill (add storage if this doesn't resolve itself): 23 pgs backfill_toofull Degraded data redundancy: 2671691/1597277593 objects degraded (0.167%), 4 pgs degraded, 4 pgs undersized 341 pgs not deep-scrubbed in time 404 pgs not scrubbed in time 11 pool(s) full services: mon: 3 daemons, quorum mero006,mero010,mero008 (age 2w) mgr: mero008.dzmakn(active, since 2w), standbys: mero006.tapija, mero010.knvfse osd: 52 osds: 51 up (since 2w), 51 in (since 2w); 104 remapped pgs rgw: 14 daemons active (14 hosts, 1 zones) data: pools: 11 pools, 417 pgs objects: 148.83M objects, 76 TiB usage: 142 TiB used, 203 TiB / 346 TiB avail pgs: 2671691/1597277593 objects degraded (0.167%) 808758591/1597277593 objects misplaced (50.634%) 313 active+clean 80 active+remapped+backfill_wait 12 active+remapped+backfill_wait+backfill_toofull 7 active+remapped+backfill_toofull 4 active+undersized+degraded+remapped+backfill_toofull 1 active+remapped+backfilling io: recovery: 13 MiB/s, 18 objects/s progress: Global Recovery Event (2w) [=====================.......] (remaining: 6d) # ceph df detail --- RAW STORAGE --- CLASS SIZE AVAIL USED RAW USED %RAW USED hdd 342 TiB 200 TiB 142 TiB 142 TiB 41.50 ssd 3.5 TiB 3.2 TiB 276 GiB 276 GiB 7.70 TOTAL 346 TiB 203 TiB 142 TiB 142 TiB 41.16 --- POOLS --- POOL ID PGS STORED (DATA) (OMAP) OBJECTS USED (DATA) (OMAP) %USED MAX AVAIL QUOTA OBJECTS QUOTA BYTES DIRTY USED COMPR UNDER COMPR .mgr 1 1 15 MiB 15 MiB 0 B 5 44 MiB 44 MiB 0 B 0 891 GiB N/A N/A N/A 0 B 0 B .rgw.root 2 32 2.8 KiB 2.8 KiB 0 B 7 72 KiB 72 KiB 0 B 0 891 GiB N/A N/A N/A 0 B 0 B default.rgw.log 3 32 27 MiB 27 MiB 30 KiB 210 81 MiB 81 MiB 91 KiB 0 891 GiB N/A N/A N/A 0 B 0 B default.rgw.control 4 32 0 B 0 B 0 B 9 0 B 0 B 0 B 0 891 GiB N/A N/A N/A 0 B 0 B default.rgw.meta 5 32 9.2 KiB 4.3 KiB 4.9 KiB 20 239 KiB 224 KiB 15 KiB 0 891 GiB N/A N/A N/A 0 B 0 B ecpool-86 6 128 68 TiB 68 TiB 0 B 100.20M 119 TiB 119 TiB 0 B 97.86 1.5 TiB N/A N/A N/A 0 B 0 B default.rgw.buckets.index 7 32 39 GiB 0 B 39 GiB 1.59k 116 GiB 0 B 116 GiB 4.17 891 GiB N/A N/A N/A 0 B 0 B default.rgw.buckets.data 8 32 11 KiB 11 KiB 0 B 1 36 KiB 36 KiB 0 B 0 891 GiB N/A N/A N/A 0 B 0 B ecpool-22 9 32 2.4 TiB 2.4 TiB 0 B 40.00M 4.8 TiB 4.8 TiB 0 B 64.62 1.3 TiB N/A N/A N/A 0 B 0 B source-ecpool-22 10 32 8.0 TiB 8.0 TiB 0 B 8.63M 16 TiB 16 TiB 0 B 85.96 1.3 TiB N/A N/A N/A 0 B 0 B default.rgw.temporary.index 11 32 13 GiB 0 B 13 GiB 809 40 GiB 0 B 40 GiB 1.47 891 GiB N/A N/A N/A 0 B 0 B # ceph osd df tree ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS TYPE NAME -1 352.78772 - 346 TiB 142 TiB 140 TiB 156 GiB 2.4 TiB 203 TiB 41.15 1.00 - root default -7 88.19592 - 88 TiB 36 TiB 36 TiB 53 GiB 638 GiB 52 TiB 41.14 1.00 - host mero006 2 hdd 7.27689 1.00000 7.3 TiB 2.3 TiB 2.2 TiB 1 KiB 44 GiB 5.0 TiB 31.11 0.76 44 up osd.2 4 hdd 7.27689 1.00000 7.3 TiB 6.9 TiB 6.8 TiB 2 KiB 62 GiB 428 GiB 94.26 2.29 112 up osd.4 6 hdd 7.27689 1.00000 7.3 TiB 2.8 TiB 2.8 TiB 6 KiB 51 GiB 4.5 TiB 38.72 0.94 48 up osd.6 8 hdd 7.27689 1.00000 7.3 TiB 2.5 TiB 2.5 TiB 1 KiB 47 GiB 4.7 TiB 35.02 0.85 47 up osd.8 10 hdd 7.27689 1.00000 7.3 TiB 2.4 TiB 2.4 TiB 1 KiB 45 GiB 4.8 TiB 33.46 0.81 40 up osd.10 12 hdd 7.27689 1.00000 7.3 TiB 3.0 TiB 3.0 TiB 1 KiB 54 GiB 4.2 TiB 41.73 1.01 52 up osd.12 14 hdd 7.27689 1.00000 7.3 TiB 2.0 TiB 2.0 TiB 39 GiB 37 GiB 5.2 TiB 27.94 0.68 69 up osd.14 16 hdd 7.27689 1.00000 7.3 TiB 2.7 TiB 2.6 TiB 4 KiB 50 GiB 4.6 TiB 36.59 0.89 46 up osd.16 18 hdd 7.27689 1.00000 7.3 TiB 3.8 TiB 3.7 TiB 1 KiB 104 GiB 3.5 TiB 52.41 1.27 78 up osd.18 20 hdd 7.27689 1.00000 7.3 TiB 2.4 TiB 2.4 TiB 14 GiB 44 GiB 4.8 TiB 33.38 0.81 80 up osd.20 22 hdd 7.27689 1.00000 7.3 TiB 2.5 TiB 2.4 TiB 6 KiB 45 GiB 4.8 TiB 34.06 0.83 49 up osd.22 24 hdd 7.27689 1.00000 7.3 TiB 2.9 TiB 2.9 TiB 1 KiB 53 GiB 4.4 TiB 39.93 0.97 48 up osd.24 0 ssd 0.87328 1.00000 894 GiB 610 MiB 49 MiB 1 KiB 561 MiB 894 GiB 0.07 0.00 0 up osd.0 -10 88.19592 - 88 TiB 35 TiB 34 TiB 53 GiB 612 GiB 54 TiB 39.25 0.95 - host mero008 29 hdd 7.27689 1.00000 7.3 TiB 2.5 TiB 2.5 TiB 4 KiB 48 GiB 4.7 TiB 34.75 0.84 46 up osd.29 31 hdd 7.27689 1.00000 7.3 TiB 3.1 TiB 3.0 TiB 3 KiB 56 GiB 4.2 TiB 42.64 1.04 51 up osd.31 33 hdd 7.27689 1.00000 7.3 TiB 1.9 TiB 1.8 TiB 39 GiB 34 GiB 5.4 TiB 25.74 0.63 69 up osd.33 35 hdd 7.27689 1.00000 7.3 TiB 2.7 TiB 2.7 TiB 1 KiB 49 GiB 4.6 TiB 37.18 0.90 53 up osd.35 37 hdd 7.27689 1.00000 7.3 TiB 1.7 TiB 1.7 TiB 1 KiB 31 GiB 5.5 TiB 23.78 0.58 30 up osd.37 39 hdd 7.27689 1.00000 7.3 TiB 2.7 TiB 2.6 TiB 6 KiB 49 GiB 4.6 TiB 36.57 0.89 80 up osd.39 41 hdd 7.27689 1.00000 7.3 TiB 2.3 TiB 2.3 TiB 1 KiB 45 GiB 5.0 TiB 31.65 0.77 46 up osd.41 43 hdd 7.27689 1.00000 7.3 TiB 2.0 TiB 2.0 TiB 6 KiB 37 GiB 5.3 TiB 27.44 0.67 39 up osd.43 45 hdd 7.27689 1.00000 7.3 TiB 6.7 TiB 6.6 TiB 3 KiB 59 GiB 625 GiB 91.61 2.23 81 up osd.45 47 hdd 7.27689 1.00000 7.3 TiB 2.5 TiB 2.4 TiB 14 GiB 45 GiB 4.8 TiB 33.99 0.83 78 up osd.47 49 hdd 7.27689 1.00000 7.3 TiB 2.6 TiB 2.5 TiB 6 KiB 47 GiB 4.7 TiB 35.63 0.87 45 up osd.49 51 hdd 7.27689 1.00000 7.3 TiB 4.0 TiB 3.9 TiB 1 KiB 111 GiB 3.3 TiB 54.68 1.33 81 up osd.51 28 ssd 0.87328 1.00000 894 GiB 934 MiB 49 MiB 1 KiB 885 MiB 893 GiB 0.10 0.00 1 up osd.28 -3 88.19794 - 88 TiB 36 TiB 36 TiB 50 GiB 639 GiB 52 TiB 41.18 1.00 - host mero010 3 hdd 7.27739 1.00000 7.3 TiB 2.7 TiB 2.6 TiB 9 KiB 49 GiB 4.6 TiB 36.57 0.89 50 up osd.3 5 hdd 7.27739 1.00000 7.3 TiB 3.9 TiB 3.8 TiB 1 KiB 106 GiB 3.4 TiB 53.71 1.31 81 up osd.5 7 hdd 7.27739 1.00000 7.3 TiB 2.7 TiB 2.6 TiB 2 KiB 49 GiB 4.6 TiB 36.86 0.90 50 up osd.7 9 hdd 7.27739 1.00000 7.3 TiB 2.9 TiB 2.9 TiB 4 KiB 54 GiB 4.3 TiB 40.23 0.98 54 up osd.9 11 hdd 7.27689 1.00000 7.3 TiB 2.9 TiB 2.9 TiB 4 KiB 53 GiB 4.3 TiB 40.49 0.98 57 up osd.11 13 hdd 7.27689 1.00000 7.3 TiB 5.3 TiB 5.3 TiB 1 KiB 35 GiB 1.9 TiB 73.34 1.78 56 up osd.13 15 hdd 7.27689 1.00000 7.3 TiB 2.3 TiB 2.3 TiB 1 KiB 44 GiB 5.0 TiB 31.93 0.78 76 up osd.15 17 hdd 7.27689 1.00000 7.3 TiB 2.8 TiB 2.7 TiB 50 GiB 50 GiB 4.5 TiB 37.85 0.92 113 up osd.17 19 hdd 7.27689 1.00000 7.3 TiB 2.5 TiB 2.4 TiB 2 KiB 45 GiB 4.8 TiB 34.05 0.83 40 up osd.19 21 hdd 7.27689 1.00000 7.3 TiB 2.7 TiB 2.7 TiB 1 KiB 49 GiB 4.6 TiB 37.17 0.90 48 up osd.21 23 hdd 7.27689 1.00000 7.3 TiB 2.7 TiB 2.7 TiB 6 KiB 50 GiB 4.6 TiB 37.10 0.90 50 up osd.23 25 hdd 7.27689 1.00000 7.3 TiB 2.9 TiB 2.8 TiB 2 KiB 53 GiB 4.4 TiB 39.82 0.97 45 up osd.25 1 ssd 0.87328 1.00000 894 GiB 1.0 GiB 49 MiB 1 KiB 978 MiB 893 GiB 0.11 0.00 1 up osd.1 -13 88.19794 - 81 TiB 35 TiB 34 TiB 49 KiB 614 GiB 46 TiB 43.22 1.05 - host mero012 27 hdd 7.27739 1.00000 7.3 TiB 3.1 TiB 3.0 TiB 1 KiB 55 GiB 4.2 TiB 42.29 1.03 49 up osd.27 30 hdd 7.27739 1.00000 7.3 TiB 2.7 TiB 2.6 TiB 6 KiB 50 GiB 4.6 TiB 36.57 0.89 47 up osd.30 32 hdd 7.27739 1.00000 7.3 TiB 2.5 TiB 2.4 TiB 3 KiB 46 GiB 4.8 TiB 33.92 0.82 43 up osd.32 34 hdd 7.27689 1.00000 7.3 TiB 2.4 TiB 2.4 TiB 6 KiB 44 GiB 4.9 TiB 32.92 0.80 47 up osd.34 36 hdd 7.27689 1.00000 7.3 TiB 2.7 TiB 2.6 TiB 10 KiB 49 GiB 4.6 TiB 36.58 0.89 49 up osd.36 38 hdd 7.27739 1.00000 7.3 TiB 6.7 TiB 6.6 TiB 1 KiB 59 GiB 603 GiB 91.90 2.23 79 up osd.38 40 hdd 7.27689 1.00000 7.3 TiB 3.2 TiB 3.1 TiB 8 KiB 58 GiB 4.1 TiB 43.92 1.07 54 up osd.40 42 hdd 7.27689 1.00000 7.3 TiB 2.5 TiB 2.5 TiB 10 KiB 46 GiB 4.7 TiB 35.01 0.85 42 up osd.42 44 hdd 7.27689 1.00000 7.3 TiB 3.0 TiB 2.9 TiB 1 KiB 54 GiB 4.3 TiB 40.76 0.99 56 up osd.44 46 hdd 7.27689 0 0 B 0 B 0 B 0 B 0 B 0 B 0 0 0 down osd.46 48 hdd 7.27689 1.00000 7.3 TiB 2.5 TiB 2.5 TiB 1 KiB 47 GiB 4.7 TiB 34.75 0.84 45 up osd.48 50 hdd 7.27689 1.00000 7.3 TiB 3.5 TiB 3.4 TiB 1 KiB 100 GiB 3.8 TiB 48.27 1.17 70 up osd.50 26 ssd 0.87328 1.00000 894 GiB 273 GiB 268 GiB 1 KiB 5.1 GiB 621 GiB 30.53 0.74 4 up osd.26 TOTAL 346 TiB 142 TiB 140 TiB 156 GiB 2.4 TiB 203 TiB 41.15 MIN/MAX VAR: 0.00/2.29 STDDEV: 18.06 # ceph osd dump epoch 2123 fsid 08416350-2b97-11ee-a9a2-ac1f6b40d3fc created 2023-07-26T09:30:36.425304+0000 modified 2023-08-16T15:26:22.553716+0000 flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit crush_version 308 full_ratio 0.95 backfillfull_ratio 0.9 nearfull_ratio 0.85 require_min_compat_client luminous min_compat_client luminous require_osd_release quincy stretch_mode_enabled false pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode on last_change 2107 flags hashpspool,full stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr pool 2 '.rgw.root' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 2107 lfor 0/0/390 flags hashpspool,full stripe_width 0 application rgw pool 3 'default.rgw.log' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 2107 lfor 0/0/392 flags hashpspool,full stripe_width 0 application rgw pool 4 'default.rgw.control' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 2107 lfor 0/0/392 flags hashpspool,full stripe_width 0 application rgw pool 5 'default.rgw.meta' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 2107 lfor 0/0/394 flags hashpspool,full stripe_width 0 pg_autoscale_bias 4 application rgw pool 6 'ecpool-86' erasure profile rgwec86 size 14 min_size 9 crush_rule 1 object_hash rjenkins pg_num 128 pgp_num 32 pgp_num_target 128 autoscale_mode on last_change 2107 lfor 0/0/1415 flags hashpspool,full stripe_width 32768 application rgw pool 7 'default.rgw.buckets.index' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 1 pgp_num_target 32 autoscale_mode on last_change 2107 lfor 0/0/1222 flags hashpspool,full stripe_width 0 pg_autoscale_bias 4 application rgw pool 8 'default.rgw.buckets.data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 1 pgp_num_target 32 autoscale_mode on last_change 2107 lfor 0/0/1223 flags hashpspool,full stripe_width 0 application rgw pool 9 'ecpool-22' erasure profile ec22 size 4 min_size 3 crush_rule 2 object_hash rjenkins pg_num 32 pgp_num 1 pgp_num_target 32 autoscale_mode on last_change 2107 lfor 0/0/1502 flags hashpspool,full stripe_width 8192 application rgw pool 10 'source-ecpool-22' erasure profile ec22 size 4 min_size 3 crush_rule 3 object_hash rjenkins pg_num 32 pgp_num 1 pgp_num_target 32 autoscale_mode on last_change 2107 lfor 0/0/1567 flags hashpspool,full stripe_width 8192 application rgw pool 11 'default.rgw.temporary.index' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 1 pgp_num_target 32 autoscale_mode on last_change 2107 lfor 0/0/1573 flags hashpspool,full stripe_width 0 pg_autoscale_bias 4 application rgw max_osd 52 osd.0 up in weight 1 up_from 907 up_thru 0 down_at 904 last_clean_interval [662,903) [v2:10.8.129.226:6802/3094491253,v1:10.8.129.226:6803/3094491253] [v2:10.8.129.226:6804/3094491253,v1:10.8.129.226:6805/3094491253] exists,up df815a8b-4b2b-4d4e-aa29-69604c2b0e85 osd.1 up in weight 1 up_from 1047 up_thru 1027 down_at 1039 last_clean_interval [870,1038) [v2:10.8.129.230:6840/158177981,v1:10.8.129.230:6841/158177981] [v2:10.8.129.230:6842/158177981,v1:10.8.129.230:6843/158177981] exists,up c61c7d97-297e-4cc0-a9ea-c88e2ed71049 osd.2 up in weight 1 up_from 975 up_thru 2102 down_at 959 last_clean_interval [700,958) [v2:10.8.129.226:6850/3076130188,v1:10.8.129.226:6851/3076130188] [v2:10.8.129.226:6852/3076130188,v1:10.8.129.226:6853/3076130188] exists,up 272222b1-5d65-4a8e-9837-622cf310adac osd.3 up in weight 1 up_from 1046 up_thru 2070 down_at 1034 last_clean_interval [890,1033) [v2:10.8.129.230:6848/1460467281,v1:10.8.129.230:6849/1460467281] [v2:10.8.129.230:6850/1460467281,v1:10.8.129.230:6851/1460467281] exists,up f4995ef4-9915-4329-9528-ef0d56d934f2 osd.4 up in weight 1 up_from 963 up_thru 1618 down_at 943 last_clean_interval [693,942) [v2:10.8.129.226:6800/1353244696,v1:10.8.129.226:6801/1353244696] [v2:10.8.129.226:6882/1353244696,v1:10.8.129.226:6883/1353244696] exists,full,up 0581480e-1748-4fd3-b37f-555fe9ce317a osd.5 up in weight 1 up_from 1076 up_thru 1172 down_at 1066 last_clean_interval [880,1065) [v2:10.8.129.230:6802/2798993078,v1:10.8.129.230:6804/2798993078] [v2:10.8.129.230:6806/2798993078,v1:10.8.129.230:6808/2798993078] exists,up bb05e4df-49ba-4d02-b74d-08a10a810e76 osd.6 up in weight 1 up_from 969 up_thru 1453 down_at 956 last_clean_interval [696,955) [v2:10.8.129.226:6888/446063068,v1:10.8.129.226:6889/446063068] [v2:10.8.129.226:6890/446063068,v1:10.8.129.226:6891/446063068] exists,up 1ee71f67-f629-4093-b0b0-aa75dc13034d osd.7 up in weight 1 up_from 1095 up_thru 1420 down_at 1082 last_clean_interval [890,1081) [v2:10.8.129.230:6856/1692245370,v1:10.8.129.230:6857/1692245370] [v2:10.8.129.230:6858/1692245370,v1:10.8.129.230:6859/1692245370] exists,up 0a1b4910-bf0d-4bf4-b083-e268060c2fcf osd.8 up in weight 1 up_from 930 up_thru 1716 down_at 911 last_clean_interval [672,910) [v2:10.8.129.226:6898/3879013574,v1:10.8.129.226:6899/3879013574] [v2:10.8.129.226:6900/3879013574,v1:10.8.129.226:6901/3879013574] exists,up 1976a58e-60a1-44d0-9f1a-54472d3e0d8c osd.9 up in weight 1 up_from 1075 up_thru 2103 down_at 1053 last_clean_interval [887,1052) [v2:10.8.129.230:6880/2181136911,v1:10.8.129.230:6881/2181136911] [v2:10.8.129.230:6882/2181136911,v1:10.8.129.230:6883/2181136911] exists,up 55b5c11f-3e97-4810-bbca-50f24b3516c2 osd.10 up in weight 1 up_from 985 up_thru 1419 down_at 967 last_clean_interval [708,966) [v2:10.8.129.226:6810/1362821292,v1:10.8.129.226:6811/1362821292] [v2:10.8.129.226:6812/1362821292,v1:10.8.129.226:6813/1362821292] exists,up 23df3ddf-150c-47cd-8a3e-f15b4ce6654e osd.11 up in weight 1 up_from 1077 up_thru 1421 down_at 1061 last_clean_interval [878,1060) [v2:10.8.129.230:6896/2021348337,v1:10.8.129.230:6897/2021348337] [v2:10.8.129.230:6898/2021348337,v1:10.8.129.230:6899/2021348337] exists,up 7b672611-2905-47f9-93d0-c2c2a98e0955 osd.12 up in weight 1 up_from 922 up_thru 1454 down_at 905 last_clean_interval [670,904) [v2:10.8.129.226:6818/2201810534,v1:10.8.129.226:6819/2201810534] [v2:10.8.129.226:6820/2201810534,v1:10.8.129.226:6821/2201810534] exists,up 8c8f5dcb-16c4-447d-bb0e-bfcf5bb0f760 osd.13 up in weight 1 up_from 1093 up_thru 1420 down_at 1080 last_clean_interval [876,1079) [v2:10.8.129.230:6800/2373212948,v1:10.8.129.230:6801/2373212948] [v2:10.8.129.230:6803/2373212948,v1:10.8.129.230:6805/2373212948] exists,up a3c5a817-9d2d-4ca9-a875-e8f81213b5fd osd.14 up in weight 1 up_from 940 up_thru 1450 down_at 920 last_clean_interval [678,919) [v2:10.8.129.226:6826/2664822750,v1:10.8.129.226:6827/2664822750] [v2:10.8.129.226:6828/2664822750,v1:10.8.129.226:6829/2664822750] exists,up f5c46114-31d1-4082-8ec8-b6f9e30d665c osd.15 up in weight 1 up_from 1072 up_thru 1176 down_at 1056 last_clean_interval [881,1055) [v2:10.8.129.230:6864/3293135491,v1:10.8.129.230:6866/3293135491] [v2:10.8.129.230:6868/3293135491,v1:10.8.129.230:6870/3293135491] exists,up b7d06d05-d461-48ac-ad48-521635c6a959 osd.16 up in weight 1 up_from 951 up_thru 1167 down_at 932 last_clean_interval [686,931) [v2:10.8.129.226:6834/1899443408,v1:10.8.129.226:6835/1899443408] [v2:10.8.129.226:6836/1899443408,v1:10.8.129.226:6837/1899443408] exists,up 80b704dd-8bb1-4385-b6ad-6cec35f0d7f3 osd.17 up in weight 1 up_from 1085 up_thru 1454 down_at 1069 last_clean_interval [882,1068) [v2:10.8.129.230:6887/3357728595,v1:10.8.129.230:6889/3357728595] [v2:10.8.129.230:6890/3357728595,v1:10.8.129.230:6891/3357728595] exists,up 1d024538-9f81-4123-b148-6e83dd0c5c68 osd.18 up in weight 1 up_from 953 up_thru 1421 down_at 938 last_clean_interval [690,937) [v2:10.8.129.226:6842/3749316919,v1:10.8.129.226:6843/3749316919] [v2:10.8.129.226:6844/3749316919,v1:10.8.129.226:6845/3749316919] exists,up 989a0734-c229-4c8c-884d-b2330d247aff osd.19 up in weight 1 up_from 1090 up_thru 1453 down_at 1075 last_clean_interval [881,1074) [v2:10.8.129.230:6832/563044682,v1:10.8.129.230:6833/563044682] [v2:10.8.129.230:6834/563044682,v1:10.8.129.230:6835/563044682] exists,up b556aaa5-03ef-49d8-8fab-8df67c9374ed osd.20 up in weight 1 up_from 933 up_thru 2029 down_at 916 last_clean_interval [677,915) [v2:10.8.129.226:6858/950678303,v1:10.8.129.226:6859/950678303] [v2:10.8.129.226:6860/950678303,v1:10.8.129.226:6861/950678303] exists,up 6e21c44e-83e9-4ee8-b361-31f04c132786 osd.21 up in weight 1 up_from 1059 up_thru 1453 down_at 1045 last_clean_interval [884,1044) [v2:10.8.129.230:6824/4293388863,v1:10.8.129.230:6825/4293388863] [v2:10.8.129.230:6826/4293388863,v1:10.8.129.230:6827/4293388863] exists,up 7b0a3ae6-932c-45eb-bb28-1e9ee59b14a7 osd.22 up in weight 1 up_from 979 up_thru 1421 down_at 963 last_clean_interval [703,962) [v2:10.8.129.226:6866/677054875,v1:10.8.129.226:6867/677054875] [v2:10.8.129.226:6868/677054875,v1:10.8.129.226:6869/677054875] exists,up 3fe49dbb-6542-46c6-a5b5-a879f79e3c46 osd.23 up in weight 1 up_from 1064 up_thru 1998 down_at 1050 last_clean_interval [882,1049) [v2:10.8.129.230:6865/1410573212,v1:10.8.129.230:6867/1410573212] [v2:10.8.129.230:6869/1410573212,v1:10.8.129.230:6871/1410573212] exists,up db7a102a-58b1-4c71-8702-8428b719000d osd.24 up in weight 1 up_from 948 up_thru 1453 down_at 926 last_clean_interval [683,925) [v2:10.8.129.226:6874/423713120,v1:10.8.129.226:6875/423713120] [v2:10.8.129.226:6876/423713120,v1:10.8.129.226:6877/423713120] exists,up a2c6c4d0-e6a5-4b67-ba61-5c83c3c30f6d osd.25 up in weight 1 up_from 1047 up_thru 1047 down_at 1030 last_clean_interval [882,1029) [v2:10.8.129.230:6816/4054846143,v1:10.8.129.230:6817/4054846143] [v2:10.8.129.230:6818/4054846143,v1:10.8.129.230:6819/4054846143] exists,up d005570f-e16e-47d1-967a-2b8f14346041 osd.26 up in weight 1 up_from 1162 up_thru 0 down_at 1158 last_clean_interval [1112,1157) [v2:10.8.129.232:6864/1071789365,v1:10.8.129.232:6865/1071789365] [v2:10.8.129.232:6866/1071789365,v1:10.8.129.232:6867/1071789365] exists,up 41e73258-ae23-42ea-bd53-2f9ffecf3b20 osd.27 up in weight 1 up_from 1171 up_thru 1997 down_at 1158 last_clean_interval [1143,1157) [v2:10.8.129.232:6840/1069997510,v1:10.8.129.232:6841/1069997510] [v2:10.8.129.232:6842/1069997510,v1:10.8.129.232:6843/1069997510] exists,up 6185dc9a-30fa-4226-b181-112bf1e91792 osd.28 up in weight 1 up_from 1019 up_thru 0 down_at 1015 last_clean_interval [829,1014) [v2:10.8.129.228:6800/2973247038,v1:10.8.129.228:6801/2973247038] [v2:10.8.129.228:6802/2973247038,v1:10.8.129.228:6803/2973247038] exists,up e44ce344-9466-42f0-8e7b-6d321be96f95 osd.29 up in weight 1 up_from 1036 up_thru 2028 down_at 1023 last_clean_interval [844,1022) [v2:10.8.129.228:6808/321767215,v1:10.8.129.228:6809/321767215] [v2:10.8.129.228:6810/321767215,v1:10.8.129.228:6811/321767215] exists,up 7aea231d-c242-41d0-b7a8-66f588256b12 osd.30 up in weight 1 up_from 1168 up_thru 1450 down_at 1158 last_clean_interval [1137,1157) [v2:10.8.129.232:6824/3167128113,v1:10.8.129.232:6825/3167128113] [v2:10.8.129.232:6826/3167128113,v1:10.8.129.232:6827/3167128113] exists,up de553426-7ae1-4e5b-838f-433a4289e660 osd.31 up in weight 1 up_from 1010 up_thru 1419 down_at 996 last_clean_interval [822,995) [v2:10.8.129.228:6816/3322840912,v1:10.8.129.228:6817/3322840912] [v2:10.8.129.228:6818/3322840912,v1:10.8.129.228:6819/3322840912] exists,up e2996892-e170-4e04-849c-f65c6bc824b5 osd.32 up in weight 1 up_from 1171 up_thru 1450 down_at 1158 last_clean_interval [1123,1157) [v2:10.8.129.232:6800/3847290653,v1:10.8.129.232:6801/3847290653] [v2:10.8.129.232:6802/3847290653,v1:10.8.129.232:6803/3847290653] exists,up 098da4f5-4a20-44d3-aa34-f598a09a6cd8 osd.33 up in weight 1 up_from 1453 up_thru 1453 down_at 1450 last_clean_interval [989,1452) [v2:10.8.129.228:6824/4274883071,v1:10.8.129.228:6825/4274883071] [v2:10.8.129.228:6906/4275883071,v1:10.8.129.228:6907/4275883071] exists,up b31676c9-8d42-4be9-8f2b-c37000c7806f osd.34 up in weight 1 up_from 1166 up_thru 1418 down_at 1157 last_clean_interval [1132,1156) [v2:10.8.129.232:6886/4287733808,v1:10.8.129.232:6889/4287733808] [v2:10.8.129.232:6892/4287733808,v1:10.8.129.232:6895/4287733808] exists,up dbe2dd67-25eb-462f-a9ed-a23f91ab474a osd.35 up in weight 1 up_from 1031 up_thru 1417 down_at 1017 last_clean_interval [841,1016) [v2:10.8.129.228:6832/616801318,v1:10.8.129.228:6833/616801318] [v2:10.8.129.228:6834/616801318,v1:10.8.129.228:6835/616801318] exists,up 1a2a04f4-1847-4c7c-95b3-d2b0f44a74b6 osd.36 up in weight 1 up_from 1166 up_thru 1166 down_at 1158 last_clean_interval [1114,1157) [v2:10.8.129.232:6832/2179735299,v1:10.8.129.232:6833/2179735299] [v2:10.8.129.232:6834/2179735299,v1:10.8.129.232:6835/2179735299] exists,up 7c662a89-e6e0-4a26-8c28-5f6ac6544b01 osd.37 up in weight 1 up_from 1013 up_thru 1166 down_at 1000 last_clean_interval [828,999) [v2:10.8.129.228:6840/1785038887,v1:10.8.129.228:6841/1785038887] [v2:10.8.129.228:6842/1785038887,v1:10.8.129.228:6843/1785038887] exists,up 681eb013-4421-4385-91fd-de97a4bc70dc osd.38 up in weight 1 up_from 1172 up_thru 1172 down_at 1157 last_clean_interval [1104,1156) [v2:10.8.129.232:6883/2199298006,v1:10.8.129.232:6884/2199298006] [v2:10.8.129.232:6887/2199298006,v1:10.8.129.232:6890/2199298006] backfillfull,exists,up 7a99c61a-b048-4eef-9dc9-e147c0671e1a osd.39 up in weight 1 up_from 1020 up_thru 1420 down_at 1006 last_clean_interval [834,1005) [v2:10.8.129.228:6848/1386878793,v1:10.8.129.228:6849/1386878793] [v2:10.8.129.228:6850/1386878793,v1:10.8.129.228:6851/1386878793] exists,up 2819b723-b58d-48b5-847e-c7a2328109a0 osd.40 up in weight 1 up_from 1167 up_thru 1419 down_at 1157 last_clean_interval [1120,1156) [v2:10.8.129.232:6816/3799748896,v1:10.8.129.232:6817/3799748896] [v2:10.8.129.232:6818/3799748896,v1:10.8.129.232:6819/3799748896] exists,up b6f8da66-aded-4cef-95f9-42b22ac5bcf7 osd.41 up in weight 1 up_from 996 up_thru 1454 down_at 982 last_clean_interval [810,981) [v2:10.8.129.228:6856/2448402679,v1:10.8.129.228:6857/2448402679] [v2:10.8.129.228:6858/2448402679,v1:10.8.129.228:6859/2448402679] exists,up 675a42f0-d87d-442d-92f9-d747334148e2 osd.42 up in weight 1 up_from 1167 up_thru 1172 down_at 1158 last_clean_interval [1150,1157) [v2:10.8.129.232:6856/3125925780,v1:10.8.129.232:6857/3125925780] [v2:10.8.129.232:6858/3125925780,v1:10.8.129.232:6859/3125925780] exists,up b5c3ad69-641e-480a-a844-22cac9bbdee5 osd.43 up in weight 1 up_from 1000 up_thru 1454 down_at 988 last_clean_interval [813,987) [v2:10.8.129.228:6864/976503328,v1:10.8.129.228:6865/976503328] [v2:10.8.129.228:6866/976503328,v1:10.8.129.228:6867/976503328] exists,up bb76b517-592f-4bb1-adbb-6c7b1df81b72 osd.44 up in weight 1 up_from 1168 up_thru 1454 down_at 1157 last_clean_interval [1155,1156) [v2:10.8.129.232:6848/3530531185,v1:10.8.129.232:6849/3530531185] [v2:10.8.129.232:6850/3530531185,v1:10.8.129.232:6851/3530531185] exists,up eebbcc73-a675-4520-affd-8eef38a45b71 osd.45 up in weight 1 up_from 1041 up_thru 1567 down_at 1027 last_clean_interval [850,1026) [v2:10.8.129.228:6872/2543213982,v1:10.8.129.228:6873/2543213982] [v2:10.8.129.228:6874/2543213982,v1:10.8.129.228:6875/2543213982] backfillfull,exists,up f061c419-4d4c-4977-89d7-ea019bee79e6 osd.46 down out weight 0 up_from 1151 up_thru 1157 down_at 1158 last_clean_interval [795,1132) [v2:10.8.129.232:6880/3886566179,v1:10.8.129.232:6881/3886566179] [v2:10.8.129.232:6882/3886566179,v1:10.8.129.232:6883/3886566179] autoout,exists d4d3519f-1600-4f54-9911-0844d3a3afe8 osd.47 up in weight 1 up_from 1002 up_thru 1573 down_at 992 last_clean_interval [819,991) [v2:10.8.129.228:6880/2059793626,v1:10.8.129.228:6881/2059793626] [v2:10.8.129.228:6882/2059793626,v1:10.8.129.228:6883/2059793626] exists,up 4bb01b8b-6cf9-44d8-84bc-41665437f285 osd.48 up in weight 1 up_from 1167 up_thru 1453 down_at 1158 last_clean_interval [1145,1157) [v2:10.8.129.232:6868/2469242884,v1:10.8.129.232:6870/2469242884] [v2:10.8.129.232:6873/2469242884,v1:10.8.129.232:6875/2469242884] exists,up 5efe2cda-d0f8-47e9-85e3-8568c5b9b2c8 osd.49 up in weight 1 up_from 987 up_thru 1167 down_at 972 last_clean_interval [800,971) [v2:10.8.129.228:6888/189999320,v1:10.8.129.228:6889/189999320] [v2:10.8.129.228:6890/189999320,v1:10.8.129.228:6891/189999320] exists,up df6a1a2b-debd-41e1-8aab-6a1f7c74ad4b osd.50 up in weight 1 up_from 1166 up_thru 1717 down_at 1158 last_clean_interval [1135,1157) [v2:10.8.129.232:6880/4261136807,v1:10.8.129.232:6881/4261136807] [v2:10.8.129.232:6882/4261136807,v1:10.8.129.232:6885/4261136807] exists,up 4b7bd74d-b810-465c-a140-e38e2ab633d8 osd.51 up in weight 1 up_from 1024 up_thru 1961 down_at 1009 last_clean_interval [837,1008) [v2:10.8.129.228:6896/3556686938,v1:10.8.129.228:6897/3556686938] [v2:10.8.129.228:6898/3556686938,v1:10.8.129.228:6899/3556686938] exists,up c627a599-ffe0-4046-a88f-a8a18fed72fc pg_upmap_items 3.2 [33,31] pg_upmap_items 6.0 [31,29,6,2,7,13] pg_upmap_items 6.1 [44,50,24,20,45,37] pg_upmap_items 6.3 [24,20] pg_upmap_items 6.4 [11,23] pg_upmap_items 6.6 [45,43] pg_upmap_items 6.7 [9,23] pg_upmap_items 6.8 [40,32] pg_upmap_items 6.9 [24,8] pg_upmap_items 6.a [7,15] pg_upmap_items 6.b [51,33] pg_upmap_items 6.e [11,3] pg_upmap_items 6.f [45,33] pg_upmap_items 6.10 [16,14] pg_upmap_items 6.11 [12,2] pg_upmap_items 6.12 [17,5] pg_upmap_items 6.13 [12,2,40,32] pg_upmap_items 6.14 [31,33,4,10] pg_upmap_items 6.16 [11,5,9,15] pg_upmap_items 6.1a [51,33] pg_upmap_items 6.1f [24,2] pg_temp 6.0 [2,4,12,20,39,49,35,29,40,38,50,36,13,21] pg_temp 6.2 [18,4,6,2,45,29,39,47,19,11,17,23,40,48] pg_temp 6.3 [7,9,5,3,22,20,14,8,51,31,45,35,50,48] pg_temp 6.4 [14,16,6,10,7,5,15,23,34,42,50,48,45,47] pg_temp 6.5 [21,11,9,17,40,34,32,48,45,35,41,31,20,6] pg_temp 6.6 [43,37,41,33,9,21,19,7,36,44,32,42,24,18] pg_temp 6.7 [29,45,43,39,6,12,14,18,11,5,3,23,48,27] pg_temp 6.8 [11,21,9,23,41,35,51,29,24,22,16,12,32,34] pg_temp 6.9 [12,16,8,18,38,30,32,44,17,25,7,23,31,43] pg_temp 6.a [17,15,11,9,36,30,27,38,8,6,16,4,31,33] pg_temp 6.b [12,24,8,6,3,7,19,21,43,37,33,47,30,27] pg_temp 6.c [20,16,24,8,35,41,31,51,44,48,40,50,19,9] pg_temp 6.e [21,25,3,17,45,47,29,37,36,42,40,48,4,22] pg_temp 6.f [41,51,47,33,18,4,12,2,5,25,17,21,40,27] pg_temp 6.10 [10,8,14,2,51,39,49,31,30,38,34,32,3,15] pg_temp 6.12 [39,35,37,45,18,10,22,6,13,15,23,5,36,34] pg_temp 6.14 [27,42,2147483647,44,23,15,25,19,41,33,37,29,22,10] pg_temp 6.15 [13,17,3,25,44,27,34,40,16,22,10,24,47,29] pg_temp 6.16 [22,16,8,4,23,21,15,5,27,30,36,42,45,41] pg_temp 6.17 [22,10,18,14,17,7,11,15,29,45,39,35,44,30] pg_temp 6.18 [34,36,40,44,9,15,7,19,43,31,47,49,4,24] pg_temp 6.19 [22,24,12,6,17,19,21,25,49,35,47,31,40,48] pg_temp 6.1b [29,33,49,41,25,11,13,9,30,36,44,27,20,10] pg_temp 6.1d [8,4,2,24,25,5,11,23,38,32,42,30,39,41] pg_temp 6.1e [44,42,50,36,20,6,12,18,35,33,31,49,3,5] pg_temp 6.20 [2,4,12,20,39,49,35,29,40,38,50,36,13,21] pg_temp 6.21 [20,16,2,10,37,29,39,43,50,30,44,38,7,3] pg_temp 6.22 [18,4,6,2,45,29,39,47,19,11,17,23,40,48] pg_temp 6.23 [7,9,5,3,22,20,14,8,51,31,45,35,50,48] pg_temp 6.24 [14,16,6,10,7,5,15,23,34,42,50,48,45,47] pg_temp 6.25 [21,11,9,17,40,34,32,48,45,35,41,31,20,6] pg_temp 6.26 [43,37,41,33,9,21,19,7,36,44,32,42,24,18] pg_temp 6.27 [29,45,43,39,6,12,14,18,11,5,3,23,48,27] pg_temp 6.28 [11,21,9,23,41,35,51,29,24,22,16,12,32,34] pg_temp 6.29 [12,16,8,18,38,30,32,44,17,25,7,23,31,43] pg_temp 6.2a [17,15,11,9,36,30,27,38,8,6,16,4,31,33] pg_temp 6.2b [12,24,8,6,3,7,19,21,43,37,33,47,30,27] pg_temp 6.2c [20,16,24,8,35,41,31,51,44,48,40,50,19,9] pg_temp 6.2d [8,4,12,20,47,49,43,51,50,2147483647,38,48,11,7] pg_temp 6.2e [21,25,3,17,45,47,29,37,36,42,40,48,4,22] pg_temp 6.2f [41,51,47,33,18,4,12,2,5,25,17,21,40,27] pg_temp 6.30 [10,8,14,2,51,39,49,31,30,38,34,32,3,15] pg_temp 6.31 [11,25,23,17,32,34,38,44,45,39,49,51,2,24] pg_temp 6.32 [39,35,37,45,18,10,22,6,13,15,23,5,36,34] pg_temp 6.35 [13,17,3,25,44,27,34,40,16,22,10,24,47,29] pg_temp 6.36 [22,16,8,4,23,21,15,5,27,30,36,42,45,41] pg_temp 6.37 [22,10,18,14,17,7,11,15,29,45,39,35,44,30] pg_temp 6.38 [34,36,40,44,9,15,7,19,43,31,47,49,4,24] pg_temp 6.39 [22,24,12,6,17,19,21,25,49,35,47,31,40,48] pg_temp 6.3b [29,33,49,41,25,11,13,9,30,36,44,27,20,10] pg_temp 6.3d [8,4,2,24,25,5,11,23,38,32,42,30,39,41] pg_temp 6.3e [44,42,50,36,20,6,12,18,35,33,31,49,3,5] pg_temp 6.40 [2,4,12,20,39,49,35,29,40,38,50,36,13,21] pg_temp 6.41 [20,16,2,10,37,29,39,43,50,30,44,38,7,3] pg_temp 6.42 [18,4,6,2,45,29,39,47,19,11,17,23,40,48] pg_temp 6.43 [7,9,5,3,22,20,14,8,51,31,45,35,50,48] pg_temp 6.44 [14,16,6,10,7,5,15,23,34,42,50,48,45,47] pg_temp 6.45 [21,11,9,17,40,34,32,48,45,35,41,31,20,6] pg_temp 6.46 [43,37,41,33,9,21,19,7,36,44,32,42,24,18] pg_temp 6.47 [29,45,43,39,6,12,14,18,11,5,3,23,48,27] pg_temp 6.48 [11,21,9,23,41,35,51,29,24,22,16,12,32,34] pg_temp 6.49 [12,16,8,18,38,30,32,44,17,25,7,23,31,43] pg_temp 6.4a [17,15,11,9,36,30,27,38,8,6,16,4,31,33] pg_temp 6.4b [12,24,8,6,3,7,19,21,43,37,33,47,30,27] pg_temp 6.4c [20,16,24,8,35,41,31,51,44,48,40,50,19,9] pg_temp 6.4e [21,25,3,17,45,47,29,37,36,42,40,48,4,22] pg_temp 6.4f [41,51,47,33,18,4,12,2,5,25,17,21,40,27] pg_temp 6.50 [10,8,14,2,51,39,49,31,30,38,34,32,3,15] pg_temp 6.51 [11,25,23,17,32,34,38,44,45,39,49,51,2,24] pg_temp 6.52 [39,35,37,45,18,10,22,6,13,15,23,5,36,34] pg_temp 6.54 [27,42,2147483647,44,23,15,25,19,41,33,37,29,22,10] pg_temp 6.55 [13,17,3,25,44,27,34,40,16,22,10,24,47,29] pg_temp 6.56 [22,16,8,4,23,21,15,5,27,30,36,42,45,41] pg_temp 6.57 [22,10,18,14,17,7,11,15,29,45,39,35,44,30] pg_temp 6.58 [34,36,40,44,9,15,7,19,43,31,47,49,4,24] pg_temp 6.59 [22,24,12,6,17,19,21,25,49,35,47,31,40,48] pg_temp 6.5d [8,4,2,24,25,5,11,23,38,32,42,30,39,41] pg_temp 6.5e [44,42,50,36,20,6,12,18,35,33,31,49,3,5] pg_temp 6.60 [2,4,12,20,39,49,35,29,40,38,50,36,13,21] pg_temp 6.61 [20,16,2,10,37,29,39,43,50,30,44,38,7,3] pg_temp 6.62 [18,4,6,2,45,29,39,47,19,11,17,23,40,48] pg_temp 6.63 [7,9,5,3,22,20,14,8,51,31,45,35,50,48] pg_temp 6.64 [14,16,6,10,7,5,15,23,34,42,50,48,45,47] pg_temp 6.65 [21,11,9,17,40,34,32,48,45,35,41,31,20,6] pg_temp 6.66 [43,37,41,33,9,21,19,7,36,44,32,42,24,18] pg_temp 6.67 [29,45,43,39,6,12,14,18,11,5,3,23,48,27] pg_temp 6.68 [11,21,9,23,41,35,51,29,24,22,16,12,32,34] pg_temp 6.69 [12,16,8,18,38,30,32,44,17,25,7,23,31,43] pg_temp 6.6a [17,15,11,9,36,30,27,38,8,6,16,4,31,33] pg_temp 6.6b [12,24,8,6,3,7,19,21,43,37,33,47,30,27] pg_temp 6.6c [20,16,24,8,35,41,31,51,44,48,40,50,19,9] pg_temp 6.6d [8,4,12,20,47,49,43,51,50,2147483647,38,48,11,7] pg_temp 6.6e [21,25,3,17,45,47,29,37,36,42,40,48,4,22] pg_temp 6.6f [41,51,47,33,18,4,12,2,5,25,17,21,40,27] pg_temp 6.70 [10,8,14,2,51,39,49,31,30,38,34,32,3,15] pg_temp 6.71 [11,25,23,17,32,34,38,44,45,39,49,51,2,24] pg_temp 6.72 [39,35,37,45,18,10,22,6,13,15,23,5,36,34] pg_temp 6.75 [13,17,3,25,44,27,34,40,16,22,10,24,47,29] pg_temp 6.76 [22,16,8,4,23,21,15,5,27,30,36,42,45,41] pg_temp 6.77 [22,10,18,14,17,7,11,15,29,45,39,35,44,30] pg_temp 6.78 [34,36,40,44,9,15,7,19,43,31,47,49,4,24] pg_temp 6.79 [22,24,12,6,17,19,21,25,49,35,47,31,40,48] pg_temp 6.7d [8,4,2,24,25,5,11,23,38,32,42,30,39,41] pg_temp 6.7e [44,42,50,36,20,6,12,18,35,33,31,49,3,5] Version-Release number of selected component (if applicable): ceph version 17.2.6-70.el9cp (fe62dcdbb2c6e05782a3e2b67d025b84ff5047cc) quincy (stable) How reproducible: 2/2 on heterogeneous clusters. Steps to Reproduce: 1. Deploy heterogeneous RHCS cluster. 2. Fill the cluster to around 30%, bring down hosts, OSDs and trigger recovery. 3. Observe that, on each host, where there are heterogenous OSDs, during recovery/ backill operations, all data is written onto to 1 OSD instead of distributing it among all with same weights. Actual results: Uneven PG distribution resulting in full OSDs with recovery/backfill Expected results: Even PG distribution among OSDs during recovery/backfill Additional info: