+++ This bug was initially created as a clone of Bug #2169255 +++ * Description of problem (please be detailed as possible and provide log snippets): 2 out of 3 OSDs are failing with ceph_abort_msg(\"bluefs enospc\") error. osd.0 ------------------------------------- meta : "crash_id": "2023-01-31T22:24:37.982692Z_a12393e2-3894-4ba0-a373-2efb7534efae", "timestamp": "2023-01-31T22:24:37.982692Z", "process_name": "ceph-osd", "entity_name": "osd.0", "ceph_version": "16.2.7-126.el8cp", "utsname_hostname": "rook-ceph-osd-0-7866955fb5-mk7r4", "assert_msg": "/builddir/build/BUILD/ceph-16.2.7/src/os/bluestore/BlueFS.cc: In function 'int BlueFS::_flush_range(BlueFS::FileWriter*, uint64_t, uint64_t)' thread 7f9a6d569080 time 2023-01-31T22:24:37.968235+0000\n/builddir/build/BUILD/ceph-16.2.7/src/os/bluestore/BlueFS.cc: 2768: ceph_abort_msg(\"bluefs enospc\")\n", log : -5> 2023-01-31T22:24:37.520+0000 7f9a6d569080 3 rocksdb: [le/block_based/filter_policy.cc:584] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. -4> 2023-01-31T22:24:37.966+0000 7f9a6d569080 1 bluefs _allocate unable to allocate 0x20000 on bdev 1, allocator name block, allocator type hybrid, capacity 0x10000000000, block size 0x1000, free 0x26ddb37000, fragmentation 0.727513, allocated 0x0 -3> 2023-01-31T22:24:37.966+0000 7f9a6d569080 -1 bluefs _allocate allocation failed, needed 0x177ef -2> 2023-01-31T22:24:37.966+0000 7f9a6d569080 -1 bluefs _flush_range allocated: 0x0 offset: 0x0 length: 0x177ef -1> 2023-01-31T22:24:37.974+0000 7f9a6d569080 -1 /builddir/build/BUILD/ceph-16.2.7/src/os/bluestore/BlueFS.cc: In function 'int BlueFS::_flush_range(BlueFS::FileWriter*, uint64_t, uint64_t)' thread 7f9a6d569080 time 2023-01-31T22:24:37.968235+0000 /builddir/build/BUILD/ceph-16.2.7/src/os/bluestore/BlueFS.cc: 2768: ceph_abort_msg("bluefs enospc") ceph version 16.2.7-126.el8cp (fe0af61d104d48cb9d116cde6e593b5fc8c197e4) pacific (stable) 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0xe5) [0x55b1f4ae7562] 2: (BlueFS::_flush_range(BlueFS::FileWriter*, unsigned long, unsigned long)+0x1131) [0x55b1f51e59e1] 3: (BlueFS::_flush(BlueFS::FileWriter*, bool, bool*)+0x90) [0x55b1f51e5cc0] 4: (BlueFS::_fsync(BlueFS::FileWriter*, std::unique_lock<std::mutex>&)+0x159) [0x55b1f51e8699] 5: (BlueRocksWritableFile::Sync()+0x6c) [0x55b1f52100dc] 6: (rocksdb::LegacyWritableFileWrapper::Sync(rocksdb::IOOptions const&, rocksdb::IODebugContext*)+0x1f) [0x55b1f56a95ff] 7: (rocksdb::WritableFileWriter::SyncInternal(bool)+0x402) [0x55b1f57bad72] 8: (rocksdb::WritableFileWriter::Sync(bool)+0x88) [0x55b1f57bc3b8] 9: (rocksdb::BuildTable(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, rocksdb::Env*, rocksdb::FileSystem*, rocksdb::ImmutableCFOptions const&, rocksdb::MutableCFOptions const&, rocksdb::FileOptions const&, rocksdb::TableCache*, rocksdb::InternalIteratorBase<rocksdb::Slice>*, std::vector<std::unique_ptr<rocksdb::FragmentedRangeTombstoneIterator, std::default_delete<rocksdb::FragmentedRangeTombstoneIterator> >, std::allocator<std::unique_ptr<rocksdb::FragmentedRangeTombstoneIterator, std::default_delete<rocksdb::FragmentedRangeTombstoneIterator> > > >, rocksdb::FileMetaData*, rocksdb::InternalKeyComparator const&, std::vector<std::unique_ptr<rocksdb::IntTblPropCollectorFactory, std::default_delete<rocksdb::IntTblPropCollectorFactory> >, std::allocator<std::unique_ptr<rocksdb::IntTblPropCollectorFactory, std::default_delete<rocksdb::IntTblPropCollectorFactory> > > > const*, unsigned int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<unsigned long, std::allocator<unsigned long> >, unsigned long, rocksdb::SnapshotChecker*, rocksdb::CompressionType, unsigned long, rocksdb::CompressionOptions const&, bool, rocksdb::InternalStats*, rocksdb::TableFileCreationReason, rocksdb::EventLogger*, int, rocksdb::Env::IOPriority, rocksdb::TableProperties*, int, unsigned long, unsigned long, rocksdb::Env::WriteLifeTimeHint, unsigned long)+0x2ddb) [0x55b1f588951b] 10: (rocksdb::DBImpl::WriteLevel0TableForRecovery(int, rocksdb::ColumnFamilyData*, rocksdb::MemTable*, rocksdb::VersionEdit*)+0xcf5) [0x55b1f56ec1e5] 11: (rocksdb::DBImpl::RecoverLogFiles(std::vector<unsigned long, std::allocator<unsigned long> > const&, unsigned long*, bool, bool*)+0x1c2e) [0x55b1f56ee91e] 12: (rocksdb::DBImpl::Recover(std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, bool, bool, bool, unsigned long*)+0xae8) [0x55b1f56efc78] ------------------------------------- osd.2 ------------------------------------- meta: "crash_id": "2023-01-31T22:22:46.110366Z_a7d81158-86f6-4a06-9e07-0bd2b6dcc00e", "timestamp": "2023-01-31T22:22:46.110366Z", "process_name": "ceph-osd", "entity_name": "osd.2", "ceph_version": "16.2.7-126.el8cp", "utsname_hostname": "rook-ceph-osd-2-54f7b46467-k98fw", assert_thread_name": "ceph-osd", "assert_msg": "/builddir/build/BUILD/ceph-16.2.7/src/os/bluestore/BlueFS.cc: In function 'int BlueFS::_flush_range(BlueFS::FileWriter*, uint64_t, uint64_t)' thread 7faf4f45a080 time 2023-01-31T22:22:46.096073+0000\n/builddir/build/BUILD/ceph-16.2.7/src/os/bluestore/BlueFS.cc: 2768: ceph_abort_msg(\"bluefs enospc\")\n", "backtrace": log -5> 2023-01-31T22:22:45.696+0000 7faf4f45a080 3 rocksdb: [le/block_based/filter_policy.cc:584] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. -4> 2023-01-31T22:22:46.094+0000 7faf4f45a080 1 bluefs _allocate unable to allocate 0x30000 on bdev 1, allocator name block, allocator type hybrid, capacity 0x10000000000, block size 0x1000, free 0x26c0119000, fragmentation 0.728368, allocated 0x0 -3> 2023-01-31T22:22:46.094+0000 7faf4f45a080 -1 bluefs _allocate allocation failed, needed 0x2aac6 -2> 2023-01-31T22:22:46.094+0000 7faf4f45a080 -1 bluefs _flush_range allocated: 0x0 offset: 0x0 length: 0x2aac6 -1> 2023-01-31T22:22:46.101+0000 7faf4f45a080 -1 /builddir/build/BUILD/ceph-16.2.7/src/os/bluestore/BlueFS.cc: In function 'int BlueFS::_flush_range(BlueFS::FileWriter*, uint64_t, uint64_t)' thread 7faf4f45a080 time 2023-01-31T22:22:46.096073+0000 /builddir/build/BUILD/ceph-16.2.7/src/os/bluestore/BlueFS.cc: 2768: ceph_abort_msg("bluefs enospc") ceph version 16.2.7-126.el8cp (fe0af61d104d48cb9d116cde6e593b5fc8c197e4) pacific (stable) 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0xe5) [0x55c6b28b5562] 2: (BlueFS::_flush_range(BlueFS::FileWriter*, unsigned long, unsigned long)+0x1131) [0x55c6b2fb39e1] 3: (BlueFS::_flush(BlueFS::FileWriter*, bool, bool*)+0x90) [0x55c6b2fb3cc0] 4: (BlueFS::_fsync(BlueFS::FileWriter*, std::unique_lock<std::mutex>&)+0x159) [0x55c6b2fb6699] 5: (BlueRocksWritableFile::Sync()+0x6c) [0x55c6b2fde0dc] 6: (rocksdb::LegacyWritableFileWrapper::Sync(rocksdb::IOOptions const&, rocksdb::IODebugContext*)+0x1f) [0x55c6b34775ff] 7: (rocksdb::WritableFileWriter::SyncInternal(bool)+0x402) [0x55c6b3588d72] 8: (rocksdb::WritableFileWriter::Sync(bool)+0x88) [0x55c6b358a3b8] 9: (rocksdb::BuildTable(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, rocksdb::Env*, rocksdb::FileSystem*, rocksdb::ImmutableCFOptions const&, rocksdb::MutableCFOptions const&, rocksdb::FileOptions const&, rocksdb::TableCache*, rocksdb::InternalIteratorBase<rocksdb::Slice>*, std::vector<std::unique_ptr<rocksdb::FragmentedRangeTombstoneIterator, std::default_delete<rocksdb::FragmentedRangeTombstoneIterator> >, std::allocator<std::unique_ptr<rocksdb::FragmentedRangeTombstoneIterator, std::default_delete<rocksdb::FragmentedRangeTombstoneIterator> > > >, rocksdb::FileMetaData*, rocksdb::InternalKeyComparator const&, std::vector<std::unique_ptr<rocksdb::IntTblPropCollectorFactory, std::default_delete<rocksdb::IntTblPropCollectorFactory> >, std::allocator<std::unique_ptr<rocksdb::IntTblPropCollectorFactory, std::default_delete<rocksdb::IntTblPropCollectorFactory> > > > const*, unsigned int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<unsigned long, std::allocator<unsigned long> >, unsigned long, rocksdb::SnapshotChecker*, rocksdb::CompressionType, unsigned long, rocksdb::CompressionOptions const&, bool, rocksdb::InternalStats*, rocksdb::TableFileCreationReason, rocksdb::EventLogger*, int, rocksdb::Env::IOPriority, rocksdb::TableProperties*, int, unsigned long, unsigned long, rocksdb::Env::WriteLifeTimeHint, unsigned long)+0x2ddb) [0x55c6b365751b] 10: (rocksdb::DBImpl::WriteLevel0TableForRecovery(int, rocksdb::ColumnFamilyData*, rocksdb::MemTable*, rocksdb::VersionEdit*)+0xcf5) [0x55c6b34ba1e5] 11: (rocksdb::DBImpl::RecoverLogFiles(std::vector<unsigned long, std::allocator<unsigned long> > const&, unsigned long*, bool, bool*)+0x1c2e) [0x55c6b34bc91e] 12: (rocksdb::DBImpl::Recover(std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, bool, bool, bool, unsigned long*)+0xae8) [0x55c6b34bdc78] ------------------------------------- From the logs, fragmentation score is just 0.727513, but it shows unable to allocate ----- bluefs _allocate unable to allocate 0x20000 on bdev 1, allocator name block, allocator type hybrid, capacity 0x10000000000, block size 0x1000, free 0x26ddb37000, fragmentation 0.727513, allocated 0x0 ----- Is this a fragmentation issue? Is it possible to recover these OSDs or redeployment is the only option? * Version of all relevant components (if applicable): ODF 4.10 ceph version 16.2.7-126.el8cp (fe0af61d104d48cb9d116cde6e593b5fc8c197e4) pacific (stable) --- Additional comment from RHEL Program Management on 2023-02-13 04:35:54 UTC --- This bug having no release flag set previously, is now set with release flag 'odf‑4.13.0' to '?', and so is being proposed to be fixed at the ODF 4.13.0 release. Note that the 3 Acks (pm_ack, devel_ack, qa_ack), if any previously set while release flag was missing, have now been reset since the Acks are to be set against a release flag. --- Additional comment from RHEL Program Management on 2023-02-13 04:35:54 UTC --- Since this bug has severity set to 'urgent', it is being proposed as a blocker for the currently set release flag. Please resolve ASAP. --- Additional comment from Karun Josy on 2023-02-13 04:42:14 UTC --- + osd.0 and osd.2 are down --------------- sh-4.4$ ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 3.00000 root default -3 1.00000 host crpvlx1dnoiocs1-nychhc-org 0 hdd 1.00000 osd.0 down 0 1.00000 -7 1.00000 host crpvlx1dnoiocs2-nychhc-org 2 hdd 1.00000 osd.2 down 1.00000 1.00000 -5 1.00000 host crpvlx1dnoiocs3-nychhc-org 1 hdd 1.00000 osd.1 up 1.00000 1.00000 --------------- + tried to collect free-dump output, but looks like it is failing to collect --------------- # oc rsh rook-ceph-osd-2-7b9dcbfdd8-rnvpw ceph-bluestore-tool --path /var/lib/ceph/osd/ceph-${osdid} free-dump > free-dump_osd2.txt Defaulted container "osd" out of: osd, log-collector, blkdevmapper (init), activate (init), expand-bluefs (init), chown-container-data-dir (init) command terminated with exit code 137 [root@crpvlx1dnoihlp scripts]# cat free-dump_osd2.txt tcmalloc: large alloc 1073750016 bytes == 0x55cd954c0000 @ 0x7fa477171650 0x7fa4771922f4 0x55cd2effaf6a 0x7fa46bed4218 0x7fa46beded5b 0x7fa46beceda4 0x7fa46becf13b 0x7fa46dab5065 0x7fa46dab5105 0x55cd2f1e94a3 0x55cd2f1ff91b 0x55cd2f24f6a2 0x55cd2f1ffb6f 0x55cd2f20f8a9 0x55cd2f1e97ad 0x55cd2f031a62 0x7fa46db1b569 0x7fa46db1bb99 0x55cd2efcba63 0x7fa46b48ecf3 0x55cd2eff541e tcmalloc: large alloc 2147491840 bytes == 0x55cdd54c2000 @ 0x7fa477171650 0x7fa4771922f4 0x55cd2effaf6a 0x7fa46bed4218 0x7fa46beded5b 0x7fa46beceda4 0x7fa46dab5056 0x7fa46dab5105 0x55cd2f1e9500 0x55cd2f1ff91b 0x55cd2f24f6a2 0x55cd2f1ffb6f 0x55cd2f20f8a9 0x55cd2f1e97ad 0x55cd2f031a62 0x7fa46db1b569 0x7fa46db1bb99 0x55cd2efcba63 0x7fa46b48ecf3 0x55cd2eff541e tcmalloc: large alloc 1614938112 bytes == 0x55cd573f6000 @ 0x7fa477171650 0x7fa4771922f4 0x55cd2effc77d 0x55cd2effdaab 0x7fa46bed3b0f 0x7fa46dab56ab 0x7fa46dab2c37 0x7fa46db18bc8 0x55cd2f031ae8 0x7fa46db1b569 0x7fa46db1bb99 0x55cd2efcba63 0x7fa46b48ecf3 0x55cd2eff541e --------------- + ODF mustgather is available in supportshell Location : /cases/03430616/0010-must-gather.tar.gz/ Let me know if we need to capture any additional logs or outputs. --- Additional comment from Vikhyat Umrao on 2023-02-22 00:53:54 UTC --- Adam, can you please take a look? Thanks, Vikhyat Karun - please take a look in bz - https://bugzilla.redhat.com/show_bug.cgi?id=2139032 after comment#45 and I hope it will help. --- Additional comment from Sunil Kumar Acharya on 2023-03-19 17:58:34 UTC --- Please triage the BZ as per the instruction shared in the email : https://lists.corp.redhat.com/archives/rhocs-eng/2023-March/002833.html --- Additional comment from Adam Kupczyk on 2023-03-29 13:55:12 UTC --- 1/3 of the problem, which is a reporting of fragmentation will be fixed: https://github.com/ceph/ceph/pull/49885 1/3 of the problem, which is a BlueFS ENOSPC error is solved: https://github.com/ceph/ceph/pull/48854 1/3 the fragmentation itself is in the near future.