Updated to kernel 6.3.3 from updates testing repository. Several minutes after logging in again I began to get errors indicating that some files could not be written. Shutdown with systemctl and while rebooting a filesystem error was detected and the system dropped to single user rescue mode. This message was in the journal: kernel: XFS (sdd1): Metadata corruption detected at xfs_buf_ioend+0x154/0x5f0 [xfs], xfs_inode block 0x897ed60 xfs_inode_buf_verify kernel: XFS (sdd1): Unmount and run xfs_repair kernel: XFS (sdd1): First 128 bytes of corrupted metadata buffer: kernel: 00000000: 41 75 74 6f 6d 61 74 69 6f 6e 43 6f 6e 74 72 6f AutomationContro kernel: 00000010: 6c 45 45 53 61 49 53 34 5f 45 45 43 34 45 4f 53 lEESaIS4_EEC4EOS kernel: 00000020: 61 49 53 74 31 30 5f 4c 69 73 74 5f 6e 6f 64 65 aISt10_List_node kernel: 00000030: 49 53 34 5f 45 45 00 4e 6f 42 61 73 65 48 6f 6f IS4_EE.NoBaseHoo kernel: 00000040: 6b 49 64 00 72 65 76 65 72 73 65 5f 69 74 65 72 kId.reverse_iter kernel: 00000050: 61 74 6f 72 3c 73 74 64 3a 3a 5f 4c 69 73 74 5f ator<std::_List_ kernel: 00000060: 63 6f 6e 73 74 5f 69 74 65 72 61 74 6f 72 3c 73 const_iterator<s kernel: 00000070: 69 67 63 3a 3a 69 6e 74 65 72 6e 61 6c 3a 3a 74 igc::internal::t kernel: XFS (sdd1): metadata I/O error in "xfs_imap_to_bp+0x5c/0x80 [xfs]" at daddr 0x897ed60 len 32 error 117 I attempted to run xfs_repair on the device (it happened to be the device mounted as /home, so the system files all loaded correct from /), but xfs_repair reported that there were dirty entries in the journal, so I should mount to let the journal entries replay. Mount failed with an error so I had to run xfs_repair -L to ignore the log replay. After running xfs_repair -L I ran xfs_repair with no additional options to verify that the filesystem was marked as clean, then rebooted. After logging in the same error occurred within a few minutes. Did the same xfs_repair, had a journal/log entry, tried to mount, couldn't mount, xfs_repair -L, xfs_repair two more times to make sure the filesystem was OK, rebooted again. After logging in the same error occurred within a few minutes. Ran xfs_repair -L etc. from rescue mode, rebooted back to previous 6.2.14 kernel. Has been running around 12 hours with no errors so far, so seems highly likely that something in 6.3.3 is choking on my XFS filesystem. I have three devices formatted as XFS, /, a data volume on /mnt/data, and /home. It is possible that /home was the only filesystem getting much write data right after boot, but I would assume that /var has a lot of writes as well (the only volume not xfs is /boot formatted as ext4). Reproducible: Always Steps to Reproduce: 1. Upgrade to kernel 6.3.3 from updates-testing on system with xfs filesystems 2. Reboot to 6.3.3 3. Actual Results: Filesystem errors Expected Results: Filesystem which works with 6.2.14 continues to work with 6.3.3
A couple of questions. A) What mount options are you using for home? B) What is the drive health like? smartctl should be able to help here. c) Do you recall when the filesystem was made? Fedora version/kernel version/year some ballpark.
(In reply to Justin M. Forbes from comment #1) > A) What mount options are you using for home? UUID=60cf5755-8094-46d2-a297-c0b4c1ce03d9 /home xfs defaults 0 2 > B) What is the drive health like? smartctl should be able to help here. The "drive" is actually a logical volume on an 8-drive RAID 5 array. All the physical drives and logical volumes are reported OK by the array controller (HP SmartArray P420) => ctrl slot=2 pd all show status physicaldrive 1I:2:1 (port 1I:box 2:bay 1, 300 GB): OK physicaldrive 1I:2:2 (port 1I:box 2:bay 2, 300 GB): OK physicaldrive 1I:2:3 (port 1I:box 2:bay 3, 300 GB): OK physicaldrive 1I:2:4 (port 1I:box 2:bay 4, 300 GB): OK physicaldrive 2I:2:5 (port 2I:box 2:bay 5, 300 GB): OK physicaldrive 2I:2:6 (port 2I:box 2:bay 6, 300 GB): OK physicaldrive 2I:2:7 (port 2I:box 2:bay 7, 300 GB): OK physicaldrive 2I:2:8 (port 2I:box 2:bay 8, 300 GB): OK => ctrl slot=2 ld all show status logicaldrive 1 (597 MB, RAID 5): OK <-- this is /boot ext4 logicaldrive 2 (4.0 GB, RAID 5): OK <-- this is swap logicaldrive 3 (100.0 GB, RAID 5): OK <-- this is / xfs logicaldrive 4 (100.0 GB, RAID 5): OK <-- this is /home xfs logicaldrive 5 (1.7 TB, RAID 5): OK <-- this is /mnt/data xfs > c) Do you recall when the filesystem was made? Fedora version/kernel > version/year some ballpark. The /etc/fstab file was created in June 2018, so I would expect that is when all the filesystems were first formatted.
Minor correction: my currently running kernel is 6.2.15, I had already updated from 6.2.14, so the note above about going from 6.3.3 back to 6.2.14 should actually be went from 6.3.3 back to 6.2.15.
Mind giving https://koji.fedoraproject.org/koji/taskinfo?taskID=101466330 a try to see if that fixes things for you?
(In reply to Justin M. Forbes from comment #4) > Mind giving https://koji.fedoraproject.org/koji/taskinfo?taskID=101466330 a > try to see if that fixes things for you? Sorry, I don't follow. That link seems to be to a src rpm for the kernel. Are you asking me to rebuild the kernel rpms with a configuration change?
Sorry, that was the top level build, clicking on the x86_64 task takes you to https://koji.fedoraproject.org/koji/taskinfo?taskID=101466371 which links the actual built rpms.
So it seems that my backport was not the culprit, and people are still able to reproduce after a revert. The next question is whether this reproduces on 6.4-rc3 or not. Is it already fixed upstream, and the fix needs to be backported? Please try: https://koji.fedoraproject.org/koji/buildinfo?buildID=2204155
Just for additional discussion on this issue: https://www.spinics.net/lists/linux-xfs/msg71084.html and https://bodhi.fedoraproject.org/updates/FEDORA-2023-514965dd8a
We need to know the full hardware and storage config of the machines hitting this problem - CPUs, memory, storage devices. sotrage device configs, storage stack layout, filesystem configs (xfs_info output), mount options, etc. This will help us find commonalities between teh environments that are seeing the issue. We need the full dmesg output from boot time when a corruption is hit. Please do not chop bits out of the logs that you think we don't need, that always results in the bits we actually need getting chopped out. Can you please turn /proc/fs/xfs/error_level up to 11 before reproducing the issue. This will dump more infromationin teh logs when certain XFS errors are hit. If the problem does not take down the root filesystem, can you please unmount the filesytsem that was shut down and take a metadump of it. PLease compress the metadump and provide a link for us to access it. This will allow us to examine the state of the filesystem on disk at teh time of the corruption. We also need a description of teh workload that is being run when these problems are hit. That will tell us how the filesytem is being modified in memory when the corruption is hit. It will also give us an idea of what the difference is between the on-disk state in teh metadump and the in-memory state when teh corruption is detected. This will give us some idea where the corrupted metadata is coming from. -Dave.
Hi! All our "crawling servers" seems to crash consistently when using this kernel, and it seems to always be the root partition crashing. The crawling servers has high load and basically runs many instances of Google Chrome. Here is an attempt to provide information about a particular crash. This is a server installed as Fedora 31 and upgraded to Fedora 38 step by step. It has worked fine till now. After booting to kernel 6.3.3, the file system crashed almost immediately. We did not have to start the processes. At other occasions it has taken longer, up to a day, and on those the file system is created more recently. It's just a hunch, we do not have that many observations. I believe we have 8 crashes totally now of 12 servers running this kernel for some time. Here are links to what you hopefully requested for the particular crash: https://pascal.opoint.com/kernel6.3/fstab https://pascal.opoint.com/kernel6.3/hardwareinfo https://pascal.opoint.com/kernel6.3/kernellog https://pascal.opoint.com/kernel6.3/metadump Please let me know if we can do anything else to help. /Rune
I think I noticed that Rune & Chris both have HP 420 raid controllers. Is this correct and I assume these are independent reports?
More questions - Rune, what was the previous kernel you were running without this problem? Chris's report said the last good kernel was 6.2.14. Narrowing that down might help. Talking to Dave, we also wonder how these raid controllers are configured. They advertise no write cache, can you confirm that the drives behind these controllers also have write caches disabled, and/or that their battery backup (if they have one) is performing correctly?
(In reply to Rune Kleveland from comment #10) > Hi! > > All our "crawling servers" seems to crash consistently when using this > kernel, and it seems to always be the root partition crashing. The crawling > servers has high load and basically runs many instances of Google Chrome. > > Here is an attempt to provide information about a particular crash. This is > a server installed as Fedora 31 and upgraded to Fedora 38 step by step. It > has worked fine till now. > > After booting to kernel 6.3.3, the file system crashed almost immediately. > We did not have to start the processes. At other occasions it has taken > longer, up to a day, and on those the file system is created more recently. > It's just a hunch, we do not have that many observations. I believe we have > 8 crashes totally now of 12 servers running this kernel for some time. > > Here are links to what you hopefully requested for the particular crash: > > https://pascal.opoint.com/kernel6.3/fstab > https://pascal.opoint.com/kernel6.3/hardwareinfo > https://pascal.opoint.com/kernel6.3/kernellog > https://pascal.opoint.com/kernel6.3/metadump So this has failed at runtime because it tripped over an inode block full of garbage: May 24 12:04:12 newton59.opoint.com kernel: XFS (sda3): Metadata corruption detected at xfs_buf_ioend+0x154/0x5f0 [xfs], xfs_inode block 0x112ac740 xfs_inode_buf_verify May 24 12:04:12 newton59.opoint.com kernel: XFS (sda3): Unmount and run xfs_repair May 24 12:04:12 newton59.opoint.com kernel: XFS (sda3): First 128 bytes of corrupted metadata buffer: May 24 12:04:12 newton59.opoint.com kernel: 00000000: 31 45 20 24 41 41 46 46 32 44 39 46 38 44 43 43 1E $AAFF2D9F8DCC May 24 12:04:12 newton59.opoint.com kernel: 00000010: 46 41 30 32 33 31 33 33 36 36 41 38 36 41 46 46 FA02313366A86AFF May 24 12:04:12 newton59.opoint.com kernel: 00000020: 38 37 35 33 42 41 38 31 41 38 39 46 20 24 41 43 8753BA81A89F $AC May 24 12:04:12 newton59.opoint.com kernel: 00000030: 31 34 44 37 37 37 33 42 46 41 31 44 32 35 45 34 14D7773BFA1D25E4 May 24 12:04:12 newton59.opoint.com kernel: 00000040: 43 46 42 39 34 36 34 38 46 30 42 43 38 39 33 44 CFB94648F0BC893D May 24 12:04:12 newton59.opoint.com kernel: 00000050: 44 31 39 45 33 37 20 24 41 44 41 44 41 31 44 45 D19E37 $ADADA1DE May 24 12:04:12 newton59.opoint.com kernel: 00000060: 46 30 34 44 45 44 37 41 36 42 42 33 32 43 38 30 F04DED7A6BB32C80 May 24 12:04:12 newton59.opoint.com kernel: 00000070: 44 36 44 43 35 39 33 42 35 42 43 30 38 39 45 46 D6DC593B5BC089EF Note the address of this block: 0x112ac740 It is also tripping over a directory data block full of garbage: May 24 12:04:12 newton59.opoint.com kernel: XFS (sda3): metadata I/O error in "xfs_imap_to_bp+0x5c/0x80 [xfs]" at daddr 0x112ac740 len 32 error 117 May 24 12:06:11 newton59.opoint.com kernel: XFS (sda3): Metadata CRC error detected at xfs_dir3_block_read_verify+0xd4/0x110 [xfs], xfs_dir3_block block 0x112a9be0 May 24 12:06:11 newton59.opoint.com kernel: XFS (sda3): Unmount and run xfs_repair May 24 12:06:11 newton59.opoint.com kernel: XFS (sda3): First 128 bytes of corrupted metadata buffer: May 24 12:06:11 newton59.opoint.com kernel: 00000000: 35 44 33 43 31 39 30 35 42 43 20 24 46 45 41 46 5D3C1905BC $FEAF May 24 12:06:11 newton59.opoint.com kernel: 00000010: 30 41 42 30 39 30 38 35 33 31 35 43 44 33 39 35 0AB09085315CD395 May 24 12:06:12 newton59.opoint.com kernel: 00000020: 46 35 31 31 43 44 45 45 39 31 30 32 34 36 32 46 F511CDEE9102462F May 24 12:06:12 newton59.opoint.com kernel: 00000030: 35 38 39 35 0a 70 20 61 63 63 65 70 74 20 34 33 5895.p accept 43 May 24 12:06:12 newton59.opoint.com kernel: 00000040: 2c 35 33 2c 38 30 2c 31 31 30 2c 31 34 33 2c 31 ,53,80,110,143,1 May 24 12:06:12 newton59.opoint.com kernel: 00000050: 39 34 2c 32 32 30 2c 34 34 33 2c 38 37 33 2c 39 94,220,443,873,9 May 24 12:06:12 newton59.opoint.com kernel: 00000060: 39 31 2c 39 39 33 2d 39 39 35 2c 31 31 39 34 2c 91,993-995,1194, May 24 12:06:12 newton59.opoint.com kernel: 00000070: 31 32 39 33 2c 33 36 39 30 2c 34 33 32 31 2c 35 1293,3690,4321,5 Note the address of this block: 0x112a9be0 And when I try to replay the log from the metadata to mount the image, it's trips over a freespace btree block full of garbage at block 0x112a9b08 and goes no further. Lets put those three corruptions together: 0x112ac740, 0x112a9be0 and 0x112a9b08. They are all within a range of ~5.8MB of disk space. That's highly suspicious. So to do a quick scan of the state of all the metadata, I ran 'xfs_repair -n dump.img' to see what active metadata blocks it found that were corrupt: $ grep Metadata t.t Metadata CRC error detected at 0x5625c07feecd, xfs_bnobt block 0x112a9b08/0x1000 Metadata CRC error detected at 0x5625c07feecd, xfs_bnobt block 0x112ac150/0x1000 Metadata CRC error detected at 0x5625c07feecd, xfs_bnobt block 0x112a9bc0/0x1000 Metadata CRC error detected at 0x5625c07feecd, xfs_bnobt block 0x112a9b90/0x1000 Metadata CRC error detected at 0x5625c07feecd, xfs_bnobt block 0x112a9b40/0x1000 Metadata CRC error detected at 0x5625c07feecd, xfs_cntbt block 0x112ac140/0x1000 Metadata CRC error detected at 0x5625c07feecd, xfs_cntbt block 0x112ab920/0x1000 Metadata CRC error detected at 0x5625c07feecd, xfs_cntbt block 0x112aa3c0/0x1000 Metadata CRC error detected at 0x5625c07feecd, xfs_cntbt block 0x112a9c80/0x1000 Metadata CRC error detected at 0x5625c07feecd, xfs_cntbt block 0x112ac158/0x1000 Metadata CRC error detected at 0x5625c07feecd, xfs_cntbt block 0x112a9828/0x1000 Metadata CRC error detected at 0x5625c07feecd, xfs_cntbt block 0x112aad30/0x1000 Metadata corruption detected at 0x5625c07eb846, xfs_inode block 0x112aa3a0/0x4000 Metadata corruption detected at 0x5625c07eb846, xfs_inode block 0x112aa920/0x4000 Metadata corruption detected at 0x5625c07eb846, xfs_inode block 0x112aac00/0x4000 Metadata corruption detected at 0x5625c07eb846, xfs_inode block 0x112aac20/0x4000 Metadata corruption detected at 0x5625c07eb846, xfs_inode block 0x112abac0/0x4000 Metadata corruption detected at 0x5625c07eb846, xfs_inode block 0x112ab980/0x4000 Metadata corruption detected at 0x5625c07eb846, xfs_inode block 0x112ac1c0/0x4000 Metadata corruption detected at 0x5625c07eb846, xfs_inode block 0x112ac2c0/0x4000 Metadata corruption detected at 0x5625c07eb846, xfs_inode block 0x112ac2e0/0x4000 Metadata corruption detected at 0x5625c07eb846, xfs_inode block 0x112ac740/0x4000 Metadata corruption detected at 0x5625c07eb846, xfs_inode block 0x112ac760/0x4000 Metadata CRC error detected at 0x5625c08176c3, xfs_dir3_block block 0x112a9b20/0x1000 Metadata CRC error detected at 0x5625c08176c3, xfs_dir3_block block 0x112a9d50/0x1000 Metadata CRC error detected at 0x5625c08176c3, xfs_dir3_block block 0x112a9d58/0x1000 Metadata CRC error detected at 0x5625c08176c3, xfs_dir3_block block 0x112a9d48/0x1000 Metadata CRC error detected at 0x5625c08176c3, xfs_dir3_block block 0x112a9be0/0x1000 Metadata CRC error detected at 0x5625c08176c3, xfs_dir3_block block 0x112ab468/0x1000 Metadata CRC error detected at 0x5625c08177a3, xfs_dir3_data block 0x112a97d0/0x1000 Metadata CRC error detected at 0x5625c08177a3, xfs_dir3_data block 0x112a97e8/0x1000 Metadata CRC error detected at 0x5625c08176c3, xfs_dir3_block block 0x112aa998/0x1000 Metadata CRC error detected at 0x5625c08176c3, xfs_dir3_block block 0x112ab918/0x1000 Metadata CRC error detected at 0x5625c08176c3, xfs_dir3_block block 0x112ab900/0x1000 Yup, there were are. THere are lots of corrupt metadata blocks in the range from daddr 0x112a9820 to 0x112ab920. Metadata corruption is limited to a range of ~2100 sectors or just on 11MB of the entire block device space (of 300GB!). All the metadata blocks in the range have exactly the same sort of ascii data in them, which indicates it was probably a single corruption event that occurred. Note that the range of the corruption is likely larger than 2100 sectors - metadump only captures the metadata blocks and not the file data or empty space in teh filesystem so teh corruption likely extends further. There's a good chance that file data was also corrupted by whatever occurred. I don't recognise what the data in teh metadata blocks is - does anyoe else? As it stands, I'm having trouble seeing how this is being caused by an XFS. All of these metadata blocks are allocated and written independently at different times, so nothign XFs does would cause them all to be overwritten at the same time. I also can't see how it is failure to persist data/metadata because that leaves previous versions of the metadata behind and that's clearly not the case here. This looks much more like a misdirected user data write has occurred than anything else. Now, XFS could map a data write incorrectly and submit it to the wrong place, but that implies metadata in memory has already been corrupted and we haven't tripped over it in any way before the user data write occurs. This could happen if there was some kind of kernel memory corruption occuring outside XFS, too. However, given the absense of any evidence that XFS is actually corrupted in memory or that any other sort of memory corruption has been occuring, there's no real evidence to point to XFS mapping data writes over the top of it's own metadata. So I'm not ruling out that it is an XFS problem, it just looks unlikely at this point. We'll see as more data comes in. Next question: does this problem still occur on a 6.4-rc3 kernel? Has anyone tested that yet? -Dave.
My report is independent. We do have P420 controllers with firmware 8.32. It also happened on a P830i controller with firmware 5.02. The latest good kernel id 6.2.15 for sure, we run it on 80 servers now. One of the servers with this problem had caching disabled due to bad battery, the others had caching enabled. Here is the output from ssacli 5.10 on two servers: ------------------------------------------ ssacli ctrl all show config detail Smart Array P420i in Slot 0 (Embedded) Bus Interface: PCI Slot: 0 Serial Number: 00143802604ACA0 Cache Serial Number: PBKUD0BRH6WF7H RAID 6 Status: Enabled Controller Status: OK Hardware Revision: B Firmware Version: 8.32 Firmware Supports Online Firmware Activation: False Rebuild Priority: Low Expand Priority: Medium Surface Scan Delay: 15 secs Surface Scan Mode: Idle Parallel Surface Scan Supported: No Queue Depth: Automatic Monitor and Performance Delay: 60 min Elevator Sort: Enabled Degraded Performance Optimization: Disabled Inconsistency Repair Policy: Disabled Wait for Cache Room: Disabled Surface Analysis Inconsistency Notification: Disabled Post Prompt Timeout: 0 secs Cache Board Present: True Cache Status: Permanently Disabled Cache Status Details: Cache disabled; backup power source failed to charge to an acceptable level Cache Disable Reason: Permanent disable condition. The posted write cache has been disabled because the backup power source attached to the flash-backed write cache module has failed to charge. Drive Write Cache: Disabled Total Cache Size: 2.0 Total Cache Memory Available: 1.8 Battery Backed Cache Size: 1.8 No-Battery Write Cache: Disabled SSD Caching RAID5 WriteBack Enabled: False SSD Caching Version: 1 Cache Backup Power Source: Capacitors Battery/Capacitor Count: 1 Battery/Capacitor Status: Failed (Replace Batteries) SATA NCQ Supported: True Spare Activation Mode: Activate on physical drive failure (default) Controller Temperature (C): 77 Cache Module Temperature (C): 56 Capacitor Temperature (C): 23 Number of Ports: 2 Internal only Encryption: Not Set Driver Name: hpsa Driver Version: 3.4.20 Driver Supports SSD Smart Path: True PCI Address (Domain:Bus:Device.Function): 0000:02:00.0 Port Max Phy Rate Limiting Supported: False Host Serial Number: CZJ3170HPG Sanitize Erase Supported: False Primary Boot Volume: None Secondary Boot Volume: None Internal Drive Cage at Port 1I, Box 1, OK Drive Bays: 4 Port: 1I Box: 1 Location: Internal Physical Drives physicaldrive 1I:1:1 (port 1I:box 1:bay 1, SAS HDD, 300 GB, OK) physicaldrive 1I:1:2 (port 1I:box 1:bay 2, SAS HDD, 300 GB, OK) Internal Drive Cage at Port 2I, Box 0, OK Drive Bays: 4 Port: 2I Box: 0 Location: Internal Physical Drives None attached Port Name: 1I Port ID: 0 Port Connection Number: 0 SAS Address: 500143802604ACA0 Port Location: Internal Port Name: 2I Port ID: 1 Port Connection Number: 1 SAS Address: 500143802604ACA4 Port Location: Internal Array: A Interface Type: SAS Unused Space: 0 MB (0.00%) Used Space: 558.73 GB (100.00%) Status: OK Array Type: Data Smart Path: disable Logical Drive: 1 Size: 279.37 GB Fault Tolerance: 1 Heads: 255 Sectors Per Track: 32 Cylinders: 65535 Strip Size: 256 KB Full Stripe Size: 256 KB Status: OK Unrecoverable Media Errors: None Caching: Disabled Unique Identifier: 600508B1001CCB753BAE91CE7C5952BB Disk Name: /dev/sda Mount Points: 278.3 GiB Partition 2, 1024 MiB Partition 1 /, /boot Disk Partition Information Partition 2: Basic, 278.3 GiB, / Partition 1: Basic, 1024 MiB, /boot Logical Drive Label: ADA98A1400143802604ACA047C0 Mirror Group 1: physicaldrive 1I:1:1 (port 1I:box 1:bay 1, SAS HDD, 300 GB, OK) Mirror Group 2: physicaldrive 1I:1:2 (port 1I:box 1:bay 2, SAS HDD, 300 GB, OK) Drive Type: Data LD Acceleration Method: All disabled physicaldrive 1I:1:1 Port: 1I Box: 1 Bay: 1 Status: OK Drive Type: Data Drive Interface Type: SAS Size: 300 GB Drive exposed to OS: False Logical/Physical Block Size: 512/512 Rotational Speed: 10000 Firmware Revision: SB2B Serial Number: EB71PCB1ALNW WWID: 500003946818356A Model: IBM-ESXSMBF2300RC Current Temperature (C): 28 PHY Count: 2 PHY Transfer Rate: 6.0Gbps, Unknown PHY Physical Link Rate: Unknown, Unknown PHY Maximum Link Rate: Unknown, Unknown Drive Authentication Status: OK Carrier Application Version: 11 Carrier Bootloader Version: 6 Sanitize Erase Supported: False Shingled Magnetic Recording Support: None physicaldrive 1I:1:2 Port: 1I Box: 1 Bay: 2 Status: OK Drive Type: Data Drive Interface Type: SAS Size: 300 GB Drive exposed to OS: False Logical/Physical Block Size: 512/512 Rotational Speed: 10000 Firmware Revision: SB2B Serial Number: EB71PCB1ALNB WWID: 50000394681830A2 Model: IBM-ESXSMBF2300RC Current Temperature (C): 28 PHY Count: 2 PHY Transfer Rate: 6.0Gbps, Unknown PHY Physical Link Rate: Unknown, Unknown PHY Maximum Link Rate: Unknown, Unknown Drive Authentication Status: OK Carrier Application Version: 11 Carrier Bootloader Version: 6 Sanitize Erase Supported: False Shingled Magnetic Recording Support: None SEP (Vendor ID PMCSIERA, Model SRCv8x6G) 380 Device Number: 380 Firmware Version: RevB WWID: 500143802604ACAF Vendor ID: PMCSIERA Model: SRCv8x6G ------------------------------------------- [root@newton62 mrexec]# ssacli ctrl all show config detail Smart Array P420i in Slot 0 (Embedded) Bus Interface: PCI Slot: 0 Serial Number: 001438022023B20 Cache Serial Number: PBKUC0BRHA909B RAID 6 Status: Enabled Controller Status: OK Hardware Revision: B Firmware Version: 8.32 Firmware Supports Online Firmware Activation: False Rebuild Priority: Low Expand Priority: Medium Surface Scan Delay: 15 secs Surface Scan Mode: Idle Parallel Surface Scan Supported: No Queue Depth: Automatic Monitor and Performance Delay: 60 min Elevator Sort: Enabled Degraded Performance Optimization: Disabled Inconsistency Repair Policy: Disabled Wait for Cache Room: Disabled Surface Analysis Inconsistency Notification: Disabled Post Prompt Timeout: 0 secs Cache Board Present: True Cache Status: OK Cache Ratio: 10% Read / 90% Write Drive Write Cache: Disabled Total Cache Size: 1.0 Total Cache Memory Available: 0.8 Battery Backed Cache Size: 0.8 No-Battery Write Cache: Disabled SSD Caching RAID5 WriteBack Enabled: False SSD Caching Version: 1 Cache Backup Power Source: Capacitors Battery/Capacitor Count: 1 Battery/Capacitor Status: OK SATA NCQ Supported: True Spare Activation Mode: Activate on physical drive failure (default) Controller Temperature (C): 81 Cache Module Temperature (C): 55 Capacitor Temperature (C): 24 Number of Ports: 2 Internal only Encryption: Not Set Driver Name: hpsa Driver Version: 3.4.20 Driver Supports SSD Smart Path: True PCI Address (Domain:Bus:Device.Function): 0000:02:00.0 Port Max Phy Rate Limiting Supported: False Host Serial Number: CZJ"400Dc Sanitize Erase Supported: False Primary Boot Volume: None Secondary Boot Volume: None Internal Drive Cage at Port 1I, Box 1, OK Drive Bays: 4 Port: 1I Box: 1 Location: Internal Physical Drives physicaldrive 1I:1:1 (port 1I:box 1:bay 1, SAS HDD, 300 GB, OK) physicaldrive 1I:1:2 (port 1I:box 1:bay 2, SAS HDD, 300 GB, OK) Internal Drive Cage at Port 2I, Box 0, OK Drive Bays: 4 Port: 2I Box: 0 Location: Internal Physical Drives None attached Port Name: 1I Port ID: 0 Port Connection Number: 0 SAS Address: 5001438022023B20 Port Location: Internal Port Name: 2I Port ID: 1 Port Connection Number: 1 SAS Address: 5001438022023B24 Port Location: Internal Array: A Interface Type: SAS Unused Space: 0 MB (0.00%) Used Space: 558.73 GB (100.00%) Status: OK Array Type: Data Smart Path: disable Logical Drive: 1 Size: 279.37 GB Fault Tolerance: 1 Heads: 255 Sectors Per Track: 32 Cylinders: 65535 Strip Size: 256 KB Full Stripe Size: 256 KB Status: OK Unrecoverable Media Errors: None Caching: Enabled Unique Identifier: 600508B1001CFEA859297B81728FCD49 Disk Name: /dev/sda Mount Points: 274.3 GiB Partition 3, 1024 MiB Partition 1 /, /boot Disk Partition Information Partition 3: Basic, 274.3 GiB, / Partition 1: Basic, 1024 MiB, /boot Logical Drive Label: ADA8464D5001438022023B20 9070 Mirror Group 1: physicaldrive 1I:1:1 (port 1I:box 1:bay 1, SAS HDD, 300 GB, OK) Mirror Group 2: physicaldrive 1I:1:2 (port 1I:box 1:bay 2, SAS HDD, 300 GB, OK) Drive Type: Data LD Acceleration Method: Controller Cache physicaldrive 1I:1:1 Port: 1I Box: 1 Bay: 1 Status: OK Drive Type: Data Drive Interface Type: SAS Size: 300 GB Drive exposed to OS: False Logical/Physical Block Size: 512/512 Rotational Speed: 10000 Firmware Revision: SB28 Serial Number: EB71PB90J2KH WWID: 50000393881275F6 Model: IBM-ESXSMBF2300RC Current Temperature (C): 30 PHY Count: 2 PHY Transfer Rate: 6.0Gbps, Unknown PHY Physical Link Rate: Unknown, Unknown PHY Maximum Link Rate: Unknown, Unknown Drive Authentication Status: OK Carrier Application Version: 11 Carrier Bootloader Version: 6 Sanitize Erase Supported: False Shingled Magnetic Recording Support: None physicaldrive 1I:1:2 Port: 1I Box: 1 Bay: 2 Status: OK Drive Type: Data Drive Interface Type: SAS Size: 300 GB Drive exposed to OS: False Logical/Physical Block Size: 512/512 Rotational Speed: 10000 Firmware Revision: HPDA Serial Number: PMVV8RBB WWID: 5000CCA0212FC98D Model: HP EG0300FBDBR Current Temperature (C): 28 Maximum Temperature (C): 45 PHY Count: 2 PHY Transfer Rate: 6.0Gbps, Unknown PHY Physical Link Rate: Unknown, Unknown PHY Maximum Link Rate: Unknown, Unknown Drive Authentication Status: OK Carrier Application Version: 11 Carrier Bootloader Version: 6 Sanitize Erase Supported: False Shingled Magnetic Recording Support: None SEP (Vendor ID PMCSIERA, Model SRCv8x6G) 380 Device Number: 380 Firmware Version: RevB WWID: 5001438022023B2F Vendor ID: PMCSIERA Model: SRCv8x6G ------------------------------------------
Rune, mind giving that rawhide kernel a try to see if 6.4-rc3 shows the issue too? There is a link to the koji build in comment 7.
I tested the 6.4 kernel for an hour, and nothing bad was observed in the kernel log. Everything seems to work after a reboot also.
Rune - just for what it's worth, there's file data in the metadata, for example: 00000b20 2d 2d 2d 2d 42 45 47 49 4e 20 52 53 41 20 50 55 |----BEGIN RSA PU| 00000b30 42 4c 49 43 20 4b 45 59 2d 2d 2d 2d 2d 0a 4d 49 |BLIC KEY-----.MI| 00000b40 47 4a 41 6f 47 42 41 4e 30 55 2f 68 44 6d 77 79 |GJAoGBAN0U/hDmwy| as well as maybe some Tor onion keys? (might take down the URL if any of that concerns you)
Just for fun, I grabbed a HP ProLiant DL360 with a P420i RAID, migrated from the 6.2 kernel to the 6.3.3 kernel with various writes along the way and saw no problems after the upgrade.
Thanks for the info about file data. I took down the URL, but I am not concerned. If anyone needs the metadump, please let me know. It's weird that I can reproduce so easily and you can't. Please contact me directly if you want access to a server where I think you can reproduce easily, and I will try to arrange that.
Another data point. After about 14 days of uptime with kernel 6.3.1: May 24 15:14:59 dvr0.lan kernel: XFS (sdc1): Reverse Mapping BTree record corruption in AG 9 detected! May 24 15:14:59 dvr0.lan kernel: XFS (sdc1): Owner 0x3c0000082, flags 0x4, start block 0x558f681 block count 0x0 May 24 15:14:59 dvr0.lan kernel: XFS (sdc1): Corruption of in-memory data (0x8) detected at xfs_defer_finish_noroll+0x285/0x690 [xfs] (fs/xfs/libxfs/xfs_defer.c:573). Shutting down filesystem. May 24 15:14:59 dvr0.lan kernel: XFS (sdc1): Please unmount the filesystem and rectify the problem(s) xfs_repair failed (said to mount to replay log) Attempt to mount: May 24 19:21:19 dvr0.lan kernel: XFS (sdc1): Mounting V5 Filesystem c80e09b1-f667-4d9f-b6c7-0a8989574238 May 24 19:21:19 dvr0.lan kernel: XFS (sdc1): Starting recovery (logdev: internal) May 24 19:21:19 dvr0.lan kernel: 00000000: 40 12 01 00 01 00 00 00 30 98 41 94 3b 99 ff ff @.......0.A.;... May 24 19:21:19 dvr0.lan kernel: XFS (sdc1): Internal error xfs_rui_item_recover at line 502 of file fs/xfs/xfs_rmap_item.c. Caller xlog_recover_p rocess_intents+0xa3/0x300 [xfs] May 24 19:21:19 dvr0.lan kernel: CPU: 3 PID: 165795 Comm: mount Not tainted 6.3.1-200.fc38.x86_64 #1 May 24 19:21:19 dvr0.lan kernel: Hardware name: /DZ68BC, BIOS BCZ6810H.86A.0039.2012.1018.1115 10/18/2012 May 24 19:21:19 dvr0.lan kernel: Call Trace: May 24 19:21:19 dvr0.lan kernel: <TASK> May 24 19:21:19 dvr0.lan kernel: dump_stack_lvl+0x47/0x60 May 24 19:21:19 dvr0.lan kernel: xfs_corruption_error+0x94/0xa0 [xfs] May 24 19:21:19 dvr0.lan kernel: ? xlog_recover_process_intents+0xa3/0x300 [xfs] May 24 19:21:19 dvr0.lan kernel: xfs_rui_item_recover+0x142/0x410 [xfs] May 24 19:21:19 dvr0.lan kernel: ? xlog_recover_process_intents+0xa3/0x300 [xfs] May 24 19:21:19 dvr0.lan kernel: ? xfs_read_agf+0x94/0x130 [xfs] May 24 19:21:19 dvr0.lan kernel: ? xfs_buf_read_map+0x62/0x2a0 [xfs] May 24 19:21:19 dvr0.lan kernel: ? xlog_recover_process_intents+0xa3/0x300 [xfs] May 24 19:21:19 dvr0.lan kernel: ? __pfx_xfs_rui_item_recover+0x10/0x10 [xfs] May 24 19:21:19 dvr0.lan kernel: xlog_recover_process_intents+0xa3/0x300 [xfs] May 24 19:21:19 dvr0.lan kernel: ? xfs_read_agf+0x94/0x130 [xfs] May 24 19:21:19 dvr0.lan kernel: ? preempt_count_add+0x6e/0xa0 May 24 19:21:19 dvr0.lan kernel: ? lock_timer_base+0x61/0x80 May 24 19:21:19 dvr0.lan kernel: xlog_recover_finish+0x31/0x340 [xfs] May 24 19:21:19 dvr0.lan kernel: xfs_log_mount_finish+0x106/0x180 [xfs] May 24 19:21:19 dvr0.lan kernel: xfs_mountfs+0x51f/0x900 [xfs] May 24 19:21:19 dvr0.lan kernel: xfs_fs_fill_super+0x4ca/0x920 [xfs] May 24 19:21:19 dvr0.lan kernel: ? __pfx_xfs_fs_fill_super+0x10/0x10 [xfs] May 24 19:21:19 dvr0.lan kernel: get_tree_bdev+0x174/0x260 May 24 19:21:19 dvr0.lan kernel: vfs_get_tree+0x29/0xd0 May 24 19:21:19 dvr0.lan kernel: path_mount+0x4a1/0xae0 May 24 19:21:19 dvr0.lan kernel: __x64_sys_mount+0x11a/0x150 May 24 19:21:19 dvr0.lan kernel: do_syscall_64+0x60/0x90 May 24 19:21:19 dvr0.lan kernel: ? syscall_exit_to_user_mode+0x1b/0x40 May 24 19:21:19 dvr0.lan kernel: ? do_syscall_64+0x6c/0x90 May 24 19:21:19 dvr0.lan kernel: ? syscall_exit_to_user_mode+0x1b/0x40 May 24 19:21:19 dvr0.lan kernel: ? do_syscall_64+0x6c/0x90 May 24 19:21:19 dvr0.lan kernel: entry_SYSCALL_64_after_hwframe+0x72/0xdc May 24 19:21:19 dvr0.lan kernel: RIP: 0033:0x7f97037d3a9e May 24 19:21:19 dvr0.lan kernel: Code: 48 8b 0d 6d 03 0c 00 f7 d8 64 89 01 48 83 c8 ff c3 66 2e 0f 1f 84 00 00 00 00 00 90 f3 0f 1e fa 49 89 ca b8 a5 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 3a 03 0c 00 f7 d8 64 89 01 48 May 24 19:21:19 dvr0.lan kernel: RSP: 002b:00007ffe575b90a8 EFLAGS: 00000246 ORIG_RAX: 00000000000000a5 May 24 19:21:19 dvr0.lan kernel: RAX: ffffffffffffffda RBX: 000055abcf56f570 RCX: 00007f97037d3a9e May 24 19:21:19 dvr0.lan kernel: RDX: 000055abcf575ee0 RSI: 000055abcf56f7c0 RDI: 000055abcf56f7a0 May 24 19:21:19 dvr0.lan kernel: RBP: 00007ffe575b91d0 R08: 0000000000000000 R09: 0000000000000007 May 24 19:21:19 dvr0.lan kernel: R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000 May 24 19:21:19 dvr0.lan kernel: R13: 000055abcf56f7a0 R14: 000055abcf575ee0 R15: 00007f9703903076 May 24 19:21:19 dvr0.lan kernel: </TASK> May 24 19:21:19 dvr0.lan kernel: XFS (sdc1): Corruption detected. Unmount and run xfs_repair May 24 19:21:19 dvr0.lan kernel: XFS (sdc1): Failed to recover intents May 24 19:21:19 dvr0.lan kernel: XFS (sdc1): Filesystem has been shut down due to log error (0x2). May 24 19:21:19 dvr0.lan kernel: XFS (sdc1): Please unmount the filesystem and rectify the problem(s). May 24 19:21:19 dvr0.lan kernel: XFS (sdc1): Ending recovery (logdev: internal) May 24 19:21:19 dvr0.lan kernel: XFS (sdc1): log mount finish failed xfs_repair with -L was required --- (A very old) desktop system, without heavy disk activity, running on a LSI MegaRaid 9260 using hardware RAID-6 (6 4TB drives in the RAID-6 virtual drive). Previously the system was running 6.2.14 without issue(s). --- Results from xfs_info meta-data=/dev/sdc1 isize=512 agcount=32, agsize=122070336 blks = sectsz=4096 attr=2, projid32bit=1 = crc=1 finobt=1, sparse=1, rmapbt=1 = reflink=1 bigtime=1 inobtcount=1 nrext64=0 data = bsize=4096 blocks=3906249216, imaxpct=5 = sunit=64 swidth=256 blks naming =version 2 bsize=4096 ascii-ci=0, ftype=1 log =internal log bsize=4096 blocks=521728, version=2 = sectsz=4096 sunit=1 blks, lazy-count=1 realtime =none extsz=4096 blocks=0, rtextents=0 --- I note that all these reported failures seem to be using hardware raid and with stripe (whether either is relevant is unknown). I am also using XFS (without issue) on a number of other systems with a 6.3 kernel, but none of them are using hardware raid or stripes. --- I downgraded to the latest 6.2 kernel for now as my workaround.
Other things of note: 6.3 has never made it past updates testing, due to this issue. That means we don't really know how widepsread it is beyond the people who have enabled updates-testing The default filesystem for fedora workstation and cloud spins is btrfs, so I would expect that even the majority of users with updates-testing enabled are not running xfs. I have not seen this on any of my machines, though they are all single disk paritions for xfs.
I have been busy with work the last couple of days, I'm glad Rune had also seen this same problem on a much larger number of machines. This occurred on my primary workstation, as you can probably imagine I am not eager to repeatedly try to duplicate something which results in corruption of the filesystem. Of course I will if necessary to help track down the bug, but I am hoping Rune's 50 guinea pigs are sufficient for the time being. > The default filesystem for fedora workstation and cloud spins is btrfs Yeah, I first setup this machine back before btrfs was considered stable enough to use in production. Red Hat had decided to make XFS the default for RHEL, and I had been using XFS on some data storage volumes in the past (i.e. not / and not /home), so I decided to setup my machine with XFS like RHEL was using as default. I will try to fill in the info Dave C requested in case it helps find some common thread. I am not sure what was meant by "storage stack layout," if @dchinner can explain what was meant I can provide additional info. Unfortunately I do not have the full dmesg info available from the time of the crash, hopefully Rune is better able to grab that from one of his machines. Did anyone recognize any of that output that got jammed into the metadata block in my first post? Could it be from gcc? I think I was compiling Ardour when the crash happened. Does the preponderance of the evidence point to some other process stomping on filesystem memory, and not a filesystem problem per-se? This is an old tower server that I am using as a workstation, so it is what would have been a basic server about 14 years ago. Ugh, I had to go look up that release date. Time for a refresh, although hard to beat the core count and memory size you can get for just a few hundred dollars of obsolete datacenter gear. Anyway...this info is current, not from the time of the filesystem corruption. Dual sockets (both populated). CPU: processor : 0 vendor_id : GenuineIntel cpu family : 6 model : 44 model name : Intel(R) Xeon(R) CPU X5650 @ 2.67GHz stepping : 2 microcode : 0x1f Memory: MemTotal: 198035844 kB MemFree: 186282764 kB MemAvailable: 189951908 kB Buffers: 10716 kB Cached: 4872220 kB SwapCached: 0 kB Active: 7719416 kB Inactive: 1964496 kB Active(anon): 4901080 kB Inactive(anon): 22592 kB Active(file): 2818336 kB Inactive(file): 1941904 kB Unevictable: 492 kB Mlocked: 492 kB SwapTotal: 12580088 kB SwapFree: 12580088 kB Zswap: 0 kB Zswapped: 0 kB Dirty: 4 kB Writeback: 0 kB AnonPages: 4801604 kB Mapped: 916436 kB Shmem: 122676 kB KReclaimable: 144676 kB Slab: 362636 kB SReclaimable: 144676 kB SUnreclaim: 217960 kB KernelStack: 30576 kB PageTables: 73212 kB SecPageTables: 0 kB NFS_Unstable: 0 kB Bounce: 0 kB WritebackTmp: 0 kB CommitLimit: 111598008 kB Committed_AS: 15729108 kB VmallocTotal: 34359738367 kB VmallocUsed: 935428 kB VmallocChunk: 0 kB Percpu: 33280 kB HardwareCorrupted: 0 kB AnonHugePages: 0 kB ShmemHugePages: 0 kB ShmemPmdMapped: 0 kB FileHugePages: 9872 kB FilePmdMapped: 0 kB CmaTotal: 0 kB CmaFree: 0 kB HugePages_Total: 0 HugePages_Free: 0 HugePages_Rsvd: 0 HugePages_Surp: 0 Hugepagesize: 2048 kB Hugetlb: 0 kB DirectMap4k: 604284 kB DirectMap2M: 17211392 kB DirectMap1G: 183500800 kB Storage: Smart Array P410 in Slot 1 Smart Array P420 in Slot 2 The controller in slot 1 is a pseudo-backup device, it has a single large logical volume with a single XFS filesystem. It gets mounted to copy archive files, then is unmounted again for safety. Smart Array P420 in Slot 2 Bus Interface: PCI Slot: 2 RAID 6 (ADG) Status: Enabled Controller Status: OK Hardware Revision: B Firmware Version: 8.32-0 Rebuild Priority: High Expand Priority: Medium Surface Scan Delay: 3 secs Surface Scan Mode: Idle Parallel Surface Scan Supported: No Queue Depth: Automatic Monitor and Performance Delay: 60 min Elevator Sort: Enabled Degraded Performance Optimization: Disabled Inconsistency Repair Policy: Disabled Wait for Cache Room: Disabled Surface Analysis Inconsistency Notification: Disabled Post Prompt Timeout: 15 secs Cache Board Present: True Cache Status: OK Cache Ratio: 50% Read / 50% Write Drive Write Cache: Disabled Total Cache Size: 1024 MB Total Cache Memory Available: 816 MB No-Battery Write Cache: Disabled SSD Caching RAID5 WriteBack Enabled: False SSD Caching Version: 1 Cache Backup Power Source: Capacitors Battery/Capacitor Count: 1 Battery/Capacitor Status: OK SATA NCQ Supported: True Spare Activation Mode: Activate on physical drive failure (default) Controller Temperature (C): 74 Cache Module Temperature (C): 42 Capacitor Temperature (C): 28 Number of Ports: 2 Internal only Driver Name: hpsa Driver Version: 3.4.20 Driver Supports SSD Smart Path: True PCI Address (Domain:Bus:Device.Function): 0000:11:00.0 Sanitize Erase Supported: False Primary Boot Volume: None Secondary Boot Volume: None Physical drive info: Smart Array P420 in Slot 2 Array A physicaldrive 1I:2:1 (port 1I:box 2:bay 1, SAS HDD, 300 GB, OK) physicaldrive 1I:2:2 (port 1I:box 2:bay 2, SAS HDD, 300 GB, OK) physicaldrive 1I:2:3 (port 1I:box 2:bay 3, SAS HDD, 300 GB, OK) physicaldrive 1I:2:4 (port 1I:box 2:bay 4, SAS HDD, 300 GB, OK) physicaldrive 2I:2:5 (port 2I:box 2:bay 5, SAS HDD, 300 GB, OK) physicaldrive 2I:2:6 (port 2I:box 2:bay 6, SAS HDD, 300 GB, OK) physicaldrive 2I:2:7 (port 2I:box 2:bay 7, SAS HDD, 300 GB, OK) physicaldrive 2I:2:8 (port 2I:box 2:bay 8, SAS HDD, 300 GB, OK) Logical volume info: Smart Array P420 in Slot 2 Array A logicaldrive 1 (597 MB, RAID 5, OK) [ this is /boot ] logicaldrive 2 (4.0 GB, RAID 5, OK) [ this is swap ] logicaldrive 3 (100.0 GB, RAID 5, OK) [ this is / ] logicaldrive 4 (100.0 GB, RAID 5, OK) [ this is /home, this one fell over ] logicaldrive 5 (1.7 TB, RAID 5, OK) [ this is /mnt/data ] Logical drive 4 was the volume which failed previously (which is mounted on /home). Smart Array P420 in Slot 2 Array A Logical Drive: 4 Size: 100.0 GB Fault Tolerance: 5 Heads: 255 Sectors Per Track: 32 Cylinders: 25700 Strip Size: 256 KB Full Stripe Size: 1792 KB Status: OK Caching: Enabled Parity Initialization Status: Initialization Completed Unique Identifier: 600508B1001C21B685B97DA3A8E264FF Disk Name: /dev/sdd Mount Points: /home 100.0 GB Partition Number 1 OS Status: LOCKED Logical Drive Label: home Drive Type: Data LD Acceleration Method: Controller Cache xfs_info /home meta-data=/dev/sdd1 isize=512 agcount=16, agsize=1638272 blks = sectsz=512 attr=2, projid32bit=1 = crc=1 finobt=1, sparse=0, rmapbt=0 = reflink=0 bigtime=0 inobtcount=0 nrext64=0 data = bsize=4096 blocks=26212352, imaxpct=25 = sunit=64 swidth=448 blks naming =version 2 bsize=4096 ascii-ci=0, ftype=1 log =internal log bsize=4096 blocks=12800, version=2 = sectsz=512 sunit=64 blks, lazy-count=1 realtime =none extsz=4096 blocks=0, rtextents=0 Mount options: # # /etc/fstab # Created by anaconda on Wed Dec 21 18:20:55 2016 # edited by chris on Fri Dec 23 2016 # # Accessible filesystems, by reference, are maintained under '/dev/disk' # See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info # UUID=f89dff8e-749a-4bfa-a42a-0144acc00820 / xfs defaults 0 1 UUID=b0efc850-2966-44af-90a5-7de8bf605093 /boot ext4 defaults 1 2 UUID=60cf5755-8094-46d2-a297-c0b4c1ce03d9 /home xfs defaults 0 2 UUID=7213c283-cdc2-421d-a840-f28eb8716c7e swap swap defaults 0 0 UUID=f3add08f-39f8-4727-a2d5-25f3a1c9c300 /mnt/data xfs defaults 0 2 # Extra array of mid-line disks UUID=4885b76e-fb84-43e6-bd27-557a1a9462ea /mnt/bigdata xfs defaults,noauto 0 2
(In reply to Chris Caudle from comment #22) > > The default filesystem for fedora workstation and cloud spins is btrfs > > Yeah, I first setup this machine back before btrfs was considered stable > enough to use in production. Red Hat had decided to make XFS the default > for RHEL, and I had been using XFS on some data storage volumes in the past > (i.e. not / and not /home), so I decided to setup my machine with XFS like > RHEL was using as default. This was not meant as "you shouldn't be using xfs" I use it here, and only have btrfs on some test VMs, it was more meant to point out that of the limited number of users willing to test updates-testing, a much smaller number would be running xfs at all. I am rather worried about how widespread this issue would be if we pushed 6.3 to stable updates right now.
Not a Red Hat user but I have the same issue and thought it might be helpful for you to have some details. It's a Debian system (currently on bookworm) using XanMod. On the XanMod 6.3.3 kernel I get the error reliably; on 6.3.1 I do not. I'm afraid I'm not a kernel expert. Posting this here as it seems to be a fairly different hardware configuration to others, but very similar issue in other respects. I'm in no way seeking support - purely in case it helps with your own diagnostics and apologies if it muddies the water. Some info pulled from syslog before I switched back to 6.3.1-x64v2-xanmod1: 2023-05-19T09:51:36.149820+01:00 localhost kernel: [595667.735185] XFS (md1): Corruption detected. Unmount and run xfs_repair 2023-05-19T09:51:36.179608+01:00 localhost kernel: [595667.737796] md1: writeback error on inode 8651095188, offset 0, sector 5920535552 2023-05-19T09:51:36.179615+01:00 localhost kernel: [595667.765370] XFS (md1): Corruption of in-memory data (0x8) detected at xfs_defer_finish_noroll+0x23b/0x480 [xfs] (fs/xfs/libxfs/xfs_defer.c:573). Shutting down filesystem. 2023-05-19T09:51:36.179616+01:00 localhost kernel: [595667.765390] XFS (md1): Please unmount the filesystem and rectify the problem(s) 2023-05-19T10:26:30.649329+01:00 localhost kernel: [ 6.535940] XFS (md1): Metadata corruption detected at xfs_buf_ioend+0x4f/0x380 [xfs], xfs_inode block 0x160e42c80 xfs_inode_buf_verify 2023-05-19T10:26:30.649331+01:00 localhost kernel: [ 6.535961] XFS (md1): Unmount and run xfs_repair Also 2023-05-21T15:55:01.742635+01:00 localhost kernel: [192412.030060] XFS (md1): Internal error bno + len > gtbno at line 1908 of file fs/xfs/libxfs/xfs_alloc.c. Caller xfs_free_ag_extent.constprop.0+0x114/0x6f0 [xfs] 2023-05-21T15:55:01.758627+01:00 localhost kernel: [192412.045969] XFS (md1): Corruption of in-memory data (0x8) detected at xfs_defer_finish_noroll+0x23b/0x480 [xfs] (fs/xfs/libxfs/xfs_defer.c:573). Shutting down filesystem. Specification: AMD Ryzen 5 5600X 4*12TB WD Gold RAID10 (mdraid, far=2 layout, 512k chunk) XFS filesystem was created on the mdraid device April 2020 No LVM in use No hardware RAID smartctl happy for all physical devices in the system xfs_repair -L required mdadm -Q /dev/md1 /dev/md1: 21.83TiB raid10 4 devices, 0 spares. Use mdadm --detail for more detail. xfs_info /home meta-data=/dev/md1 isize=512 agcount=32, agsize=183105408 blks = sectsz=4096 attr=2, projid32bit=1 = crc=1 finobt=1, sparse=1, rmapbt=0 = reflink=0 bigtime=0 inobtcount=0 nrext64=0 data = bsize=4096 blocks=5859373056, imaxpct=5 = sunit=128 swidth=512 blks naming =version 2 bsize=4096 ascii-ci=0, ftype=1 log =internal log bsize=4096 blocks=521728, version=2 = sectsz=4096 sunit=1 blks, lazy-count=1 realtime =none extsz=4096 blocks=0, rtextents=0 Mount options: /dev/md1 /home xfs defaults 0 0
I could reproduce on 6.3.4 kernel. The 6.4 kernel still runs fine on the same type of hardware.
So this was a patch backported to stable for either 6.3.2 or 6.3.3. The only xfs specific patch there is xfs-don-t-consider-future-format-versions-valid, but there are about 1300 patches total in those updates.
Rune, can you give 6.3.2 a try? That would narrow it down a good bit more. There is a build at https://koji.fedoraproject.org/koji/buildinfo?buildID=2199126
(In reply to Justin M. Forbes from comment #27) > Rune, can you give 6.3.2 a try? That would narrow it down a good bit more. > There is a build at > https://koji.fedoraproject.org/koji/buildinfo?buildID=2199126 I think that's a good idea. (Was wondering about testing 6.3.0 to eliminate all -stable backport variables?)
(In reply to Eric Sandeen from comment #28) > (In reply to Justin M. Forbes from comment #27) > > Rune, can you give 6.3.2 a try? That would narrow it down a good bit more. > > There is a build at > > https://koji.fedoraproject.org/koji/buildinfo?buildID=2199126 > > I think that's a good idea. (Was wondering about testing 6.3.0 to eliminate > all -stable backport variables?) We have at least one verification that 6.3.1 is good. We know 6.3.3 is bad. I built 6.3.2 for kernel test week (where no one reported this, but it could just be no one impacted tested with that kernel). A proper test from someone who can reliably reproduce would tell us if this came in with 6.3.2 (694 patches) or 6.3.3 (245 patches). Also of note, if 6.2.15 did not see this, that was the release at the same time as 6.3.2. It too had over 600 patches, most of the same ones. 6.2.16 was the 6.3.3 equivalent, which I built, but never pushed as an update, so we have no data on.
(In reply to Justin M. Forbes from comment #29) > We have at least one verification that 6.3.1 is good. And the report in comment #20 failed with 6.3.1 Given that no one has a trivial reproducer (although Rune seems to be far in the lead) I would be reluctant to believe that lack of failures demonstrates a good kernel level.
We do also have 6.3.1 kernels: https://koji.fedoraproject.org/koji/buildinfo?buildID=2196477 and 6.3.0 kernels (built against rawhide, but should run fine for testing as long as you don't need an out of tree module) https://koji.fedoraproject.org/koji/buildinfo?buildID=2191871 Should anyone need the link to it for testing, this was 6.2.16: https://koji.fedoraproject.org/koji/buildinfo?buildID=2202061
I might have a reproducer here. I took Rune's metadump, xfs_mdrestore'd it to an image, and used that in a loopback mount as the TEST_DEV in xfstests. Running generic/013 (which runs fsstress on TEST_DEV) seems to reliably re-corrupt the image under the 6.3.3 kernel and succeeds under the 6.2.15 kernel. (It's remotely possible that there is latent corruption in the image even after the xfs_repair, and only 6.3 is catching it, but that seems unlikely.) I'll keep you posted...
I can confirm that 6.2.2 also corrupts the file systems. I happened just after reboot, I couldn't log in and run a command before it died.
Sorry, typo, 6.3.2.
@Justin : I just stumbled upon this bug report and want to share my experience. I am running Fedora Linux 38 Workstation with kernel 6.3 on a XFS file system since you provided the first point release. My partition setup -> 1. partition /boot/efi - format FAT32 | 2. partition /boot - format XFS | 3. partition (LVM) / - format XFS ... All 6.3 kernel versions worked as expected without reported issues. Today I installed kernel 6.3.4 and nothing changed, everything works as expected. Below you can see that no metadata issue gets reported. Maybe the problem occurs only on RAID setups ? Hope this helps. journalctl | grep metadata May 26 11:00:34 ******** systemd[2174]: Starting gvfs-metadata.service - Virtual filesystem metadata service... May 26 11:00:34 ******** systemd[2174]: Started gvfs-metadata.service - Virtual filesystem metadata service. May 26 11:08:52 ******** systemd[2174]: Starting tracker-extract-3.service - Tracker metadata extractor... May 26 11:08:52 ******** systemd[2174]: Started tracker-extract-3.service - Tracker metadata extractor. journalctl | grep xfs May 26 11:00:21 ******** systemd-fsck[651]: /usr/sbin/fsck.xfs: XFS file system. May 26 11:00:23 ******** kernel: xfs filesystem being remounted at / supports timestamps until 2038 (0x7fffffff) May 26 11:00:23 ******** kernel: xfs filesystem being mounted at /boot supports timestamps until 2038 (0x7fffffff)
(In reply to Christian Labisch from comment #35) > Maybe the problem occurs only on RAID setups ? No, I encountered this problem despite not using any form of RAID.
I landed on 74c36a8689d ("xfs: use xfs_alloc_vextent_this_ag() where appropriate") as the first bad commit between 6.2 and 6.3 which triggers the reproducer for me. Haven't looked for problems in the commit yet; 6.4-rc seems ok again, I'm going to see if I can bisect to what fixed it, and see whether it was intentional...
(In reply to Eric Sandeen from comment #37) > I landed on 74c36a8689d ("xfs: use xfs_alloc_vextent_this_ag() where > appropriate") as the first bad commit between 6.2 and 6.3 which triggers the > reproducer for me. > > Haven't looked for problems in the commit yet; 6.4-rc seems ok again, I'm > going to see if I can bisect to what fixed it, and see whether it was > intentional... That is great detective work. Thanks. I now see the light at the end of the tunnel (and I don't think it is another filesystem corrupting train).
Please backport this commit to a test 6.3.3 kernel build and see if that fixes the problem: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/fs/xfs/libxfs/xfs_bmap.c?id=9419092fb2630c30e4ffeb9ef61007ef0c61827a This is a bug fix that we thought just fixed a livelock on stripe aligned filesystems. I'm guessing that in certain circumstances instead of livelocking on repeated failed allocations, it results in a broken mapping being returned to the writeback code and hence misdirecting the writeback IO. -Dave.
Thanks Dave, I'll try that right now.
(In reply to Joseph C. Sible from comment #36) > (In reply to Christian Labisch from comment #35) > > Maybe the problem occurs only on RAID setups ? > > No, I encountered this problem despite not using any form of RAID. Out of curiosity, are you using stripes? (sunit != 0 and/or swidth != 0)? That might suggest a common case.
(In reply to Dave Chinner from comment #39) > Please backport this commit to a test 6.3.3 kernel build and see if that > fixes the problem: > > https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/fs/ > xfs/libxfs/xfs_bmap.c?id=9419092fb2630c30e4ffeb9ef61007ef0c61827a > > This is a bug fix that we thought just fixed a livelock on stripe aligned > filesystems. > > I'm guessing that in certain circumstances instead of livelocking on > repeated failed allocations, it results in a broken mapping being returned > to the writeback code and hence misdirecting the writeback IO. > > -Dave. Looks like that fixes it Dave, thank you. @jforbes if you want to build a test 6.3.x kernel with that commit ...?
https://koji.fedoraproject.org/koji/buildinfo?buildID=2206484 has that commit. Please test and let me know, it is an official build, so I will push to updates-testing once I have confirmation.
Thanks Justin! This passes the reproducer I had, I'll loop it for a while and then run through full xfstests a few times.
Build 2206484 has been stable for 90 minutes on the same type of hardware that all the other 6.3 kernels crashed within a couple of minutes after boot. So this seems to fix the issue for me.
FEDORA-2023-2f35633034 has been submitted as an update to Fedora 37. https://bodhi.fedoraproject.org/updates/FEDORA-2023-2f35633034
FEDORA-2023-514965dd8a has been submitted as an update to Fedora 38. https://bodhi.fedoraproject.org/updates/FEDORA-2023-514965dd8a
(In reply to Gary Buhrmaster from comment #41) > (In reply to Joseph C. Sible from comment #36) > > (In reply to Christian Labisch from comment #35) > > > Maybe the problem occurs only on RAID setups ? > > > > No, I encountered this problem despite not using any form of RAID. > > Out of curiosity, are you using stripes? (sunit != 0 and/or swidth != 0)? > That might suggest a common case. Nope, those are both 0 on the filesystem where I encountered this.
FEDORA-2023-514965dd8a has been pushed to the Fedora 38 testing repository. Soon you'll be able to install the update with the following command: `sudo dnf upgrade --enablerepo=updates-testing --refresh --advisory=FEDORA-2023-514965dd8a` You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2023-514965dd8a See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.
FEDORA-2023-2f35633034 has been pushed to the Fedora 37 testing repository. Soon you'll be able to install the update with the following command: `sudo dnf upgrade --enablerepo=updates-testing --refresh --advisory=FEDORA-2023-2f35633034` You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2023-2f35633034 See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.
FEDORA-2023-2f35633034 has been pushed to the Fedora 37 stable repository. If problem still persists, please make note of it in this bug report.
No repro after only about an hour running kernel-6.3.4-201.fc38.x86_64. I will keep it running for the next few days, hopefully that fully fixed the problem.
FEDORA-2023-514965dd8a has been pushed to the Fedora 38 stable repository. If problem still persists, please make note of it in this bug report.
(In reply to Joseph C. Sible from comment #48) > (In reply to Gary Buhrmaster from comment #41) > > (In reply to Joseph C. Sible from comment #36) > > > (In reply to Christian Labisch from comment #35) > > > > Maybe the problem occurs only on RAID setups ? > > > > > > No, I encountered this problem despite not using any form of RAID. > > > > Out of curiosity, are you using stripes? (sunit != 0 and/or swidth != 0)? > > That might suggest a common case. > > Nope, those are both 0 on the filesystem where I encountered this. I don't see any details of the problem you encountered; it's important not to lump different problems together. Have you reported details of this anywhere? Thanks, -Eric
(In reply to Eric Sandeen from comment #54) > I don't see any details of the problem you encountered; it's important not > to lump different problems together. Have you reported details of this > anywhere? I have not. Unfortunately, I didn't save enough information from when I encountered this to make a report, since this was the first time I've ever encountered XFS metadata corruption, and it happened to be on a 6.3.x kernel and the very same day that this bug was mentioned on news sites like Phoronix, so I just assumed it was what was affecting me. I'm on 6.3.5 now, so if it does reoccur, then next time I'll save enough information for a report.