Description of problem: I'm not certain where the problem is, but chose 'anaconda' since the install results in a failed installation. I tried installing F25 with XFS root filesystem in LVM, using UEFI on an NVMe SSD. The installation completes successfully, but upon reboot the system is unable to mount the root filesystem at switchroot. Hence, unusable installation. I tried booting into rescue mode from USB to check on the filesystem at /dev/mapper/fedora-root. I am able to mount it cleanly under /mnt and can see the installation files. However, if I run xfs_repair /dev/mapper/fedora-root, xfs_repair reports there's no superblock and searches for secondary superblock to no avail. If I then reformat it again manually like so: mkfs.xfs -f /dev/mapper/fedora-root I can then mount it again under /mnt successfully. AND, I can run xfs_repair on /dev/mapper/fedora-root without issue. So, it would appear that the mkfs.xfs during install, or at some point after the install makes the XFS root file system unmountable during boot. But the filesystem is mountable in rescue mode (booting off F25 live usb), but xfs_repair thinks it is broken. Version-Release number of selected component (if applicable): Fresh F25 installation from live USB How reproducible: I've reproduced it at least 3 times consecutively. Steps to Reproduce: 1. boot F25 live USB 2. begin installation of F25 3. choose XFS file root filesystem on LVM 4. let installation complete 5. reboot into new installation - fails. Actual results: fresh installation fails to boot Expected results: fresh installation boots Additional info:
I've been able to consistently reproduce the problem above a couple more times. For contrast, I decided to repeat the steps above except to use ext4 instead of xfs. When I use ext4, the system is bootable after installation is complete. So, it does seem to be xfs related. Just adding more info here.... but if there's anything else I can provide to help, please let me know.
today, i retried the installation, but after the installation completed (step 4) above, before rebooting I opened up a terminal and checked on the XFS file system. I was able to unmount and remount it under /mnt/sysimage without issue. But running xfs_repair /dev/mapper/fedora-root resulted in the same behavior above (can't find superblock). Trying to boot into the new installation fails as above. I then tried again in an attempt to pre-create the XFS filesystem instead of letting the installer create them, but found out the installer insists of creating the / file system; so that was no go. I then tried to install again, but after step 4, checked on the filesystem by running xfs_info /mnt/sysimage (which worked fine), then umount -R /mnt/sysimage, then xfs_repair /dev/mapper/fedora-root, which actually succeeded! (not sure why) Rebooted and the freshly installed Fedora on XFS booted up just fine for the 1st time. Then I rebooted again, and the XFS file system was broken again and could not boot Fedora anymore. I then booted with USB drive to Fedora 25 Live to check on the file system at /dev/mapper/fedora-root. Tried to mount it, but gave following errors: [ 77.626307] SGI XFS with ACLs, security attributes, no debug enabled [ 77.628509] XFS (dm-1): Mounting V5 Filesystem [ 77.685399] XFS (dm-1): Starting recovery (logdev: internal) [ 77.691471] XFS (dm-1): Metadata CRC error detected at xfs_agfl_read_verify+0xcd/0x100 [xfs], xfs_agfl block 0x4600003 [ 77.691488] XFS (dm-1): Unmount and run xfs_repair [ 77.691489] XFS (dm-1): First 64 bytes of corrupted metadata buffer: [ 77.691495] ffff8bdb937c6e00: 62 fd fc 89 76 10 54 8f cc 93 02 ce 51 2e 12 7a b...v.T.....Q..z [ 77.691509] ffff8bdb937c6e10: 97 4a da 48 59 bf 01 13 ae 6a 27 69 c2 eb ee 4a .J.HY....j'i...J [ 77.691510] ffff8bdb937c6e20: c8 5c 9b a3 b4 90 21 dc 80 7b 1c b7 fd c1 42 2b .\....!..{....B+ [ 77.691511] ffff8bdb937c6e30: 78 cc 15 d7 37 0d 64 d8 f6 13 ca c8 c8 2b db a9 x...7.d......+.. [ 77.691550] XFS (dm-1): metadata I/O error: block 0x4600003 ("xfs_trans_read_buf_map") error 74 numblks 1 [ 77.691585] XFS (dm-1): Internal error xfs_trans_cancel at line 984 of file fs/xfs/xfs_trans.c. Caller xfs_efi_recover+0x18e/0x1c0 [xfs] [ 77.691586] CPU: 1 PID: 2729 Comm: mount Not tainted 4.8.6-300.fc25.x86_64 #1 [ 77.691587] Hardware name: LENOVO 20ENCTO1WW/20ENCTO1WW, BIOS N1EET62W (1.35 ) 11/10/2016 [ 77.691588] 0000000000000286 00000000f0f0a52e ffff8bdb1fa9fba8 ffffffff9d3e5ebd [ 77.691590] ffff8bdb107db050 0000000000000001 ffff8bdb1fa9fbc0 ffffffffc0d37adc [ 77.691591] ffffffffc0d5b04e ffff8bdb1fa9fbe8 ffffffffc0d53346 ffff8bdb107d9188 [ 77.691593] Call Trace: [ 77.691596] [<ffffffff9d3e5ebd>] dump_stack+0x63/0x86 [ 77.691610] [<ffffffffc0d37adc>] xfs_error_report+0x3c/0x40 [xfs] [ 77.691625] [<ffffffffc0d5b04e>] ? xfs_efi_recover+0x18e/0x1c0 [xfs] [ 77.691640] [<ffffffffc0d53346>] xfs_trans_cancel+0xb6/0xe0 [xfs] [ 77.691654] [<ffffffffc0d5b04e>] xfs_efi_recover+0x18e/0x1c0 [xfs] [ 77.691667] [<ffffffffc0d5d6cc>] xlog_recover_process_efi+0x2c/0x50 [xfs] [ 77.691680] [<ffffffffc0d5f91a>] xlog_recover_process_intents.isra.34+0xca/0xe0 [xfs] [ 77.691693] [<ffffffffc0d632a3>] xlog_recover_finish+0x23/0xb0 [xfs] [ 77.691707] [<ffffffffc0d55fb9>] xfs_log_mount_finish+0x29/0x50 [xfs] [ 77.691721] [<ffffffffc0d4d14b>] xfs_mountfs+0x6bb/0x880 [xfs] [ 77.691734] [<ffffffffc0d4faa9>] xfs_fs_fill_super+0x399/0x4b0 [xfs] [ 77.691736] [<ffffffff9d2541e8>] mount_bdev+0x178/0x1b0 [ 77.691749] [<ffffffffc0d4f710>] ? xfs_test_remount_options.isra.12+0x60/0x60 [xfs] [ 77.691761] [<ffffffffc0d4e525>] xfs_fs_mount+0x15/0x20 [xfs] [ 77.691762] [<ffffffff9d254cc8>] mount_fs+0x38/0x150 [ 77.691763] [<ffffffff9d1e3b55>] ? __alloc_percpu+0x15/0x20 [ 77.691765] [<ffffffff9d271ea7>] vfs_kern_mount+0x67/0x100 [ 77.691766] [<ffffffff9d27435d>] do_mount+0x1dd/0xc50 [ 77.691767] [<ffffffff9d24c1c5>] ? __check_object_size+0x105/0x1dc [ 77.691768] [<ffffffff9d1dde7f>] ? memdup_user+0x4f/0x70 [ 77.691769] [<ffffffff9d2750d3>] SyS_mount+0x83/0xd0 [ 77.691771] [<ffffffff9d802572>] entry_SYSCALL_64_fastpath+0x1a/0xa4 [ 77.691772] XFS (dm-1): xfs_do_force_shutdown(0x8) called from line 985 of file fs/xfs/xfs_trans.c. Return address = 0xffffffffc0d5335f [ 77.691774] XFS (dm-1): Corruption of in-memory data detected. Shutting down filesystem [ 77.691775] XFS (dm-1): Please umount the filesystem and rectify the problem(s) [ 77.691778] XFS (dm-1): Failed to recover intents [ 77.691778] XFS (dm-1): log mount finish failed [ 77.691786] XFS (dm-1): xfs_log_force: error -5 returned. I have checked the hardware with Lenovo's built-in diagnostics in the BIOS and have found no errors with memory, cpu, motherboard. The hardware is a ThinkPad P50.
another update: earlier today, i also tested installing F25 on a different machine (Thinkpad T460), following the same procedure as outlined above. that other machine installed just fine with XFS root file system and has since rebooted several times without issues. As I was considering why i'm having so much trouble with XFS root file system on the ThinkPad P50 machine, I realized that the T460 uses a SATA SSD (2.5"), while the ThinkPad P50 uses an NVMe M.2 SSD. The P50 also has an opening for a SATA 2.5" SSD, so I decided to put in a spare 750GB SATA SSD in the P50 and installed F25 with XFS root and everything went smoothly! I have since updated the packages on the P50 machine using the SATA SSD and rebooted several times without issue. So, perhaps there's an issue with XFS and the nvme driver? just speculating...
After searching around I decided to report my issue on this bug because I'm seeing consistent XFS IO issues with Fedora 25 on an Intel SSD. I was upgrading my homemade NAS server from CentOS 7 (ext4) to Fedora 25. I went with default install, which is XFS on root. First boot worked consistently but after I used the system more and more XFS I/O errors were showing on console and eventually the system would freeze in strange ways, or a second boot would not work due to corrupt FS. I didn't take pictures of the console unfortunately but every time it was corruption or IO errors on dm-0, which was the LVM root volume. This repeated or 3 re-installs until I finally chose ext4 on root. I did create a /var too and forgot to switch it from xfs to ext4 on re-install #4. So after reinstall #4 I didn't see the usual errors but after a yum update I noticed errors on /var instead, journald was unable to rotate logs or something. It was clearly IO related. So I did a 5th re-install with all ext4 volumes and after that no more issues. Kernel: 4.9.4-201.fc25.x86_64
(In reply to Stefan Midjich from comment #4) > After searching around I decided to report my issue on this bug because I'm > seeing consistent XFS IO issues with Fedora 25 on an Intel SSD. > > I was upgrading my homemade NAS server from CentOS 7 (ext4) to Fedora 25. > > I went with default install, which is XFS on root. > > First boot worked consistently but after I used the system more and more XFS > I/O errors were showing on console and eventually the system would freeze in > strange ways, or a second boot would not work due to corrupt FS. > > I didn't take pictures of the console unfortunately but every time it was > corruption or IO errors on dm-0, which was the LVM root volume. > > This repeated or 3 re-installs until I finally chose ext4 on root. I did > create a /var too and forgot to switch it from xfs to ext4 on re-install #4. > > So after reinstall #4 I didn't see the usual errors but after a yum update I > noticed errors on /var instead, journald was unable to rotate logs or > something. It was clearly IO related. > > So I did a 5th re-install with all ext4 volumes and after that no more > issues. > > Kernel: 4.9.4-201.fc25.x86_64 Stefan, thanks for sharing your experience in this bug report. Unfortunately, I'm not certain who to assign this to. I don't think it is a problem in anaconda anymore. The common factors is XFS on root, and SSD. In my case, it was a NVMe SSD by Intel. Is your Intel SSD also a NVMe SSD? Thanks.
Yes it is an Intel NVM SSD. At least it's using the nvme driver. And I did experience clear IO errors when I left xfs on /var but not root, so I don't think it's limited to the root volume. It's only that the XFS errors on the root volume were more immediately fatal than the ones on /var. But I still had to reinstall with ext4 on /var before I could use the OS. *-storage description: Non-Volatile memory controller product: Intel Corporation vendor: Intel Corporation physical id: 0 bus info: pci@0000:01:00.0 version: 03 width: 64 bits clock: 33MHz capabilities: storage pm pciexpress msix nvm_express bus_master cap_list configuration: driver=nvme latency=0 resources: irq:16 memory:df100000-df103fff
(In reply to Stefan Midjich from comment #6) > Yes it is an Intel NVM SSD. At least it's using the nvme driver. > > And I did experience clear IO errors when I left xfs on /var but not root, > so I don't think it's limited to the root volume. > > It's only that the XFS errors on the root volume were more immediately fatal > than the ones on /var. But I still had to reinstall with ext4 on /var before > I could use the OS. > > *-storage > description: Non-Volatile memory controller > product: Intel Corporation > vendor: Intel Corporation > physical id: 0 > bus info: pci@0000:01:00.0 > version: 03 > width: 64 bits > clock: 33MHz > capabilities: storage pm pciexpress msix nvm_express bus_master > cap_list > configuration: driver=nvme latency=0 > resources: irq:16 memory:df100000-df103fff thanks for confirming nvme ssd. ok, i'm going to change the component to kernel, whether it is in the XFS module or nvme driver, hopefully this helps get this bug report to the right people. i may also post on the xfs mailing list too.
Comment #2 contains the only useful information so far ... Please include anaconda install logs, full dmesg from failed boots, xfs_repair output, etc. A compressed xfs_metadump image may also be helpful. Testing a rawhide install with a newer kernel might also be informative.
since this issue might be SSD hardware specific, the model of Intel SSD I was using when this problem showed up was SSDPEKKW512G7X1. Stefan: Do you know the model of your Intel SSD? Is it same as mine or?
Almost, they rarely ever sell the same models all across the world. Mine was purchased in Sweden so likely some european release. Model Number: INTEL SSDPEKKW128G7 Serial Number: BTPY63140LFA128A Firmware Version: PSF100C
Google search brought me to this bug report - Not using Fedora, but I think I am seeing the same bug with Ubuntu 16.04, the Intel SSD NVME as reported above and XFS. Don't know if any of this information can help find the bug. I managed to network install a clean Ubuntu 16.04, with root on an XFS formatted NVMe SSD (this took a few attempts as the XFS corrupt bug can strike during the installation). Captured the following: Intel SSD Model Number: INTEL SSDPEKKW128G7 Firmware Version: PSF100C Linux version 4.4.0-59-generic (buildd@lgw01-11) (gcc version 5.4.0 20160609 (Ubuntu 5.4.0-6ubuntu1~16.04.4) ) #80-Ubuntu SMP Fri Jan 6 17:47:47 UTC 2017 (Ubuntu 4.4.0-59.80-generic 4.4.35) # lspci -kvvv <snip> 03:00.0 Non-Volatile memory controller: Intel Corporation Device f1a5 (rev 03) (prog-if 02 [NVM Express]) Subsystem: Intel Corporation Device 390a Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+ Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 0, Cache Line Size: 64 bytes Interrupt: pin A routed to IRQ 0 Region 0: Memory at dfe00000 (64-bit, non-prefetchable) [size=16K] Capabilities: [40] Power Management version 3 Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-) Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME- Capabilities: [70] Express (v2) Endpoint, MSI 00 DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s unlimited, L1 unlimited ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ DevCtl: Report errors: Correctable+ Non-Fatal+ Fatal+ Unsupported+ RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop- FLReset- MaxPayload 128 bytes, MaxReadReq 512 bytes DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr+ TransPend- LnkCap: Port #0, Speed 8GT/s, Width x4, ASPM L0s L1, Exit Latency L0s <1us, L1 <8us ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp+ LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+ ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- LnkSta: Speed 5GT/s, Width x4, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt- DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR+, OBFF Via message DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis- Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance De-emphasis: -6dB LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1- EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest- Capabilities: [b0] MSI-X: Enable+ Count=16 Masked- Vector table: BAR=0 offset=00002000 PBA: BAR=0 offset=00002100 Capabilities: [100 v2] Advanced Error Reporting UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol- CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr- CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+ AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn- Capabilities: [158 v1] #19 Capabilities: [178 v1] Latency Tolerance Reporting Max snoop latency: 0ns Max no snoop latency: 0ns Capabilities: [180 v1] L1 PM Substates L1SubCap: PCI-PM_L1.2+ PCI-PM_L1.1+ ASPM_L1.2+ ASPM_L1.1+ L1_PM_Substates+ PortCommonModeRestoreTime=10us PortTPowerOnTime=10us Kernel driver in use: nvme Kernel modules: nvme # xfs_info / meta-data=/dev/nvme0n1p1 isize=512 agcount=4, agsize=7292352 blks = sectsz=512 attr=2, projid32bit=1 = crc=1 finobt=1 spinodes=0 data = bsize=4096 blocks=29169408, imaxpct=25 = sunit=0 swidth=0 blks naming =version 2 bsize=4096 ascii-ci=0 ftype=1 log =internal bsize=4096 blocks=14242, version=2 = sectsz=512 sunit=0 blks, lazy-count=1 realtime =none extsz=4096 blocks=0, rtextents=0 Then on reboot, I see this: [ 5.592019] SGI XFS with ACLs, security attributes, realtime, no debug enabled [ 5.608577] XFS (nvme0n1p1): Mounting V5 Filesystem [ 5.629694] XFS (nvme0n1p1): Starting recovery (logdev: internal) [ 5.641458] XFS (nvme0n1p1): Metadata CRC error detected at xfs_agfl_read_verify+0xcd/0x100 [xfs], xfs_agfl block 0x6f45c03 [ 5.653920] XFS (nvme0n1p1): Unmount and run xfs_repair [ 5.659768] XFS (nvme0n1p1): First 64 bytes of corrupted metadata buffer: [ 5.667361] ffff88026faec800: 70 85 79 a7 01 5d ea c6 eb 8c ae 3b 3f df b1 45 p.y..].....;?..E [ 5.677091] ffff88026faec810: d6 59 2d 9c e7 af 32 5c ab f1 b7 f6 82 e1 12 3d .Y-...2\.......= [ 5.686822] ffff88026faec820: b9 fd 17 ee 68 6f 0b 20 b2 ae 7e c5 c7 29 54 a3 ....ho. ..~..)T. [ 5.696552] ffff88026faec830: ab c7 cb 7c 68 bd a7 de c3 d3 e8 50 aa e6 7f b9 ...|h......P.... [ 5.706311] XFS (nvme0n1p1): metadata I/O error: block 0x6f45c03 ("xfs_trans_read_buf_map") error 74 numblks 1 [ 5.717545] XFS (nvme0n1p1): Internal error xfs_trans_cancel at line 990 of file /build/linux-EO9xOi/linux-4.4.0/fs/xfs/xfs_trans.c. Caller xlog_recover_process_efi+0x161/0x190 [xfs] [ 5.735827] CPU: 6 PID: 379 Comm: exe Not tainted 4.4.0-59-generic #80-Ubuntu [ 5.743801] Hardware name: To be filled by O.E.M. To be filled by O.E.M./To be filled by O.E.M., BIOS 5.6.5 10/26/2016 [ 5.755760] 0000000000000286 00000000475ae75a ffff88026fa5bb90 ffffffff813f7583 [ 5.764047] ffff88026fc3c000 0000000000000001 ffff88026fa5bba8 ffffffffc02e36fc [ 5.772334] ffffffffc0308a71 ffff88026fa5bbd0 ffffffffc02fed86 ffff88026fb2e000 [ 5.780622] Call Trace: [ 5.783355] [<ffffffff813f7583>] dump_stack+0x63/0x90 [ 5.789126] [<ffffffffc02e36fc>] xfs_error_report+0x3c/0x40 [xfs] [ 5.796063] [<ffffffffc0308a71>] ? xlog_recover_process_efi+0x161/0x190 [xfs] [ 5.804166] [<ffffffffc02fed86>] xfs_trans_cancel+0xb6/0xe0 [xfs] [ 5.811103] [<ffffffffc0308a71>] xlog_recover_process_efi+0x161/0x190 [xfs] [ 5.819011] [<ffffffffc030aa5f>] xlog_recover_process_efis.isra.29+0x7f/0xe0 [xfs] [ 5.827601] [<ffffffffc030da51>] xlog_recover_finish+0x21/0xa0 [xfs] [ 5.834821] [<ffffffffc0301a99>] xfs_log_mount_finish+0x29/0x50 [xfs] [ 5.842137] [<ffffffffc02f8724>] xfs_mountfs+0x5b4/0x870 [xfs] [ 5.848773] [<ffffffffc02fb830>] xfs_fs_fill_super+0x3b0/0x4d0 [xfs] [ 5.855973] [<ffffffff81211f9d>] mount_bdev+0x26d/0x2c0 [ 5.861937] [<ffffffffc02fb480>] ? xfs_parseargs+0xab0/0xab0 [xfs] [ 5.868970] [<ffffffffc02f9c75>] xfs_fs_mount+0x15/0x20 [xfs] [ 5.875488] [<ffffffff812129c8>] mount_fs+0x38/0x160 [ 5.881132] [<ffffffff811b32f5>] ? __alloc_percpu+0x15/0x20 [ 5.887457] [<ffffffff8122ed57>] vfs_kern_mount+0x67/0x110 [ 5.893683] [<ffffffff8123163f>] do_mount+0x25f/0xda0 [ 5.899424] [<ffffffff811f0af4>] ? __kmalloc_track_caller+0x1b4/0x250 [ 5.906721] [<ffffffff811ad9d2>] ? memdup_user+0x42/0x70 [ 5.912753] [<ffffffff812324bf>] SyS_mount+0x9f/0x100 [ 5.918496] [<ffffffff818384f2>] entry_SYSCALL_64_fastpath+0x16/0x71 [ 5.925706] XFS (nvme0n1p1): xfs_do_force_shutdown(0x8) called from line 991 of file /build/linux-EO9xOi/linux-4.4.0/fs/xfs/xfs_trans.c. Return address = 0xffffffffc02fed9f [ 5.943032] XFS (nvme0n1p1): Corruption of in-memory data detected. Shutting down filesystem [ 5.952570] XFS (nvme0n1p1): Please umount the filesystem and rectify the problem(s) [ 5.961235] XFS (nvme0n1p1): Failed to recover EFIs [ 5.966688] XFS (nvme0n1p1): log mount finish failed [ 5.972257] XFS (nvme0n1p1): xfs_log_force: error -5 returned.
Thanks Neil for sharing your info. If possible, can you try to collect the information Eric asked for in comment #8? I plan to do this myself, but the machine I had this problem on is no longer in my possession and I'm waiting to get another similarly configured system. The sooner we can get better info to the developers, the greater the chance this will get fixed. Thanks.
(In reply to Bond Masuda from comment #12) > Thanks Neil for sharing your info. If possible, can you try to collect the > information Eric asked for in comment #8? I plan to do this myself, but the > machine I had this problem on is no longer in my possession and I'm waiting > to get another similarly configured system. The sooner we can get better > info to the developers, the greater the chance this will get fixed. Thanks. Sorry this took a while, but I've now recreated the issue and captured the data as requested in comment #8. Note - no Anaconda output as this is Ubuntu ;) Steps to recreate: * netbooted machine (from CEPH cluster) that has the Intel SSD attached - SSD is not initially mounted or used by root filesystem * mkfs.xfs the partition /dev/nvme0n1p1 that was showing the problems before and mounted as /mnt/ssd (logs below) * spent best part of the morning hammering the disk but no problems seem until .... * rebooted (netboot again) and tried to re-mount the xfs * dmesg reports issues and xfs_repair/xfs_metadump fails (logs below) # xfs_info /mnt/ssd meta-data=/dev/nvme0n1p1 isize=512 agcount=4, agsize=7292352 blks = sectsz=512 attr=2, projid32bit=1 = crc=1 finobt=1, sparse=0 data = bsize=4096 blocks=29169408, imaxpct=25 = sunit=0 swidth=0 blks naming =version 2 bsize=4096 ascii-ci=0 ftype=1 log =internal log bsize=4096 blocks=14242, version=2 = sectsz=512 sunit=0 blks, lazy-count=1 realtime =none extsz=4096 blocks=0, rtextents=0 # xfs_repair /dev/nvme0n1p1 Phase 1 - find and verify superblock... Phase 2 - using internal log - zero log... - scan filesystem freespace and inode maps... - found root inode chunk Phase 3 - for each AG... - scan and clear agi unlinked lists... - process known inodes and perform inode discovery... - agno = 0 - agno = 1 - agno = 2 - agno = 3 - process newly discovered inodes... Phase 4 - check for duplicate blocks... - setting up duplicate extent list... - check for inodes claiming duplicate blocks... - agno = 0 - agno = 1 - agno = 3 - agno = 2 Phase 5 - rebuild AG headers and trees... - reset superblock... Phase 6 - check inode connectivity... - resetting contents of realtime bitmap and summary inodes - traversing filesystem ... - traversal finished ... - moving disconnected inodes to lost+found ... Phase 7 - verify and correct link counts... done < hammered disk with lots of I/O - no failures > # reboot ############################## REBOOT ############################## # mount /dev/nvme0n1p1 /mnt/ssd <disk is corrupted> # dmesg [ 0.000000] [0mInitializing cgroup subsys cpuset [ 0.000000] [0mInitializing cgroup subsys cpu [ 0.000000] [0mInitializing cgroup subsys cpuacct [ 0.000000] [0mLinux version 4.4.14-040414-generic (kernel@tangerine) (gcc version 5.3.1 20160413 (Ubuntu 5.3.1-14ubuntu2.1) ) #201606241434 SMP Fri Jun 24 18:36:45 UTC 2016 [ 0.000000] [0m[33mCommand line[0m: BOOT_IMAGE=kernels/AVnz7lPEa-28Ib2EfbYQ/vmlinuz boot=rbd rbdroot=10.1.1.15,10.1.1.26,10.1.1.36:AVnz7lLFa-28Ib2EfbX5:AQAGXpBYmSJ5OxAAe1qfEUwBEq3g2B9Rb6cMhA==:instance_images:xenial-AVnz7lPEa-28Ib2EfbYQ:: rw ksdevice=bootif RM_SN=00-00B0-4089100006 INSTANCE_ID=AVnz7lPEa-28Ib2EfbYQ console=ttyS0,115200n8 rootfstype=ext4 initrd=kernels/AVnz7lPEa-28Ib2EfbYQ/initrd.img BOOTIF=01-b0-40-89-10-00-06 [ 0.000000] [0mKERNEL supported cpus: [ 0.000000] [0m Intel GenuineIntel [ 0.000000] [0m AMD AuthenticAMD [ 0.000000] [0m Centaur CentaurHauls [ 0.000000] [0m[33mx86/fpu[0m: Legacy x87 FPU detected. [ 0.000000] [0m[33mx86/fpu[0m: Using 'lazy' FPU context switches. [ 0.000000] [0m[33me820[0m: BIOS-provided physical RAM map: [ 0.000000] [0m[33mBIOS-e820[0m: [mem 0x0000000000000000-0x000000000009bfff] usable [ 0.000000] [0m[33mBIOS-e820[0m: [mem 0x000000000009c000-0x000000000009ffff] reserved [ 0.000000] [0m[33mBIOS-e820[0m: [mem 0x00000000000e0000-0x00000000000fffff] reserved [ 0.000000] [0m[33mBIOS-e820[0m: [mem 0x0000000000100000-0x000000007db91fff] usable [ 0.000000] [0m[33mBIOS-e820[0m: [mem 0x000000007db92000-0x000000007e5cbfff] reserved [ 0.000000] [0m[33mBIOS-e820[0m: [mem 0x000000007e5cc000-0x000000007e8b7fff] usable [ 0.000000] [0m[33mBIOS-e820[0m: [mem 0x000000007e8b8000-0x000000007f432fff] ACPI NVS [ 0.000000] [0m[33mBIOS-e820[0m: [mem 0x000000007f433000-0x000000007f639fff] reserved [ 0.000000] [0m[33mBIOS-e820[0m: [mem 0x000000007f63a000-0x000000007f7fffff] usable [ 0.000000] [0m[33mBIOS-e820[0m: [mem 0x00000000e0000000-0x00000000e3ffffff] reserved [ 0.000000] [0m[33mBIOS-e820[0m: [mem 0x00000000fed01000-0x00000000fed03fff] reserved [ 0.000000] [0m[33mBIOS-e820[0m: [mem 0x00000000fed08000-0x00000000fed08fff] reserved [ 0.000000] [0m[33mBIOS-e820[0m: [mem 0x00000000fed0c000-0x00000000fed0ffff] reserved [ 0.000000] [0m[33mBIOS-e820[0m: [mem 0x00000000fed1c000-0x00000000fed1cfff] reserved [ 0.000000] [0m[33mBIOS-e820[0m: [mem 0x00000000fef00000-0x00000000feffffff] reserved [ 0.000000] [0m[33mBIOS-e820[0m: [mem 0x00000000ff800000-0x00000000ffffffff] reserved [ 0.000000] [0m[33mBIOS-e820[0m: [mem 0x0000000100000000-0x000000027fffffff] usable [ 0.000000] [0m[33mNX (Execute Disable) protection[0m: active [ 0.000000] [0mSMBIOS 2.8 present. [ 0.000000] [0m[33mDMI[0m: To be filled by O.E.M. To be filled by O.E.M./To be filled by O.E.M., BIOS 5.6.5 10/26/2016 [ 0.000000] [0m[33me820[0m: update [mem 0x00000000-0x00000fff] usable ==> reserved [ 0.000000] [0m[33me820[0m: remove [mem 0x000a0000-0x000fffff] usable [ 0.000000] [0m[33me820[0m: last_pfn = 0x280000 max_arch_pfn = 0x400000000 [ 0.000000] [0m[33mMTRR default type[0m: write-back [ 0.000000] [0mMTRR fixed ranges enabled: [ 0.000000] [0m 00000-9FFFF write-back [ 0.000000] [0m A0000-BFFFF uncachable [ 0.000000] [0m C0000-FFFFF write-protect [ 0.000000] [0mMTRR variable ranges enabled: [ 0.000000] [0m 0 base 080000000 mask F80000000 uncachable [ 0.000000] [0m 1 base 280000000 mask F80000000 uncachable [ 0.000000] [0m 2 base 300000000 mask F00000000 uncachable [ 0.000000] [0m 3 base 400000000 mask C00000000 uncachable [ 0.000000] [0m 4 base 800000000 mask 800000000 uncachable [ 0.000000] [0m 5 base 07F800000 mask FFF800000 uncachable [ 0.000000] [0m 6 disabled [ 0.000000] [0m 7 disabled [ 0.000000] [0m[33mx86/PAT[0m: Configuration [0-7]: WB WC UC- UC WB WC UC- WT [ 0.000000] [0m[33me820[0m: last_pfn = 0x7f800 max_arch_pfn = 0x400000000 [ 0.000000] [0mfound SMP MP-table at [mem 0x000fd810-0x000fd81f] mapped at [ffff8800000fd810] [ 0.000000] [0mScanning 1 areas for low memory corruption [ 0.000000] [0mBase memory trampoline at [ffff880000095000] 95000 size 24576 [ 0.000000] [0mBRK [0x021f7000, 0x021f7fff] PGTABLE [ 0.000000] [0mBRK [0x021f8000, 0x021f8fff] PGTABLE [ 0.000000] [0mBRK [0x021f9000, 0x021f9fff] PGTABLE [ 0.000000] [0mBRK [0x021fa000, 0x021fafff] PGTABLE [ 0.000000] [0mBRK [0x021fb000, 0x021fbfff] PGTABLE [ 0.000000] [0mBRK [0x021fc000, 0x021fcfff] PGTABLE [ 0.000000] [0m[33mRAMDISK[0m: [mem 0x7c926000-0x7db91fff] [ 0.000000] [0m[33mACPI[0m: Early table checksum verification disabled [ 0.000000] [0m[33mACPI[0m[1m: RSDP 0x00000000000F0530 000024 (v02 ALASKA)[0m [ 0.000000] [0m[33mACPI[0m[1m: XSDT 0x000000007EC14090 00009C (v01 ALASKA A M I 01072009 AMI 00010013)[0m [ 0.000000] [0m[33mACPI[0m[1m: FACP 0x000000007EC16AB8 00010C (v05 ALASKA A M I 01072009 AMI 00010013)[0m [ 0.000000] [0m[33mACPI[0m[1m: DSDT 0x000000007EC141C0 0028F5 (v02 ALASKA A M I 01072009 INTL 20061109)[0m [ 0.000000] [0m[33mACPI[0m[1m: FACS 0x000000007F430F80 000040[0m [ 0.000000] [0m[33mACPI[0m[1m: FPDT 0x000000007EC16BC8 000044 (v01 ALASKA A M I 01072009 AMI 00010013)[0m [ 0.000000] [0m[33mACPI[0m[1m: FIDT 0x000000007EC16C10 00009C (v01 ALASKA A M I 01072009 AMI 00010013)[0m [ 0.000000] [0m[33mACPI[0m[1m: MCFG 0x000000007EC16CB0 00003C (v01 ALASKA A M I 01072009 MSFT 00000097)[0m [ 0.000000] [0m[33mACPI[0m[1m: WDAT 0x000000007EC16CF0 0001AC (v01 ALASKA A M I 01072009 MSFT 00010013)[0m [ 0.000000] [0m[33mACPI[0m[1m: UEFI 0x000000007EC16EA0 000042 (v01 00000000 00000000)[0m [ 0.000000] [0m[33mACPI[0m[1m: APIC 0x000000007EC16EE8 000098 (v03 INTEL TIANO 00000001 MSFT 00000000)[0m [ 0.000000] [0m[33mACPI[0m[1m: BDAT 0x000000007EC16F80 000030 (v01 00000000 00000000)[0m [ 0.000000] [0m[33mACPI[0m[1m: HPET 0x000000007EC16FB0 000038 (v01 INTEL 00000001 MSFT 01000013)[0m [ 0.000000] [0m[33mACPI[0m[1m: SSDT 0x000000007EC16FE8 0009F1 (v01 PmRef CpuPm 00003000 INTL 20061109)[0m [ 0.000000] [0m[33mACPI[0m[1m: SPCR 0x000000007EC179E0 000050 (v01 A M I APTIO V 01072009 AMI. 00000005)[0m [ 0.000000] [0m[33mACPI[0m[1m: HEST 0x000000007EC17A30 0000A8 (v01 INTEL AVOTON B 00000001 INTL 00000001)[0m [ 0.000000] [0m[33mACPI[0m[1m: BERT 0x000000007EC17AD8 000030 (v01 INTEL AVOTON B 00000001 INTL 00000001)[0m [ 0.000000] [0m[33mACPI[0m[1m: ERST 0x000000007EC17B08 000230 (v01 INTEL AVOTON B 00000001 INTL 00000001)[0m [ 0.000000] [0m[33mACPI[0m[1m: EINJ 0x000000007EC17D38 000150 (v01 INTEL AVOTON B 00000001 INTL 00000001)[0m [ 0.000000] [0m[33mACPI[0m: Local APIC address 0xfee00000 [ 0.000000] [0mNo NUMA configuration found [ 0.000000] [0mFaking a node at [mem 0x0000000000000000-0x000000027fffffff] [ 0.000000] [0mNODE_DATA(0) allocated [mem 0x27fff4000-0x27fff7fff] [ 0.000000] [0mZone ranges: [ 0.000000] [0m DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] [0m DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] [0m Normal [mem 0x0000000100000000-0x000000027fffffff] [ 0.000000] [0mMovable zone start for each node [ 0.000000] [0mEarly memory node ranges [ 0.000000] [0m[33m node 0[0m: [mem 0x0000000000001000-0x000000000009bfff] [ 0.000000] [0m[33m node 0[0m: [mem 0x0000000000100000-0x000000007db91fff] [ 0.000000] [0m[33m node 0[0m: [mem 0x000000007e5cc000-0x000000007e8b7fff] [ 0.000000] [0m[33m node 0[0m: [mem 0x000000007f63a000-0x000000007f7fffff] [ 0.000000] [0m[33m node 0[0m: [mem 0x0000000100000000-0x000000027fffffff] [ 0.000000] [0mInitmem setup node 0 [mem 0x0000000000001000-0x000000027fffffff] [ 0.000000] [0m[33mOn node 0 totalpages[0m: 2088927 [ 0.000000] [0m[33m DMA zone[0m: 64 pages used for memmap [ 0.000000] [0m[33m DMA zone[0m: 22 pages reserved [ 0.000000] [0m[33m DMA zone[0m: 3995 pages, LIFO batch:0 [ 0.000000] [0m[33m DMA32 zone[0m: 8002 pages used for memmap [ 0.000000] [0m[33m DMA32 zone[0m: 512068 pages, LIFO batch:31 [ 0.000000] [0m[33m Normal zone[0m: 24576 pages used for memmap [ 0.000000] [0m[33m Normal zone[0m: 1572864 pages, LIFO batch:31 [ 0.000000] [0m[33mACPI[0m: PM-Timer IO Port: 0x408 [ 0.000000] [0m[33mACPI[0m: Local APIC address 0xfee00000 [ 0.000000] [0m[33mACPI[0m: LAPIC_NMI (acpi_id[0x01] high edge lint[0x1]) [ 0.000000] [0m[33mACPI[0m: LAPIC_NMI (acpi_id[0x02] high edge lint[0x1]) [ 0.000000] [0m[33mIOAPIC[0][0m: apic_id 2, version 32, address 0xfec00000, GSI 0-23 [ 0.000000] [0m[33mACPI[0m: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] [0m[33mACPI[0m: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] [0m[33mACPI[0m: IRQ0 used by override. [ 0.000000] [0m[33mACPI[0m: IRQ9 used by override. [ 0.000000] [0mUsing ACPI (MADT) for SMP configuration information [ 0.000000] [0m[33mACPI[0m: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] [0m[33msmpboot[0m: Allowing 8 CPUs, 0 hotplug CPUs [ 0.000000] [0m[33mPM[0m: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] [0m[33mPM[0m: Registered nosave memory: [mem 0x0009c000-0x0009ffff] [ 0.000000] [0m[33mPM[0m: Registered nosave memory: [mem 0x000a0000-0x000dffff] [ 0.000000] [0m[33mPM[0m: Registered nosave memory: [mem 0x000e0000-0x000fffff] [ 0.000000] [0m[33mPM[0m: Registered nosave memory: [mem 0x7db92000-0x7e5cbfff] [ 0.000000] [0m[33mPM[0m: Registered nosave memory: [mem 0x7e8b8000-0x7f432fff] [ 0.000000] [0m[33mPM[0m: Registered nosave memory: [mem 0x7f433000-0x7f639fff] [ 0.000000] [0m[33mPM[0m: Registered nosave memory: [mem 0x7f800000-0xdfffffff] [ 0.000000] [0m[33mPM[0m: Registered nosave memory: [mem 0xe0000000-0xe3ffffff] [ 0.000000] [0m[33mPM[0m: Registered nosave memory: [mem 0xe4000000-0xfed00fff] [ 0.000000] [0m[33mPM[0m: Registered nosave memory: [mem 0xfed01000-0xfed03fff] [ 0.000000] [0m[33mPM[0m: Registered nosave memory: [mem 0xfed04000-0xfed07fff] [ 0.000000] [0m[33mPM[0m: Registered nosave memory: [mem 0xfed08000-0xfed08fff] [ 0.000000] [0m[33mPM[0m: Registered nosave memory: [mem 0xfed09000-0xfed0bfff] [ 0.000000] [0m[33mPM[0m: Registered nosave memory: [mem 0xfed0c000-0xfed0ffff] [ 0.000000] [0m[33mPM[0m: Registered nosave memory: [mem 0xfed10000-0xfed1bfff] [ 0.000000] [0m[33mPM[0m: Registered nosave memory: [mem 0xfed1c000-0xfed1cfff] [ 0.000000] [0m[33mPM[0m: Registered nosave memory: [mem 0xfed1d000-0xfeefffff] [ 0.000000] [0m[33mPM[0m: Registered nosave memory: [mem 0xfef00000-0xfeffffff] [ 0.000000] [0m[33mPM[0m: Registered nosave memory: [mem 0xff000000-0xff7fffff] [ 0.000000] [0m[33mPM[0m: Registered nosave memory: [mem 0xff800000-0xffffffff] [ 0.000000] [0m[33me820[0m: [mem 0x7f800000-0xdfffffff] available for PCI devices [ 0.000000] [0mBooting paravirtualized kernel on bare hardware [ 0.000000] [0m[33mclocksource[0m: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645519600211568 ns [ 0.000000] [0m[33msetup_percpu[0m: NR_CPUS:256 nr_cpumask_bits:256 nr_cpu_ids:8 nr_node_ids:1 [ 0.000000] [0m[33mPERCPU[0m: Embedded 33 pages/cpu @ffff88027fc00000 s97496 r8192 d29480 u262144 [ 0.000000] [0m[33mpcpu-alloc[0m: s97496 r8192 d29480 u262144 alloc=1*2097152 [ 0.000000] [0m[33mpcpu-alloc[0m: [0] 0 1 2 3 4 5 6 7 [ 0.000000] [0m[33mBuilt 1 zonelists in Node order, mobility grouping on. Total pages[0m: 2056263 [ 0.000000] [0m[33mPolicy zone[0m: Normal [ 0.000000] [0m[33mKernel command line[0m: BOOT_IMAGE=kernels/AVnz7lPEa-28Ib2EfbYQ/vmlinuz boot=rbd rbdroot=10.1.1.15,10.1.1.26,10.1.1.36:AVnz7lLFa-28Ib2EfbX5:AQAGXpBYmSJ5OxAAe1qfEUwBEq3g2B9Rb6cMhA==:instance_images:xenial-AVnz7lPEa-28Ib2EfbYQ:: rw ksdevice=bootif RM_SN=00-00B0-4089100006 INSTANCE_ID=AVnz7lPEa-28Ib2EfbYQ console=ttyS0,115200n8 rootfstype=ext4 initrd=kernels/AVnz7lPEa-28Ib2EfbYQ/initrd.img BOOTIF=01-b0-40-89-10-00-06 [ 0.000000] [0m[33mPID hash table entries[0m: 4096 (order: 3, 32768 bytes) [ 0.000000] [0m[33mCalgary[0m: detecting Calgary via BIOS EBDA area [ 0.000000] [0m[33mCalgary[0m: Unable to locate Rio Grande table in EBDA - bailing! [ 0.000000] [0m[33mMemory[0m: 8119856K/8355708K available (8244K kernel code, 1266K rwdata, 3956K rodata, 1460K init, 1292K bss, 235852K reserved, 0K cma-reserved) [ 0.000000] [0m[33mSLUB[0m: HWalign=64, Order=0-3, MinObjects=0, CPUs=8, Nodes=1 [ 0.000000] [0mHierarchical RCU implementation. [ 0.000000] [0m Build-time adjustment of leaf fanout to 64. [ 0.000000] [0m RCU restricting CPUs from NR_CPUS=256 to nr_cpu_ids=8. [ 0.000000] [0m[33mRCU[0m: Adjusting geometry for rcu_fanout_leaf=64, nr_cpu_ids=8 [ 0.000000] [0mNR_IRQS:16640 nr_irqs:488 16 [ 0.000000] [0m[33mConsole[0m: colour dummy device 80x25 [ 0.000000] [0mconsole [ttyS0] enabled [ 0.000000] [0m[33mclocksource[0m: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 133484882848 ns [ 0.000000] [0mhpet clockevent registered [ 0.000000] [0m[33mtsc[0m: Fast TSC calibration using PIT [ 0.000000] [0m[33mtsc[0m: Detected 2400.097 MHz processor [ 0.000035] [0mCalibrating delay loop (skipped), value calculated using timer frequency.. 4800.19 BogoMIPS (lpj=9600388) [ 0.011887] [0m[33mpid_max[0m: default: 32768 minimum: 301 [ 0.017050] [0m[33mACPI[0m: Core revision 20150930 [ 0.025112] [0m[33mACPI[0m[1m: 2 ACPI AML tables successfully acquired and loaded[0m [ 0.032259] [0mSecurity Framework initialized [ 0.036841] [0m[33mYama[0m: becoming mindful. [ 0.040752] [0m[33mAppArmor[0m: AppArmor initialized [ 0.046117] [0m[33mDentry cache hash table entries[0m: 1048576 (order: 11, 8388608 bytes) [ 0.057980] [0m[33mInode-cache hash table entries[0m: 524288 (order: 10, 4194304 bytes) [ 0.067677] [0m[33mMount-cache hash table entries[0m: 16384 (order: 5, 131072 bytes) [ 0.075399] [0m[33mMountpoint-cache hash table entries[0m: 16384 (order: 5, 131072 bytes) [ 0.083983] [0mInitializing cgroup subsys io [ 0.088486] [0mInitializing cgroup subsys memory [ 0.093358] [0mInitializing cgroup subsys devices [ 0.098330] [0mInitializing cgroup subsys freezer [ 0.103302] [0mInitializing cgroup subsys net_cls [ 0.108272] [0mInitializing cgroup subsys perf_event [ 0.113535] [0mInitializing cgroup subsys net_prio [ 0.118604] [0mInitializing cgroup subsys hugetlb [ 0.123578] [0mInitializing cgroup subsys pids [ 0.128280] [0m[33mCPU[0m: Physical Processor ID: 0 [ 0.132764] [0m[33mCPU[0m: Processor Core ID: 0 [ 0.140641] [0m[33mmce[0m: CPU supports 6 MCE banks [ 0.145132] [0m[33mCPU0[0m: Thermal monitoring enabled (TM1) [ 0.150491] [0m[33mprocess[0m: using mwait in idle threads [ 0.155657] [0m[33mLast level iTLB entries[0m: 4KB 48, 2MB 0, 4MB 0 [ 0.161694] [0m[33mLast level dTLB entries[0m: 4KB 128, 2MB 16, 4MB 16, 1GB 0 [ 0.169005] [0m[33mFreeing SMP alternatives memory[0m: 28K (ffffffff820ab000 - ffffffff820b2000) [ 0.179909] [0m[33mftrace[0m: allocating 31549 entries in 124 pages [ 0.201835] [0m[33m..TIMER[0m: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.248233] [0mTSC deadline timer enabled [ 0.248240] [0m[33msmpboot[0m: CPU0: Intel(R) Atom(TM) CPU C2750 @ 2.40GHz (family: 0x6, model: 0x4d, stepping: 0x8) [ 0.259283] [0m[33mPerformance Events[0m: PEBS fmt2+, 8-deep LBR, Silvermont events, full-width counters, Intel PMU driver. [ 0.270790] [0m[33m... version[0m: 3 [ 0.275267] [0m[33m... bit width[0m: 40 [ 0.279839] [0m[33m... generic registers[0m: 2 [ 0.284316] [0m[33m... value mask[0m: 000000ffffffffff [ 0.290248] [0m[33m... max period[0m: 000000ffffffffff [ 0.296181] [0m[33m... fixed-purpose events[0m: 3 [ 0.300657] [0m[33m... event mask[0m: 0000000700000003 [ 0.307891] [0m[33mx86[0m: Booting SMP configuration: [ 0.312576] [0m[33m.... node #0, CPUs[0m: #1 [ 0.322945] [0m[33mNMI watchdog[0m: enabled on all CPUs, permanently consumes one hw-PMU counter. [ 0.332307] [0m[1m #2 #3 #4 #5 #6 #7[0m [ 0.371651] [0m[33mx86[0m: Booted up 1 node, 8 CPUs [ 0.376331] [0m[33msmpboot[0m: Total of 8 processors activated (38401.55 BogoMIPS) [ 0.385096] [0m[33mdevtmpfs[0m: initialized [ 0.395886] [0m[33mevm[0m: security.selinux [ 0.399588] [0m[33mevm[0m: security.SMACK64 [ 0.403286] [0m[33mevm[0m: security.SMACK64EXEC [ 0.407365] [0m[33mevm[0m: security.SMACK64TRANSMUTE [ 0.411938] [0m[33mevm[0m: security.SMACK64MMAP [ 0.416025] [0m[33mevm[0m: security.ima [ 0.419333] [0m[33mevm[0m: security.capability [ 0.423476] [0m[33mPM[0m: Registering ACPI NVS region [mem 0x7e8b8000-0x7f432fff] (12038144 bytes) [ 0.432930] [0m[33mclocksource[0m: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns [ 0.443997] [0m[33mpinctrl core[0m: initialized pinctrl subsystem [ 0.450034] [0m[33mRTC time[0m: 12:57:16, date: 01/31/17 [ 0.455240] [0m[33mNET[0m: Registered protocol family 16 [ 0.468220] [0m[33mcpuidle[0m: using governor ladder [ 0.480798] [0m[33mcpuidle[0m: using governor menu [ 0.485183] [0m[1mPCCT header not found.[0m [ 0.489115] [0m[33mACPI[0m: bus type PCI registered [ 0.493595] [0m[33macpiphp[0m: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.500906] [0m[33mPCI[0m: MMCONFIG for domain 0000 [bus 00-ff] at [mem 0xe0000000-0xefffffff] (base 0xe0000000) [ 0.511310] [0m[33mPCI[0m: MMCONFIG at [mem 0xe0000000-0xefffffff] reserved in E820 [ 0.518897] [0m[33mPCI[0m: MMCONFIG for 0000 [bus00-3f] at [mem 0xe0000000-0xe3ffffff] (base 0xe0000000) (size reduced!) [ 0.530087] [0m[33mPCI[0m: Using configuration type 1 for base access [ 0.545186] [0m[33mACPI[0m: Added _OSI(Module Device) [ 0.549857] [0m[33mACPI[0m: Added _OSI(Processor Device) [ 0.554822] [0m[33mACPI[0m: Added _OSI(3.0 _SCP Extensions) [ 0.560076] [0m[33mACPI[0m: Added _OSI(Processor Aggregator Device) [ 0.569649] [0m[33mACPI[0m[1m: Dynamic OEM Table Load:[0m [ 0.574147] [0m[33mACPI[0m[1m: SSDT 0xFFFF880275D99800 000490 (v01 PmRef Cpu0Ist 00003000 INTL 20061109)[0m [ 0.584045] [0m[33mACPI[0m[1m: Dynamic OEM Table Load:[0m [ 0.588538] [0m[33mACPI[0m[1m: SSDT 0xFFFF880275F01800 0001FA (v01 PmRef Cpu0Cst 00003001 INTL 20061109)[0m [ 0.598727] [0m[33mACPI[0m[1m: Dynamic OEM Table Load:[0m [ 0.603221] [0m[33mACPI[0m[1m: SSDT 0xFFFF880275D9A000 00047A (v01 PmRef ApIst 00003000 INTL 20061109)[0m [ 0.613357] [0m[33mACPI[0m[1m: Dynamic OEM Table Load:[0m [ 0.617850] [0m[33mACPI[0m[1m: SSDT 0xFFFF880275F01A00 000119 (v01 PmRef ApCst 00003000 INTL 20061109)[0m [ 0.629809] [0m[33mACPI[0m: Interpreter enabled [ 0.633907] [0m[33mACPI Exception[0m[1m: AE_NOT_FOUND, While evaluating Sleep State [\_S1_] (20150930/hwxface-580)[0m [ 0.644229] [0m[33mACPI Exception[0m[1m: AE_NOT_FOUND, While evaluating Sleep State [\_S2_] (20150930/hwxface-580)[0m [ 0.654550] [0m[33mACPI Exception[0m[1m: AE_NOT_FOUND, While evaluating Sleep State [\_S3_] (20150930/hwxface-580)[0m [ 0.664871] [0m[33mACPI Exception[0m[1m: AE_NOT_FOUND, While evaluating Sleep State [\_S4_] (20150930/hwxface-580)[0m [ 0.675200] [0m[33mACPI[0m: (supports S0 S5) [ 0.678994] [0m[33mACPI[0m: Using IOAPIC for interrupt routing [ 0.684640] [0m[33mHEST[0m: Table parsing has been initialized. [ 0.690288] [0m[33mPCI[0m: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.706199] [0m[33mACPI[0m: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.713115] [0m[33macpi PNP0A08:00[0m: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI] [ 0.722728] [0m[33macpi PNP0A08:00[0m: _OSC: OS now controls [PCIeHotplug PME AER PCIeCapability] [ 0.731686] [0m[33macpi PNP0A08:00[0m: [Firmware Info]: MMCONFIG for domain 0000 [bus 00-3f] only partially covers this bridge [ 0.743843] [0mPCI host bridge to bus 0000:00 [ 0.748423] [0m[33mpci_bus 0000:00[0m: root bus resource [io 0x0000-0x0cf7 window] [ 0.756010] [0m[33mpci_bus 0000:00[0m: root bus resource [io 0x0d00-0xffff window] [ 0.763597] [0m[33mpci_bus 0000:00[0m: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.771959] [0m[33mpci_bus 0000:00[0m: root bus resource [mem 0x80000000-0xdfffffff window] [ 0.780323] [0m[33mpci_bus 0000:00[0m: root bus resource [mem 0x280000000-0xfffffffff window] [ 0.788881] [0m[33mpci_bus 0000:00[0m: root bus resource [bus 00-ff] [ 0.795018] [0m[33mpci 0000:00:00.0[0m: [8086:1f01] type 00 class 0x060000 [ 0.795165] [0m[33mpci 0000:00:01.0[0m: [8086:1f10] type 01 class 0x060400 [ 0.795197] [0m[33mpci 0000:00:01.0[0m: reg 0x10: [mem 0xdff80000-0xdff9ffff 64bit] [ 0.795238] [0m[33mpci 0000:00:01.0[0m: PME# supported from D0 D3hot D3cold [ 0.795314] [0m[33mpci 0000:00:01.0[0m: System wakeup disabled by ACPI [ 0.801708] [0m[33mpci 0000:00:02.0[0m: [8086:1f11] type 01 class 0x060400 [ 0.801739] [0m[33mpci 0000:00:02.0[0m: reg 0x10: [mem 0xdff60000-0xdff7ffff 64bit] [ 0.801780] [0m[33mpci 0000:00:02.0[0m: PME# supported from D0 D3hot D3cold [ 0.801853] [0m[33mpci 0000:00:02.0[0m: System wakeup disabled by ACPI [ 0.808244] [0m[33mpci 0000:00:03.0[0m: [8086:1f12] type 01 class 0x060400 [ 0.808275] [0m[33mpci 0000:00:03.0[0m: reg 0x10: [mem 0xdff40000-0xdff5ffff 64bit] [ 0.808315] [0m[33mpci 0000:00:03.0[0m: PME# supported from D0 D3hot D3cold [ 0.808388] [0m[33mpci 0000:00:03.0[0m: System wakeup disabled by ACPI [ 0.814786] [0m[33mpci 0000:00:0e.0[0m: [8086:1f14] type 00 class 0x060000 [ 0.814939] [0m[33mpci 0000:00:0f.0[0m: [8086:1f16] type 00 class 0x080600 [ 0.814996] [0m[33mpci 0000:00:0f.0[0m: PME# supported from D0 D3hot D3cold [ 0.815131] [0m[33mpci 0000:00:13.0[0m: [8086:1f15] type 00 class 0x088000 [ 0.815165] [0m[33mpci 0000:00:13.0[0m: reg 0x10: [mem 0xdffac000-0xdffac3ff 64bit] [ 0.815357] [0m[33mpci 0000:00:14.0[0m: [8086:1f45] type 00 class 0x020000 [ 0.815390] [0m[33mpci 0000:00:14.0[0m: reg 0x10: [mem 0xdff20000-0xdff3ffff 64bit] [ 0.815401] [0m[33mpci 0000:00:14.0[0m: reg 0x18: [io 0xf080-0xf09f] [ 0.815421] [0m[33mpci 0000:00:14.0[0m: reg 0x20: [mem 0xdffa4000-0xdffa7fff 64bit] [ 0.815469] [0m[33mpci 0000:00:14.0[0m: PME# supported from D0 D3hot D3cold [ 0.815613] [0m[33mpci 0000:00:14.1[0m: [8086:1f45] type 00 class 0x020000 [ 0.815647] [0m[33mpci 0000:00:14.1[0m: reg 0x10: [mem 0xdff00000-0xdff1ffff 64bit] [ 0.815657] [0m[33mpci 0000:00:14.1[0m: reg 0x18: [io 0xf060-0xf07f] [ 0.815678] [0m[33mpci 0000:00:14.1[0m: reg 0x20: [mem 0xdffa0000-0xdffa3fff 64bit] [ 0.815726] [0m[33mpci 0000:00:14.1[0m: PME# supported from D0 D3hot D3cold [ 0.815873] [0m[33mpci 0000:00:16.0[0m: [8086:1f2c] type 00 class 0x0c0320 [ 0.815910] [0m[33mpci 0000:00:16.0[0m: reg 0x10: [mem 0xdffab000-0xdffab3ff] [ 0.816007] [0m[33mpci 0000:00:16.0[0m: PME# supported from D0 D3hot D3cold [ 0.816080] [0m[33mpci 0000:00:16.0[0m: System wakeup disabled by ACPI [ 0.822468] [0m[33mpci 0000:00:17.0[0m: [8086:1f22] type 00 class 0x010601 [ 0.822494] [0m[33mpci 0000:00:17.0[0m: reg 0x10: [io 0xf110-0xf117] [ 0.822504] [0m[33mpci 0000:00:17.0[0m: reg 0x14: [io 0xf100-0xf103] [ 0.822514] [0m[33mpci 0000:00:17.0[0m: reg 0x18: [io 0xf0f0-0xf0f7] [ 0.822523] [0m[33mpci 0000:00:17.0[0m: reg 0x1c: [io 0xf0e0-0xf0e3] [ 0.822533] [0m[33mpci 0000:00:17.0[0m: reg 0x20: [io 0xf040-0xf05f] [ 0.822542] [0m[33mpci 0000:00:17.0[0m: reg 0x24: [mem 0xdffaa000-0xdffaa7ff] [ 0.822578] [0m[33mpci 0000:00:17.0[0m: PME# supported from D3hot [ 0.822706] [0m[33mpci 0000:00:18.0[0m: [8086:1f32] type 00 class 0x010601 [ 0.822732] [0m[33mpci 0000:00:18.0[0m: reg 0x10: [io 0xf0d0-0xf0d7] [ 0.822742] [0m[33mpci 0000:00:18.0[0m: reg 0x14: [io 0xf0c0-0xf0c3] [ 0.822752] [0m[33mpci 0000:00:18.0[0m: reg 0x18: [io 0xf0b0-0xf0b7] [ 0.822761] [0m[33mpci 0000:00:18.0[0m: reg 0x1c: [io 0xf0a0-0xf0a3] [ 0.822771] [0m[33mpci 0000:00:18.0[0m: reg 0x20: [io 0xf020-0xf03f] [ 0.822780] [0m[33mpci 0000:00:18.0[0m: reg 0x24: [mem 0xdffa9000-0xdffa97ff] [ 0.822815] [0m[33mpci 0000:00:18.0[0m: PME# supported from D3hot [ 0.822948] [0m[33mpci 0000:00:1f.0[0m: [8086:1f38] type 00 class 0x060100 [ 0.823139] [0m[33mpci 0000:00:1f.3[0m: [8086:1f3c] type 00 class 0x0c0500 [ 0.823160] [0m[33mpci 0000:00:1f.3[0m: reg 0x10: [mem 0xdffa8000-0xdffa801f] [ 0.823194] [0m[33mpci 0000:00:1f.3[0m: reg 0x20: [io 0xf000-0xf01f] [ 0.823404] [0m[33mpci 0000:00:01.0[0m: PCI bridge to [bus 01] [ 0.829027] [0m[33mpci 0000:00:02.0[0m: PCI bridge to [bus 02] [ 0.834666] [0m[33mpci 0000:03:00.0[0m: [8086:f1a5] type 00 class 0x010802 [ 0.834713] [0m[33mpci 0000:03:00.0[0m: reg 0x10: [mem 0xdfe00000-0xdfe03fff 64bit] [ 0.842603] [0m[33mpci 0000:00:03.0[0m: PCI bridge to [bus 03] [ 0.848155] [0m[33mpci 0000:00:03.0[0m: bridge window [mem 0xdfe00000-0xdfefffff] [ 0.848349] [0m[33mACPI[0m: PCI Interrupt Link [LNKA] (IRQs 6 7 10 *11 12) [ 0.855203] [0m[33mACPI[0m: PCI Interrupt Link [LNKB] (IRQs 6 7 10 11 12) *0, disabled. [ 0.863325] [0m[33mACPI[0m: PCI Interrupt Link [LNKC] (IRQs 6 *7 10 11 12) [ 0.870175] [0m[33mACPI[0m: PCI Interrupt Link [LNKD] (IRQs 6 7 10 *11 12) [ 0.877026] [0m[33mACPI[0m: PCI Interrupt Link [LNKE] (IRQs 6 7 *10 11 12) [ 0.883878] [0m[33mACPI[0m: PCI Interrupt Link [LNKF] (IRQs *6 7 10 11 12) [ 0.890732] [0m[33mACPI[0m: PCI Interrupt Link [LNKG] (IRQs 6 *7 10 11 12) [ 0.897584] [0m[33mACPI[0m: PCI Interrupt Link [LNKH] (IRQs 6 7 *10 11 12) [ 0.904767] [0m[33mACPI[0m[1m: Enabled 1 GPEs in block 00 to 1F[0m [ 0.910331] [0m[33mvgaarb[0m: loaded [ 0.913791] [0mSCSI subsystem initialized [ 0.918085] [0mlibata version 3.00 loaded. [ 0.918156] [0m[33mACPI[0m: bus type USB registered [ 0.922668] [0m[33musbcore[0m: registered new interface driver usbfs [ 0.928821] [0m[33musbcore[0m: registered new interface driver hub [ 0.934805] [0m[33musbcore[0m: registered new device driver usb [ 0.940758] [0m[33mPCI[0m: Using ACPI for IRQ routing [ 0.946457] [0m[33mPCI[0m: pci_cache_line_size set to 64 bytes [ 0.946503] [0m[33me820[0m: reserve RAM buffer [mem 0x0009c000-0x0009ffff] [ 0.946506] [0m[33me820[0m: reserve RAM buffer [mem 0x7db92000-0x7fffffff] [ 0.946509] [0m[33me820[0m: reserve RAM buffer [mem 0x7e8b8000-0x7fffffff] [ 0.946511] [0m[33me820[0m: reserve RAM buffer [mem 0x7f800000-0x7fffffff] [ 0.946718] [0m[33mNetLabel[0m: Initializing [ 0.950517] [0m[33mNetLabel[0m: domain hash size = 128 [ 0.955383] [0m[33mNetLabel[0m: protocols = UNLABELED CIPSOv4 [ 0.960949] [0m[33mNetLabel[0m: unlabeled traffic allowed by default [ 0.967330] [0m[33mhpet0[0m: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.972804] [0m[33mhpet0[0m: 3 comparators, 64-bit 14.318180 MHz counter [ 0.981372] [0m[33mclocksource[0m: Switched to clocksource hpet [ 0.998140] [0m[33mAppArmor[0m: AppArmor Filesystem Enabled [ 1.003527] [0m[33mpnp[0m: PnP ACPI init [ 1.007366] [0m[33msystem 00:00[0m: [mem 0xe0000000-0xefffffff] could not be reserved [ 1.015146] [0m[33msystem 00:00[0m: [mem 0x7f800000-0x7fffffff] could not be reserved [ 1.022937] [0m[33msystem 00:00[0m: Plug and Play ACPI device, IDs PNP0c01 (active) [ 1.023021] [0m[33msystem 00:01[0m: [io 0x0680-0x069f] has been reserved [ 1.029644] [0m[33msystem 00:01[0m: [io 0x0400-0x047f] could not be reserved [ 1.036652] [0m[33msystem 00:01[0m: [io 0x0500-0x05fe] has been reserved [ 1.043273] [0m[33msystem 00:01[0m: [mem 0xfed00000-0xfedfffff] could not be reserved [ 1.051059] [0m[33msystem 00:01[0m: [mem 0x000c0000-0x000dffff] could not be reserved [ 1.058844] [0m[33msystem 00:01[0m: [mem 0x000e0000-0x000fffff] could not be reserved [ 1.066630] [0m[33msystem 00:01[0m: [mem 0xffa00000-0xffffffff] has been reserved [ 1.074027] [0m[33msystem 00:01[0m: [mem 0xfee00000-0xfeefffff] has been reserved [ 1.081426] [0m[33msystem 00:01[0m: Plug and Play ACPI device, IDs PNP0c02 (active) [ 1.081485] [0m[33mpnp 00:02[0m: Plug and Play ACPI device, IDs PNP0b00 (active) [ 1.081573] [0m[33mpnp 00:03[0m: Plug and Play ACPI device, IDs PNP0501 (active) [ 1.081654] [0m[33mpnp 00:04[0m: Plug and Play ACPI device, IDs PNP0501 (active) [ 1.081841] [0m[33mpnp[0m: PnP ACPI: found 5 devices [ 1.094719] [0m[33mclocksource[0m: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.104682] [0m[33mpci 0000:00:01.0[0m: PCI bridge to [bus 01] [ 1.110246] [0m[33mpci 0000:00:02.0[0m: PCI bridge to [bus 02] [ 1.115809] [0m[33mpci 0000:00:03.0[0m: PCI bridge to [bus 03] [ 1.121371] [0m[33mpci 0000:00:03.0[0m: bridge window [mem 0xdfe00000-0xdfefffff] [ 1.128971] [0m[33mpci_bus 0000:00[0m: resource 4 [io 0x0000-0x0cf7 window] [ 1.128975] [0m[33mpci_bus 0000:00[0m: resource 5 [io 0x0d00-0xffff window] [ 1.128979] [0m[33mpci_bus 0000:00[0m: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.128982] [0m[33mpci_bus 0000:00[0m: resource 7 [mem 0x80000000-0xdfffffff window] [ 1.128986] [0m[33mpci_bus 0000:00[0m: resource 8 [mem 0x280000000-0xfffffffff window] [ 1.128990] [0m[33mpci_bus 0000:03[0m: resource 1 [mem 0xdfe00000-0xdfefffff] [ 1.129043] [0m[33mNET[0m: Registered protocol family 2 [ 1.134218] [0m[33mTCP established hash table entries[0m: 65536 (order: 7, 524288 bytes) [ 1.142535] [0m[33mTCP bind hash table entries[0m: 65536 (order: 8, 1048576 bytes) [ 1.150366] [0m[33mTCP[0m: Hash tables configured (established 65536 bind 65536) [ 1.157779] [0m[33mUDP hash table entries[0m: 4096 (order: 5, 131072 bytes) [ 1.164667] [0m[33mUDP-Lite hash table entries[0m: 4096 (order: 5, 131072 bytes) [ 1.172129] [0m[33mNET[0m: Registered protocol family 1 [ 1.189522] [0m[33mPCI[0m: CLS 64 bytes, default 64 [ 1.189622] [0mTrying to unpack rootfs image as initramfs... [ 1.687602] [0m[33mFreeing initrd memory[0m: 18864K (ffff88007c926000 - ffff88007db92000) [ 1.695882] [0m[33mPCI-DMA[0m: Using software bounce buffering for IO (SWIOTLB) [ 1.703097] [0msoftware IO TLB [mem 0x78926000-0x7c926000] (64MB) mapped at [ffff880078926000-ffff88007c925fff] [ 1.714378] [0mScanning for low memory corruption every 60 seconds [ 1.721794] [0m[33mfutex hash table entries[0m: 2048 (order: 5, 131072 bytes) [ 1.728874] [0m[33maudit[0m: initializing netlink subsys (disabled) [ 1.734935] [0m[33maudit[0m: type=2000 audit(1485867436.644:1): initialized [ 1.742190] [0mInitialise system trusted keyring [ 1.747281] [0mHugeTLB registered 2 MB page size, pre-allocated 0 pages [ 1.757097] [0m[33mzbud[0m: loaded [ 1.760258] [0m[33mVFS[0m: Disk quotas dquot_6.6.0 [ 1.764710] [0m[33mVFS[0m: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.773332] [0mfuse init (API version 7.23) [ 1.778006] [0mKey type big_key registered [ 1.783129] [0mKey type asymmetric registered [ 1.787719] [0mAsymmetric key parser 'x509' registered [ 1.793255] [0mBlock layer SCSI generic (bsg) driver version 0.4 loaded (major 249) [ 1.801618] [0mio scheduler noop registered [ 1.806013] [0mio scheduler deadline registered (default) [ 1.811840] [0mio scheduler cfq registered [ 1.816930] [0m[33maer 0000:00:01.0:pcie02[0m: service driver aer loaded [ 1.816968] [0m[33maer 0000:00:02.0:pcie02[0m: service driver aer loaded [ 1.817007] [0m[33maer 0000:00:03.0:pcie02[0m: service driver aer loaded [ 1.817036] [0m[33mpcieport 0000:00:01.0[0m: Signaling PME through PCIe PME interrupt [ 1.824825] [0m[33mpcie_pme 0000:00:01.0:pcie01[0m: service driver pcie_pme loaded [ 1.824842] [0m[33mpcieport 0000:00:02.0[0m: Signaling PME through PCIe PME interrupt [ 1.832629] [0m[33mpcie_pme 0000:00:02.0:pcie01[0m: service driver pcie_pme loaded [ 1.832646] [0m[33mpcieport 0000:00:03.0[0m: Signaling PME through PCIe PME interrupt [ 1.840431] [0m[33mpci 0000:03:00.0[0m: Signaling PME through PCIe PME interrupt [ 1.847730] [0m[33mpcie_pme 0000:00:03.0:pcie01[0m: service driver pcie_pme loaded [ 1.847744] [0m[33mpci_hotplug[0m: PCI Hot Plug PCI Core version: 0.5 [ 1.853987] [0m[33mpciehp[0m: PCI Express Hot Plug Controller Driver version: 0.4 [ 1.861442] [0m[33mintel_idle[0m: MWAIT substates: 0x3000020 [ 1.861445] [0m[33mintel_idle[0m: v0.4.1 model 0x4D [ 1.861447] [0m[33mintel_idle[0m: lapic_timer_reliable_states 0xffffffff [ 1.861846] [0m[33minput[0m: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 1.870121] [0m[33mACPI[0m: Power Button [PWRF] [ 1.875178] [0m[33mERST[0m: Error Record Serialization Table (ERST) support is initialized. [ 1.883555] [0m[33mpstore[0m: Registered erst as persistent store backend [ 1.890932] [0m[33mGHES[0m: APEI firmware first mode is enabled by APEI bit and WHEA _OSC. [ 1.899443] [0m[33mSerial[0m: 8250/16550 driver, 32 ports, IRQ sharing enabled [ 1.926701] [0m[33m00:03[0m: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 1.955078] [0m[33m00:04[0m: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 1.966756] [0mLinux agpgart interface v0.103 [ 1.979493] [0m[33mbrd[0m: module loaded [ 1.986313] [0m[33mloop[0m: module loaded [ 1.990228] [0m[33mlibphy[0m: Fixed MDIO Bus: probed [ 1.994815] [0m[33mtun[0m: Universal TUN/TAP device driver, 1.6 [ 2.000462] [0m[33mtun[0m: (C) 1999-2004 Max Krasnyansky <maxk> [ 2.007492] [0mPPP generic driver version 2.4.2 [ 2.012392] [0m[33mehci_hcd[0m: USB 2.0 'Enhanced' Host Controller (EHCI) Driver [ 2.019704] [0m[33mehci-pci[0m: EHCI PCI platform driver [ 2.024842] [0m[33mehci-pci 0000:00:16.0[0m: EHCI Host Controller [ 2.030696] [0m[33mehci-pci 0000:00:16.0[0m: new USB bus registered, assigned bus number 1 [ 2.038979] [0m[33mehci-pci 0000:00:16.0[0m: debug port 2 [ 2.047948] [0m[33mehci-pci 0000:00:16.0[0m: cache line size of 64 is not supported [ 2.047971] [0m[33mehci-pci 0000:00:16.0[0m: irq 23, io mem 0xdffab000 [ 2.065351] [0m[33mehci-pci 0000:00:16.0[0m: USB 2.0 started, EHCI 1.00 [ 2.071847] [0m[33musb usb1[0m: New USB device found, idVendor=1d6b, idProduct=0002 [ 2.079440] [0m[33musb usb1[0m: New USB device strings: Mfr=3, Product=2, SerialNumber=1 [ 2.087517] [0m[33musb usb1[0m: Product: EHCI Host Controller [ 2.092972] [0m[33musb usb1[0m: Manufacturer: Linux 4.4.14-040414-generic ehci_hcd [ 2.100465] [0m[33musb usb1[0m: SerialNumber: 0000:00:16.0 [ 2.105828] [0m[33mhub 1-0:1.0[0m: USB hub found [ 2.110031] [0m[33mhub 1-0:1.0[0m: 8 ports detected [ 2.114862] [0m[33mehci-platform[0m: EHCI generic platform driver [ 2.120726] [0m[33mohci_hcd[0m: USB 1.1 'Open' Host Controller (OHCI) Driver [ 2.127648] [0m[33mohci-pci[0m: OHCI PCI platform driver [ 2.132638] [0m[33mohci-platform[0m: OHCI generic platform driver [ 2.138500] [0m[33muhci_hcd[0m: USB Universal Host Controller Interface driver [ 2.145703] [0m[33mi8042[0m: PNP: No PS/2 controller found. Probing ports directly. [ 2.425285] [0m[33mserio[0m: i8042 KBD port at 0x60,0x64 irq 1 [ 2.425440] [0m[33musb 1-1[0m: new high-speed USB device number 2 using ehci-pci [ 2.438527] [0m[33mmousedev[0m: PS/2 mouse device common for all mice [ 2.456519] [0m[33minput[0m: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 2.466302] [0m[33mrtc_cmos 00:02[0m: RTC can wake from S4 [ 2.471659] [0m[33mrtc_cmos 00:02[0m: rtc core: registered rtc_cmos as rtc0 [ 2.478509] [0m[33mrtc_cmos 00:02[0m: alarms up to one month, 242 bytes nvram, hpet irqs [ 2.486602] [0mi2c /dev entries driver [ 2.490603] [0m[33mdevice-mapper[0m: uevent: version 1.0.3 [ 2.495954] [0m[33mdevice-mapper[0m: ioctl: 4.34.0-ioctl (2015-10-28) initialised: dm-devel [ 2.505419] [0m[33mledtrig-cpu[0m: registered to indicate activity on CPUs [ 2.512994] [0m[33mNET[0m: Registered protocol family 10 [ 2.518458] [0m[33mNET[0m: Registered protocol family 17 [ 2.523460] [0mKey type dns_resolver registered [ 2.528870] [0m[33mmicrocode[0m: CPU0 sig=0x406d8, pf=0x1, revision=0x128 [ 2.535512] [0m[33mmicrocode[0m: CPU1 sig=0x406d8, pf=0x1, revision=0x128 [ 2.542168] [0m[33mmicrocode[0m: CPU2 sig=0x406d8, pf=0x1, revision=0x128 [ 2.548815] [0m[33mmicrocode[0m: CPU3 sig=0x406d8, pf=0x1, revision=0x128 [ 2.555441] [0m[33mmicrocode[0m: CPU4 sig=0x406d8, pf=0x1, revision=0x128 [ 2.562107] [0m[33mmicrocode[0m: CPU5 sig=0x406d8, pf=0x1, revision=0x128 [ 2.568747] [0m[33mmicrocode[0m: CPU6 sig=0x406d8, pf=0x1, revision=0x128 [ 2.575406] [0m[33mmicrocode[0m: CPU7 sig=0x406d8, pf=0x1, revision=0x128 [ 2.582131] [0m[33mmicrocode[0m: Microcode Update Driver: v2.01 <tigran.co.uk>, Peter Oruba [ 2.591975] [0m[33musb 1-1[0m: New USB device found, idVendor=8087, idProduct=07db [ 2.592478] [0mregistered taskstats version 1 [ 2.592502] [0mLoading compiled-in X.509 certificates [ 2.594079] [0m[33mLoaded X.509 cert 'Build time autogenerated kernel key[0m: 52c19602ea6144ad758fa5f376afa3c0a054cacb' [ 2.594120] [0m[33mzswap[0m: loaded using pool lzo/zbud [ 2.597586] [0mKey type trusted registered [ 2.604045] [0mKey type encrypted registered [ 2.604052] [0m[33mAppArmor[0m: AppArmor sha1 policy hashing enabled [ 2.604057] [0m[33mima[0m: No TPM chip found, activating TPM-bypass! [ 2.604107] [0m[33mevm[0m: HMAC attrs: 0x1 [ 2.604577] [0m[33m Magic number[0m: 1:15:988 [ 2.604588] [0m[33mmachinecheck machinecheck1[0m: hash matches [ 2.609419] [0m[33mrtc_cmos 00:02[0m: setting system clock to 2017-01-31 12:57:18 UTC (1485867438) [ 2.630640] [0mBIOS EDD facility v0.16 2004-Jun-25, 0 devices found [ 2.630641] [0mEDD information not available. [ 2.630754] [0m[33mPM[0m: Hibernation image not present or could not be loaded. [ 2.680246] [0m[33musb 1-1[0m: New USB device strings: Mfr=0, Product=0, SerialNumber=0 [ 2.680674] [0m[33mhub 1-1:1.0[0m: USB hub found [ 2.680843] [0m[33mhub 1-1:1.0[0m: 4 ports detected [ 2.699363] [0m[33mFreeing unused kernel memory[0m: 1460K (ffffffff81f3e000 - ffffffff820ab000) [ 2.708135] [0m[33mWrite protecting the kernel read-only data[0m: 14336k [ 2.713406] [0m[33mtsc[0m: Refined TSC clocksource calibration: 2399.999 MHz [ 2.713410] [0m[33mclocksource[0m: tsc: mask: 0xffffffffffffffff max_cycles: 0x2298364cab5, max_idle_ns: 440795214892 ns [ 2.734474] [0m[33mFreeing unused kernel memory[0m: 1984K (ffff880001810000 - ffff880001a00000) [ 2.744112] [0m[33mFreeing unused kernel memory[0m: 140K (ffff880001ddd000 - ffff880001e00000) [ 2.772350] [0m[33mrandom[0m: systemd-udevd urandom read with 9 bits of entropy available [ 2.826885] [0mFUJITSU Extended Socket Network Device Driver - version 1.0 - Copyright (c) 2015 FUJITSU LIMITED [ 2.840744] [0m[33mpps_core[0m: LinuxPPS API ver. 1 registered [ 2.846325] [0m[33mpps_core[0m: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <giometti> [ 2.856952] [0m[33mahci 0000:00:17.0[0m: version 3.0 [ 2.859360] [0mPTP clock support registered [ 2.866427] [0mdca service started, version 1.12.1 [ 2.873425] [0m[33mahci 0000:00:17.0[0m: AHCI 0001.0300 32 slots 4 ports 3 Gbps 0xf impl SATA mode [ 2.882485] [0m[33mahci 0000:00:17.0[0m: flags: 64bit ncq led clo pio deso sadm sds apst [ 2.891153] [0m[33migb[0m: Intel(R) Gigabit Ethernet Network Driver - version 5.3.0-k [ 2.891724] [0m[33mscsi host0[0m: ahci [ 2.892045] [0m[33mscsi host1[0m: ahci [ 2.892300] [0m[33mscsi host2[0m: ahci [ 2.892554] [0m[33mscsi host3[0m: ahci [ 2.892639] [0m[33mata1[0m: SATA max UDMA/133 abar m2048@0xdffaa000 port 0xdffaa100 irq 27 [ 2.892641] [0m[33mata2[0m: SATA max UDMA/133 abar m2048@0xdffaa000 port 0xdffaa180 irq 27 [ 2.892644] [0m[33mata3[0m: SATA max UDMA/133 abar m2048@0xdffaa000 port 0xdffaa200 irq 27 [ 2.892646] [0m[33mata4[0m: SATA max UDMA/133 abar m2048@0xdffaa000 port 0xdffaa280 irq 27 [ 2.892886] [0m[33mahci 0000:00:18.0[0m: AHCI 0001.0300 32 slots 2 ports 6 Gbps 0x3 impl SATA mode [ 2.892890] [0m[33mahci 0000:00:18.0[0m: flags: 64bit ncq led clo pio deso sadm sds apst [ 2.893576] [0m[33mscsi host4[0m: ahci [ 2.893891] [0m[33mscsi host5[0m: ahci [ 2.893976] [0m[33mata5[0m: SATA max UDMA/133 abar m2048@0xdffa9000 port 0xdffa9100 irq 28 [ 2.893978] [0m[33mata6[0m: SATA max UDMA/133 abar m2048@0xdffa9000 port 0xdffa9180 irq 28 [ 2.985064] [0m[33migb[0m: Copyright (c) 2007-2014 Intel Corporation. [ 3.029999] [0m[33migb 0000:00:14.0[0m: added PHC on eth0 [ 3.035072] [0m[33migb 0000:00:14.0[0m: Intel(R) Gigabit Ethernet Network Connection [ 3.042831] [0m[33migb 0000:00:14.0[0m: eth0: PBA No: 001900-000 [ 3.048574] [0m[33migb 0000:00:14.0[0m: Using MSI-X interrupts. 8 rx queue(s), 8 tx queue(s) [ 3.060273] [0m[33m nvme0n1[0m: p1 p2 < p5 > [ 3.095688] [0m[33migb 0000:00:14.1[0m: added PHC on eth1 [ 3.100759] [0m[33migb 0000:00:14.1[0m: Intel(R) Gigabit Ethernet Network Connection [ 3.108519] [0m[33migb 0000:00:14.1[0m: eth1: PBA No: 001900-000 [ 3.114265] [0m[33migb 0000:00:14.1[0m: Using MSI-X interrupts. 8 rx queue(s), 8 tx queue(s) [ 3.124472] [0m[33migb 0000:00:14.0 enp0s20f0[0m: renamed from eth0 [ 3.141735] [0m[33migb 0000:00:14.1 enp0s20f1[0m: renamed from eth1 [ 3.209347] [0m[33mata1[0m: SATA link down (SStatus 0 SControl 300) [ 3.209350] [0m[33mata5[0m: SATA link down (SStatus 0 SControl 300) [ 3.221444] [0m[33mata6[0m: SATA link down (SStatus 0 SControl 300) [ 3.221448] [0m[33mata3[0m: SATA link down (SStatus 0 SControl 300) [ 3.221468] [0m[33mata2[0m: SATA link down (SStatus 0 SControl 300) [ 3.221490] [0m[33mata4[0m: SATA link down (SStatus 0 SControl 300) [ 3.268934] [0mKey type ceph registered [ 3.274194] [0m[33mlibceph[0m: loaded (mon/osd proto 15/24) [ 3.285259] [0m[33mrbd[0m: loaded [ 3.380918] [0m[33migb 0000:00:14.0 enp0s20f0[0m: igb: enp0s20f0 NIC Link is Up 2500 Mbps Full Duplex, Flow Control: RX/TX [ 3.713596] [0m[33mclocksource[0m: Switched to clocksource tsc [ 4.518104] [0m[33mlibceph[0m: client14101 fsid f20fc3cf-19dd-40fa-b507-98be312d3846 [ 4.528208] [0m[33mlibceph[0m: mon1 10.1.1.26:6789 session established [ 4.551610] [0m[33mrbd[0m[1m: image xenial-AVnz7lPEa-28Ib2EfbYQ: WARNING: kernel layering is EXPERIMENTAL![0m [ 4.571247] [0m[33mrbd[0m: rbd0: added with size 0x280000000 [ 4.600562] [0m[33mEXT4-fs (rbd0)[0m: mounted filesystem with ordered data mode. Opts: (null) [ 4.978562] [0m[33msystemd[1][0m: systemd 229 running in system mode. (+PAM +AUDIT +SELINUX +IMA +APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ -LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD -IDN) [ 4.999045] [0m[33msystemd[1][0m: Detected architecture x86-64. [ 5.026256] [0m[33msystemd[1][0m: Set hostname to <application-AVnz7lLFa-28Ib2EfbX5-1>. [ 5.289126] [0m[33msystemd[1][0m: Listening on udev Control Socket. [ 5.309404] [0m[33msystemd[1][0m: Started Forward Password Requests to Wall Directory Watch. [ 5.329350] [0m[33msystemd[1][0m: Listening on udev Kernel Socket. [ 5.345539] [0m[33msystemd[1][0m: Set up automount Arbitrary Executable File Formats File System Automount Point. [ 5.369300] [0m[33msystemd[1][0m: Reached target Swap. [ 5.385355] [0m[33msystemd[1][0m: Listening on /dev/initctl Compatibility Named Pipe. [ 5.405371] [0m[33msystemd[1][0m: Started Dispatch Password Requests to Console Directory Watch. [ 5.429383] [0m[33msystemd[1][0m: Listening on Journal Audit Socket. [ 5.445307] [0m[33msystemd[1][0m: Reached target Remote File Systems (Pre). [ 5.461299] [0m[33msystemd[1][0m: Reached target Remote File Systems. [ 5.477341] [0m[33msystemd[1][0m: Listening on Journal Socket (/dev/log). [ 5.493520] [0m[33msystemd[1][0m: Created slice System Slice. [ 5.509300] [0m[33msystemd[1][0m: Reached target Slices. [ 5.525491] [0m[33msystemd[1][0m: Created slice system-serial\x2dgetty.slice. [ 5.545293] [0m[33msystemd[1][0m: Reached target Paths. [ 5.561321] [0m[33msystemd[1][0m: Listening on Syslog Socket. [ 5.577343] [0m[33msystemd[1][0m: Listening on Journal Socket. [ 5.609334] [0m[33msystemd[1][0m: Mounting Huge Pages File System... [ 5.626432] [0m[33msystemd[1][0m: Starting Journal Service... [ 5.642334] [0m[33msystemd[1][0m: Starting Remount Root and Kernel File Systems... [ 5.666503] [0m[33msystemd[1][0m: Starting Create list of required static device nodes for the current kernel... [ 5.689339] [0m[33msystemd[1][0m: Reached target Sockets. [ 5.737408] [0m[33msystemd[1][0m: Starting Load Kernel Modules... [ 5.754344] [0m[33msystemd[1][0m: Mounting POSIX Message Queue File System... [ 5.776618] [0m[33msystemd[1][0m: Mounting Debug File System... [ 5.793319] [0m[33msystemd[1][0m: Reached target Encrypted Volumes. [ 5.809502] [0m[33msystemd[1][0m: Created slice system-getty.slice. [ 5.838963] [0m[33msystemd[1][0m: Mounted POSIX Message Queue File System. [ 5.857347] [0m[33msystemd[1][0m: Mounted Huge Pages File System. [ 5.873291] [0m[33msystemd[1][0m: Mounted Debug File System. [ 5.889397] [0m[33msystemd[1][0m: Started Journal Service. [ 6.217826] [0m[33msystemd-journald[259][0m: Received request to flush runtime journal from PID 1 [ 6.517626] [0mipmi message handler version 39.2 [ 6.524826] [0mIPMI System Interface driver. [ 6.524863] [0m[33mipmi_si[0m: Adding default-specified kcs state machine [ 6.524867] [0m[33mipmi_si[0m: Trying default-specified kcs state machine at i/o address 0xca2, slave address 0x0, irq 0 [ 6.524875] [0m[33mipmi_si[0m: Interface detection failed [ 6.543754] [0m[33mshpchp[0m: Standard Hot Plug PCI Controller Driver version: 0.4 [ 6.547373] [0m[33mipmi_si[0m: Adding default-specified smic state machine [ 6.547380] [0m[33mipmi_si[0m: Trying default-specified smic state machine at i/o address 0xca9, slave address 0x0, irq 0 [ 6.547388] [0m[33mipmi_si[0m: Interface detection failed [ 6.573436] [0m[33mipmi_si[0m: Adding default-specified bt state machine [ 6.575764] [0m[33mipmi_si[0m: Trying default-specified bt state machine at i/o address 0xe4, slave address 0x0, irq 0 [ 6.575777] [0m[33mipmi_si[0m: Interface detection failed [ 6.594960] [0mSSE version of gcm_enc/dec engaged. [ 6.596823] [0m[33mgpio_ich[0m: GPIO from 452 to 511 on gpio_ich [ 6.601378] [0m[33mipmi_si[0m[1m: Unable to find any System Interface(s)[0m [ 6.672341] [0mIPMI SSIF Interface driver [ 6.750320] [0m[33mrandom[0m: nonblocking pool is initialized [ 6.989349] [0m[33migb 0000:00:14.1 enp0s20f1[0m: igb: enp0s20f1 NIC Link is Up 2500 Mbps Full Duplex, Flow Control: RX/TX [ 25.553003] [0mSGI XFS with ACLs, security attributes, realtime, no debug enabled [ 25.569124] [0m[33mXFS (nvme0n1p1)[0m: Mounting V5 Filesystem [ 25.593336] [0m[33mXFS (nvme0n1p1)[0m: Ending clean mount [ 43.649181] [0m[33mXFS (nvme0n1p1)[0m[7m[31m: Metadata CRC error detected at xfs_agfl_read_verify+0xcd/0x100 [xfs], xfs_agfl block 0x37a2e03[0m [ 43.661668] [0m[33mXFS (nvme0n1p1)[0m[7m[31m: Unmount and run xfs_repair[0m [ 43.667630] [0m[33mXFS (nvme0n1p1)[0m[7m[31m: First 64 bytes of corrupted metadata buffer:[0m [ 43.675271] [0m[33mffff880274a52a00[0m[7m[31m: 11 6e 4d aa 75 2a f0 0d a5 e4 1e fc 8b 70 4a ac .nM.u*.......pJ.[0m [ 43.685079] [0m[33mffff880274a52a10[0m[7m[31m: 1f 11 44 86 da a8 e7 ad bc 06 0f da d1 3a 6c cb ..D..........:l.[0m [ 43.694934] [0m[33mffff880274a52a20[0m[7m[31m: 80 5d 72 69 47 ff 5a 93 9f e1 3b aa ef 0f a4 41 .]riG.Z...;....A[0m [ 43.704706] [0m[33mffff880274a52a30[0m[7m[31m: 55 f1 ae 6a 96 c1 82 39 9b 8e d2 4a a4 b5 2b 58 U..j...9...J..+X[0m [ 43.714576] [0m[33mXFS (nvme0n1p1)[0m[7m[31m: metadata I/O error: block 0x37a2e03 ("xfs_trans_read_buf_map") error 74 numblks 1[0m [ 43.726121] [0m[33mXFS (nvme0n1p1)[0m[7m[31m: Internal error xfs_trans_cancel at line 990 of file /home/kernel/COD/linux/fs/xfs/xfs_trans.c. Caller xfs_inactive_truncate+0xda/0x110 [xfs][0m [ 43.743287] [0m[33mCPU[0m[1m: 0 PID: 893 Comm: rm Not tainted 4.4.14-040414-generic #201606241434[0m [ 43.743319] [0m[33mHardware name[0m[1m: To be filled by O.E.M. To be filled by O.E.M./To be filled by O.E.M., BIOS 5.6.5 10/26/2016[0m [ 43.743324] [0m[1m 0000000000000286 000000009eb89034 ffff880274d3fd98 ffffffff813d0743[0m [ 43.743332] [0m[1m ffff880274b30910 0000000000000001 ffff880274d3fdb0 ffffffffc030b70c[0m [ 43.743336] [0m[1m ffffffffc031c0da ffff880274d3fdd8 ffffffffc0326d96 00000000ffffff8b[0m [ 43.743340] [0m[1mCall Trace:[0m [ 43.743376] [0m[1m [<ffffffff813d0743>] dump_stack+0x63/0x90[0m [ 43.743805] [0m[1m [<ffffffffc030b70c>] xfs_error_report+0x3c/0x40 [xfs][0m [ 43.743868] [0m[1m [<ffffffffc031c0da>] ? xfs_inactive_truncate+0xda/0x110 [xfs][0m [ 43.743908] [0m[1m [<ffffffffc0326d96>] xfs_trans_cancel+0xb6/0xe0 [xfs][0m [ 43.743944] [0m[1m [<ffffffffc031c0da>] xfs_inactive_truncate+0xda/0x110 [xfs][0m [ 43.743994] [0m[1m [<ffffffffc031ce12>] xfs_inactive+0x102/0x120 [xfs][0m [ 43.744028] [0m[1m [<ffffffffc03222e5>] xfs_fs_evict_inode+0xa5/0x100 [xfs][0m [ 43.744042] [0m[1m [<ffffffff812253ae>] evict+0xbe/0x190[0m [ 43.744046] [0m[1m [<ffffffff81225691>] iput+0x1c1/0x240[0m [ 43.744056] [0m[1m [<ffffffff8121a199>] do_unlinkat+0x199/0x2d0[0m [ 43.744061] [0m[1m [<ffffffff8121ad0b>] SyS_unlinkat+0x1b/0x30[0m [ 43.744076] [0m[1m [<ffffffff81808772>] entry_SYSCALL_64_fastpath+0x16/0x71[0m [ 43.744092] [0m[33mXFS (nvme0n1p1)[0m: xfs_do_force_shutdown(0x8) called from line 991 of file /home/kernel/COD/linux/fs/xfs/xfs_trans.c. Return address = 0xffffffffc0326daf [ 43.748591] [0m[33mXFS (nvme0n1p1)[0m[7m[31m: Corruption of in-memory data detected. Shutting down filesystem[0m [ 43.758185] [0m[33mXFS (nvme0n1p1)[0m[7m[31m: Please umount the filesystem and rectify the problem(s)[0m [ 49.895065] [0m[33mlibceph[0m[1m: osd1 10.1.1.26:6800 socket closed (con state OPEN)[0m [ 49.898898] [0m[33mlibceph[0m[1m: osd1 10.1.1.26:6800 socket closed (con state OPEN)[0m [ 49.902393] [0m[33mlibceph[0m[1m: osd1 10.1.1.26:6800 socket closed (con state OPEN)[0m [ 49.905527] [0m[33mlibceph[0m[1m: osd1 10.1.1.26:6800 socket closed (con state OPEN)[0m [ 49.908276] [0m[33mlibceph[0m[1m: osd1 10.1.1.26:6800 socket closed (con state OPEN)[0m [ 49.911427] [0m[33mlibceph[0m[1m: osd1 10.1.1.26:6800 socket closed (con state OPEN)[0m [ 49.914404] [0m[33mlibceph[0m[1m: osd1 10.1.1.26:6800 socket closed (con state OPEN)[0m [ 49.917446] [0m[33mlibceph[0m[1m: osd1 10.1.1.26:6800 socket closed (con state OPEN)[0m [ 49.919995] [0m[33mlibceph[0m[1m: osd1 10.1.1.26:6800 socket closed (con state OPEN)[0m [ 55.604390] [0m[33mXFS (nvme0n1p1)[0m[1m: xfs_log_force: error -5 returned.[0m [ 85.683948] [0m[33mXFS (nvme0n1p1)[0m[1m: xfs_log_force: error -5 returned.[0m # mount sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime) proc on /proc type proc (rw,nosuid,nodev,noexec,relatime) udev on /dev type devtmpfs (rw,nosuid,relatime,size=4059940k,nr_inodes=1014985,mode=755) devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000) tmpfs on /run type tmpfs (rw,nosuid,noexec,relatime,size=814236k,mode=755) /dev/rbd0 on / type ext4 (rw,relatime,stripe=1024,data=ordered) securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime) tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev) tmpfs on /run/lock type tmpfs (rw,nosuid,nodev,noexec,relatime,size=5120k) tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755) cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/lib/systemd/systemd-cgroups-agent,name=systemd) pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime) cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio) cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,hugetlb) cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids) cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset) cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer) cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls,net_prio) cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event) cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct) cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices) cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory) systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=26,pgrp=1,timeout=0,minproto=5,maxproto=5,direct) hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime) mqueue on /dev/mqueue type mqueue (rw,relatime) debugfs on /sys/kernel/debug type debugfs (rw,relatime) fusectl on /sys/fs/fuse/connections type fusectl (rw,relatime) /dev/nvme0n1p1 on /mnt/ssd type xfs (rw,relatime,attr2,inode64,noquota) # umount /mnt/ssd # xfs_repair /dev/nvme0n1p1 Phase 1 - find and verify superblock... couldn't verify primary superblock - not enough secondary superblocks with matching geometry !!! attempting to find secondary superblock... <snipped lots of .....> found candidate secondary superblock... unable to verify superblock, continuing... <snipped lots of .....> Sorry, could not find valid secondary superblock Exiting now. # xfs_metadump /dev/nvme0n1p1 /tmp/xfs_dump bad magic number xfs_metadump: cannot read superblock for ag 1 Metadata CRC error detected at xfs_agfl block 0x37a2e03/0x200 bad magic number xfs_metadump: cannot read superblock for ag 3 Metadata CRC error detected at xfs_agfl block 0xa6e8a03/0x200 xfs_metadump: Filesystem log is dirty; image will contain unobfuscated metadata in log. cache_purge: shake on cache 0x9e4ba0 left 2 nodes!? If the xfs_metadump output is useful, I will find somewhere to post it.
Hi, not sure if this is still ongoing, but I just stumbled on this and wanted to help. The data corruption is caused by a caching bug in the NVMe device, and firmware revision PSF119K fixes it.
Travis Gummels: that bugzilla link is not public and I cannot view it.
(In reply to Keith Busch from comment #14) > Hi, not sure if this is still ongoing, but I just stumbled on this and > wanted to help. The data corruption is caused by a caching bug in the NVMe > device, and firmware revision PSF119K fixes it. I'll try this version of the firmware over the next couple of days and report back here.
(In reply to Neil from comment #16) > (In reply to Keith Busch from comment #14) > > Hi, not sure if this is still ongoing, but I just stumbled on this and > > wanted to help. The data corruption is caused by a caching bug in the NVMe > > device, and firmware revision PSF119K fixes it. > > I'll try this version of the firmware over the next couple of days and > report back here. Hm, I can't find a Linux tool to perform the firmware update. Keith, any pointers?
(In reply to Neil from comment #17) > Hm, I can't find a Linux tool to perform the firmware update. Keith, any > pointers? I think Intel provides a bootable iso on the same page as the sad toolbox. Read the description and it should be mentioned there
The firmware is not publicly available as yet. I have escalated this issue to the group who owns this component as high as I can, and waiting to hear the release ETA.
I am experiencing this exact issue with an Intel 600p NVMe SSD on brand new loads of CentOS 7 and Fedora 25. Strangely enough, I loaded Windows 10 on the same device as a test and it worked fine (no corruption). I'm running firmware 109C on my SSD.
(In reply to Greg Rapp from comment #20) > Strangely enough, I loaded Windows 10 on the same device as a test and it worked fine > (no corruption). I'm running firmware 109C on my SSD. I think that makes sense. The issue requires conditions that don't occur in file systems that are not XFS.
Keith, can you elaborate on what conditions those are? Is it related to the 512-byte IO xfs does for logging? I'm wondering what xfs does differently, here. Thanks, -Eric
(In reply to Eric Sandeen from comment #22) > Keith, can you elaborate on what conditions those are? Is it related to the > 512-byte IO xfs does for logging? I'm wondering what xfs does differently, > here. AIUI, partial page writes are related, so your suspicion sounds like it's on the right track. I get my information indirectly though, so I may be remiss to provide additional details before release notes are published.
Im experiencing this on an intel 600P being shared through Vmware to Redhat EL7 and Centos 7 machines.
I'm experiencing this across a cluster of machines with the Intel 600p running CentOS 7.
I'm also experiencing this across a cluster of machines with the Intel 600p running CentOS 7.
Sorry for the delay. I've just been told the official firmware update will be made generally available in May. I do not have a more precise date.
(In reply to Keith Busch from comment #23) > (In reply to Eric Sandeen from comment #22) > > Keith, can you elaborate on what conditions those are? Is it related to the > > 512-byte IO xfs does for logging? I'm wondering what xfs does differently, > > here. > > AIUI, partial page writes are related, so your suspicion sounds like it's on > the right track. I get my information indirectly though, so I may be remiss > to provide additional details before release notes are published. Ok, fair enough. It'd be interesting to know if this problem also occurs on, say, 1024-byte fs block sized ext4 filesystems. Or if 512-sector-sized direct IO on a 4k ext4 filesystem has issues ... Mostly I'd just like to know what to look for when we get "filesystem" bug reports triggered by this problem. -Eric
metoo. running centos7 from sata ssd, installed brand new intel pcie ssd, partition, mkfs.xfs, mount, fill with data, umount, reboot, xfs filesystem is corrupted. mount says: [root@alpha00 ~]# mount /dev/nvme0n1p2 /mnt/tmp mount: mount /dev/nvme0n1p2 on /mnt/tmp failed: Structure needs cleaning xfs_repair says: [root@alpha00 ~]# xfs_repair /dev/nvme0n1p2 Phase 1 - find and verify superblock... couldn't verify primary superblock - not enough secondary superblocks with matching geometry !!! then prints many dots. Intel's name is mud. [root@alpha00 ~]# nvme list Node SN Model Version Namespace Usage Format FW Rev ---------------- -------------------- ---------------------------------------- -------- --------- -------------------------- ---------------- -------- /dev/nvme0n1 BTPY63260MME128A INTEL SSDPEKKW128G7 1.2 1 128.04 GB / 128.04 GB 512 B + 0 B PSF100C [root@alpha00 ~]# K.O.
Fixed for me. Installed intel firmware update from https://downloadcenter.intel.com/download/26491?v=t file issdfut_2.1.7.iso, xfs corruption is gone. now will it boot? [root@alpha00 ~]# nvme list Node SN Model Version Namespace Usage Format FW Rev ---------------- -------------------- ---------------------------------------- -------- --------- -------------------------- ---------------- -------- /dev/nvme0n1 BTPY63260MME128A INTEL SSDPEKKW128G7 1.2 1 128.04 GB / 128.04 GB 512 B + 0 B PSF109C [root@alpha00 ~]# K.O.
False success. xfs filesystem mounted after reboot, then immediately corrupted on first write. intel's name is mud. K.O.
Konstantin: According to Keith (from Intel) post above, you need firmware PSF119K, which is not yet public until May.
*********** MASS BUG UPDATE ************** We apologize for the inconvenience. There is a large number of bugs to go through and several of them have gone stale. Due to this, we are doing a mass bug update across all of the Fedora 25 kernel bugs. Fedora 25 has now been rebased to 4.10.9-200.fc25. Please test this kernel update (or newer) and let us know if you issue has been resolved or if it is still present with the newer kernel. If you have moved on to Fedora 26, and are still experiencing this issue, please change the version to Fedora 26. If you experience different issues, please open a new bug report for those.
This issue is technically not kernel related as it is a firmware bug in the relevant SSD models. It would, however, be useful to keep this bug around so that hopefully someone will add an update if the updated firmware release is made public. I guess that technically it doesn't have to be open to be useful for that, but having it open does increase the chance that someone who has a similar problem will find it.
Intel firmware package was updated to issdfut_2.2.0 https://downloadcenter.intel.com/download/26491?v=t 600p firmware is still 109C, with this problem. Waiting. K.O.
Using blk-trace, I found that 600p is handling FUA commands incredibly fast. So fast that it's only 1/10 the time it takes to flush the same amount data. It's about 100us to (buffered) write 32K data, while it takes over 800us to flush it. On the other hand, to FUA write 4K data, it only takes about 50us, which is much the same time it takes to do *buffered* write. So in short, it takes about 110us to write and flush 4K data, while FUA only takes half the time. If it's really the FUA problem, then all fs addressing FUA commands will be affected. Maybe adding a new module parameter for nvme host module to disable FUA support and fall back to write + flush will be a workaround?
I think I'm also hit by this bug on an Intel 600p SSD (Non-Volatile memory controller: Intel Corporation Device f1a5 (rev 03)). LVM+XFS installation does not fail later than after reboot. However, also LVM+etx4 fails during Installation for me. Really hadn't thought, that a company like Intel does not do the _smallest amount_ of testing before releasing their products. What a confession of failure. Does anyone have information about the ETA of the new firmware?
Created attachment 1278035 [details] Optimally aligns IO The official word I have is: “We are in the process of validating a solution in the Firmware that we believe will resolve this issue. Current plan is for this to be available in an upcoming Intel® SSD Toolbox and Intel® FUT release. Release date is still to be determined, but we will update you once we have a schedule” I also understand the firmware update may use a newer version than 119K that I mentioned earlier, so please just be aware of that. In the meantime, I've attached a patch against current Linux mainline that marginally helps performance on this family of devices and has a side effect of forcing all page aligned/sized access: it makes it impossible for xfs (or any filesystem) to issue sub-page writes. I am not planning to push this upstream, though.
Intel firmware package was updated to issdfut_2.2.1 https://downloadcenter.intel.com/download/26491?v=t 600p firmware is 121C, according to release notes https://downloadmirror.intel.com/26491/eng/Intel_SSD_Firmware_Update_Tool_2_2_1_Release_Notes_017.pdf This firmware versions contain the following enhancements: • Improved link initializations This firmware versions contain fixes for the following issues: • Drive hangs intermittently after Format NVM command. • Format NVM command occasionally failing with PCIe ASPM enabled. • Data miscompare caused by intermittent data corruption during heavy write workload with small file transfer size. • Incorrect drive behavior for command with Forced Unit Access setting. K.O.
Installed firmware 121C from the issdfut_2.2.1 package. There is some improvement. Before update, the CentOS7 installer died with XFS filesystem corruption errors. After the update, the CentOS7 installer froze around package 600 (out of 2000 or so). As a try, I ran the CentOS7 installer against a kingston 120GB SATA SSD (no freeze, no crash, full success), then copied the XFS filesystem to the Intel SSD (rsync, voodoo dance with grub2 and restorecon). No freeze, no corruption, successful boot from Intel SSD. Connected network, run "yum update" to install latest updates, reboot into latest CentOS7 kernel, all good so far. So the best I can tell, with this new firmware, the Intel SSD now seems to be usable. I have two more blank machines where I can run the CentOS7 installer to see if the freeze during installation was a fluke or all machines do this. Big thanks to whoever got this firmware update out the door at intel. K.O.
I have installed the latest firmware 121C as well. However on my HP Zbook Studio G3 I still get a freeze during installation using the latest images of CentOS 7 as well as a freeze after some seconds when logging into a freshly installed Debian 9. I am adding the kern.log from the Debian installation here in the hope that this may help to debug the issue. I will also try to build and install a fresh kernel from kernel.org this weekend to see if the issue is still present in the latest upstream sources.
This is the part from the log detailing the crash: May 30 21:13:45 debian kernel: [ 51.634914] INFO: rcu_sched self-detected stall on CPU May 30 21:13:45 debian kernel: [ 51.634918] 5-...: (5249 ticks this GP) idle=0e5/140000000000001/0 softirq=2613/2613 fqs=2624 May 30 21:13:45 debian kernel: [ 51.634919] (t=5250 jiffies g=590 c=589 q=832) May 30 21:13:45 debian kernel: [ 51.634920] Task dump for CPU 5: May 30 21:13:45 debian kernel: [ 51.634921] kworker/5:2 R running task 0 345 2 0x00000008 May 30 21:13:45 debian kernel: [ 51.634926] Workqueue: pm pm_runtime_work May 30 21:13:45 debian kernel: [ 51.634927] ffffffffa19132c0 ffffffffa0ca3bbb 0000000000000005 ffffffffa19132c0 May 30 21:13:45 debian kernel: [ 51.634928] ffffffffa0d7a3b6 ffff95b1ff558fc0 ffffffffa184a640 0000000000000000 May 30 21:13:45 debian kernel: [ 51.634930] ffffffffa19132c0 00000000ffffffff ffffffffa0cdedf4 00000000009e3378 May 30 21:13:45 debian kernel: [ 51.634931] Call Trace: May 30 21:13:45 debian kernel: [ 51.634932] <IRQ> May 30 21:13:45 debian kernel: [ 51.634958] [<ffffffffa0ca3bbb>] ? sched_show_task+0xcb/0x130 May 30 21:13:45 debian kernel: [ 51.634960] [<ffffffffa0d7a3b6>] ? rcu_dump_cpu_stacks+0x92/0xb2 May 30 21:13:45 debian kernel: [ 51.634961] [<ffffffffa0cdedf4>] ? rcu_check_callbacks+0x754/0x8a0 May 30 21:13:45 debian kernel: [ 51.634973] [<ffffffffa0ced0b3>] ? update_wall_time+0x473/0x790 May 30 21:13:45 debian kernel: [ 51.634991] [<ffffffffa0cf48b0>] ? tick_sched_handle.isra.12+0x50/0x50 May 30 21:13:45 debian kernel: [ 51.634993] [<ffffffffa0ce5708>] ? update_process_times+0x28/0x50 May 30 21:13:45 debian kernel: [ 51.634994] [<ffffffffa0cf4880>] ? tick_sched_handle.isra.12+0x20/0x50 May 30 21:13:45 debian kernel: [ 51.634994] [<ffffffffa0cf48e8>] ? tick_sched_timer+0x38/0x70 May 30 21:13:45 debian kernel: [ 51.634995] [<ffffffffa0ce60ec>] ? __hrtimer_run_queues+0xdc/0x240 May 30 21:13:45 debian kernel: [ 51.634996] [<ffffffffa0ce67bc>] ? hrtimer_interrupt+0x9c/0x1a0 May 30 21:13:45 debian kernel: [ 51.634998] [<ffffffffa1206a69>] ? smp_apic_timer_interrupt+0x39/0x50 May 30 21:13:45 debian kernel: [ 51.634999] [<ffffffffa1205d82>] ? apic_timer_interrupt+0x82/0x90 May 30 21:13:45 debian kernel: [ 51.635013] <EOI> May 30 21:13:45 debian kernel: [ 51.635015] [<ffffffffa0f431cb>] ? ioread32+0x2b/0x30 May 30 21:13:45 debian kernel: [ 51.635043] [<ffffffffc0bc111f>] ? nvkm_pmu_init+0x6f/0x450 [nouveau] May 30 21:13:45 debian kernel: [ 51.635053] [<ffffffffc0b7959c>] ? nvkm_subdev_init+0x8c/0x1f0 [nouveau] May 30 21:13:45 debian kernel: [ 51.635072] [<ffffffffc0bcc913>] ? nvkm_device_init+0x163/0x260 [nouveau] May 30 21:13:45 debian kernel: [ 51.635089] [<ffffffffc0bd0791>] ? nvkm_udevice_init+0x41/0x60 [nouveau] May 30 21:13:45 debian kernel: [ 51.635098] [<ffffffffc0b7806b>] ? nvkm_object_init+0x3b/0x180 [nouveau] May 30 21:13:45 debian kernel: [ 51.635107] [<ffffffffc0b780db>] ? nvkm_object_init+0xab/0x180 [nouveau] May 30 21:13:45 debian kernel: [ 51.635124] [<ffffffffc0c11c14>] ? nouveau_do_resume+0x44/0x120 [nouveau] May 30 21:13:45 debian kernel: [ 51.635142] [<ffffffffc0c11fd6>] ? nouveau_pmops_runtime_resume+0x76/0x110 [nouveau] May 30 21:13:45 debian kernel: [ 51.635144] [<ffffffffa0f763b8>] ? pci_pm_runtime_resume+0x78/0xa0 May 30 21:13:45 debian kernel: [ 51.635145] [<ffffffffa107d55e>] ? __rpm_callback+0x2e/0x70 May 30 21:13:45 debian kernel: [ 51.635147] [<ffffffffa0f76340>] ? pci_restore_standard_config+0x40/0x40 May 30 21:13:45 debian kernel: [ 51.635148] [<ffffffffa107d5bf>] ? rpm_callback+0x1f/0x70 May 30 21:13:45 debian kernel: [ 51.635149] [<ffffffffa0f76340>] ? pci_restore_standard_config+0x40/0x40 May 30 21:13:45 debian kernel: [ 51.635150] [<ffffffffa107dd79>] ? rpm_resume+0x499/0x6a0 May 30 21:13:45 debian kernel: [ 51.635151] [<ffffffffa107ed60>] ? pm_runtime_work+0x50/0x90 May 30 21:13:45 debian kernel: [ 51.635153] [<ffffffffa0c90374>] ? process_one_work+0x184/0x410 May 30 21:13:45 debian kernel: [ 51.635153] [<ffffffffa0c9064d>] ? worker_thread+0x4d/0x480 May 30 21:13:45 debian kernel: [ 51.635154] [<ffffffffa0c90600>] ? process_one_work+0x410/0x410 May 30 21:13:45 debian kernel: [ 51.635155] [<ffffffffa0c7bafa>] ? do_group_exit+0x3a/0xa0 May 30 21:13:45 debian kernel: [ 51.635157] [<ffffffffa0c965c7>] ? kthread+0xd7/0xf0 May 30 21:13:45 debian kernel: [ 51.635158] [<ffffffffa0c964f0>] ? kthread_park+0x60/0x60 May 30 21:13:45 debian kernel: [ 51.635160] [<ffffffffa12043b5>] ? ret_from_fork+0x25/0x30 May 30 21:14:10 debian kernel: [ 76.506589] NMI watchdog: BUG: soft lockup - CPU#5 stuck for 23s! [kworker/5:2:345] May 30 21:14:10 debian kernel: [ 76.506592] Modules linked in: rfcomm intel_rapl snd_hda_codec_conexant snd_hda_codec_generic x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass crct10dif_pclmul crc32_pclmul ghash_clmulni_intel intel_cstate cmac bnep hid_generic nls_ascii nls_cp437 vfat fat i2c_designware_platform i2c_designware_core hp_wmi sparse_keymap uvcvideo intel_uncore videobuf2_vmalloc videobuf2_memops videobuf2_v4l2 intel_rapl_perf videobuf2_core evdev videodev joydev efi_pstore pcspkr media efivars iwlwifi nouveau snd_hda_intel serio_raw snd_hda_codec i915 rtsx_pci_ms cfg80211 mxm_wmi snd_hda_core memstick snd_hwdep ttm snd_pcm snd_timer btusb drm_kms_helper snd iTCO_wdt btrtl iTCO_vendor_support soundcore drm idma64 sg virt_dma mei_me i2c_algo_bit shpchp mei processor_thermal_device intel_lpss_pci May 30 21:14:10 debian kernel: [ 76.506640] intel_soc_dts_iosf battery hci_uart btbcm btqca btintel int3403_thermal bluetooth int340x_thermal_zone int3400_thermal hp_accel intel_lpss_acpi lis3lv02d intel_lpss rfkill acpi_thermal_rel input_polldev video wmi tpm_infineon hp_wireless ac acpi_pad button parport_pc ppdev lp parport efivarfs ip_tables x_tables autofs4 ext4 crc16 jbd2 crc32c_generic fscrypto ecb mbcache sd_mod uas usb_storage rtsx_pci_sdmmc mmc_core crc32c_intel aesni_intel aes_x86_64 glue_helper lrw gf128mul ablk_helper cryptd psmouse e1000e i2c_i801 ptp ahci i2c_smbus pps_core xhci_pci libahci rtsx_pci xhci_hcd mfd_core libata nvme nvme_core scsi_mod usbcore usb_common thermal i2c_hid hid May 30 21:14:10 debian kernel: [ 76.506691] CPU: 5 PID: 345 Comm: kworker/5:2 Not tainted 4.9.0-3-amd64 #1 Debian 4.9.25-1 May 30 21:14:10 debian kernel: [ 76.506691] Hardware name: HP Studio G3/80D4, BIOS N82 Ver. 01.16 04/14/2017 May 30 21:14:10 debian kernel: [ 76.506695] Workqueue: pm pm_runtime_work May 30 21:14:10 debian kernel: [ 76.506695] task: ffff95b1d5b9f080 task.stack: ffffa3d043998000 May 30 21:14:10 debian kernel: [ 76.506696] RIP: 0010:[<ffffffffa0f431cb>] [<ffffffffa0f431cb>] ioread32+0x2b/0x30 May 30 21:14:10 debian kernel: [ 76.506699] RSP: 0018:ffffa3d04399bbc0 EFLAGS: 00000296 May 30 21:14:10 debian kernel: [ 76.506699] RAX: 00000000ffffffff RBX: ffff95b1d540e400 RCX: 0000000000000018 May 30 21:14:10 debian kernel: [ 76.506699] RDX: ffffa3d05110a014 RSI: ffffa3d05110a014 RDI: ffffa3d05110a04c May 30 21:14:10 debian kernel: [ 76.506700] RBP: 0000000000000000 R08: 0000000000009bd4 R09: 0000000000009bd4 May 30 21:14:10 debian kernel: [ 76.506700] R10: 0000000000000003 R11: 0000000000000000 R12: ffff95b1d9f2c9c8 May 30 21:14:10 debian kernel: [ 76.506701] R13: ffff95b1d6b68a80 R14: ffffffffffffffff R15: 0000000000000001 May 30 21:14:10 debian kernel: [ 76.506701] FS: 0000000000000000(0000) GS:ffff95b1ff540000(0000) knlGS:0000000000000000 May 30 21:14:10 debian kernel: [ 76.506702] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 May 30 21:14:10 debian kernel: [ 76.506702] CR2: 00007fc68400c650 CR3: 00000005ef407000 CR4: 00000000003406e0 May 30 21:14:10 debian kernel: [ 76.506703] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 May 30 21:14:10 debian kernel: [ 76.506703] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 May 30 21:14:10 debian kernel: [ 76.506704] Stack: May 30 21:14:10 debian kernel: [ 76.506704] ffffffffc0bc111f ffff95b1d9f2c9c8 ffff95b1d540e400 00000007217ea463 May 30 21:14:10 debian kernel: [ 76.506705] 0000000000000000 000000071be29d69 ffffffffc0b7959c 0000000000000011 May 30 21:14:10 debian kernel: [ 76.506706] ffff95b1d540e400 0000000000000000 0000000000000000 ffffffffc0bcc913 May 30 21:14:10 debian kernel: [ 76.506708] Call Trace: May 30 21:14:10 debian kernel: [ 76.506729] [<ffffffffc0bc111f>] ? nvkm_pmu_init+0x6f/0x450 [nouveau] May 30 21:14:10 debian kernel: [ 76.506738] [<ffffffffc0b7959c>] ? nvkm_subdev_init+0x8c/0x1f0 [nouveau] May 30 21:14:10 debian kernel: [ 76.506754] [<ffffffffc0bcc913>] ? nvkm_device_init+0x163/0x260 [nouveau] May 30 21:14:10 debian kernel: [ 76.506769] [<ffffffffc0bd0791>] ? nvkm_udevice_init+0x41/0x60 [nouveau] May 30 21:14:10 debian kernel: [ 76.506778] [<ffffffffc0b7806b>] ? nvkm_object_init+0x3b/0x180 [nouveau] May 30 21:14:10 debian kernel: [ 76.506786] [<ffffffffc0b780db>] ? nvkm_object_init+0xab/0x180 [nouveau] May 30 21:14:10 debian kernel: [ 76.506802] [<ffffffffc0c11c14>] ? nouveau_do_resume+0x44/0x120 [nouveau] May 30 21:14:10 debian kernel: [ 76.506818] [<ffffffffc0c11fd6>] ? nouveau_pmops_runtime_resume+0x76/0x110 [nouveau] May 30 21:14:10 debian kernel: [ 76.506820] [<ffffffffa0f763b8>] ? pci_pm_runtime_resume+0x78/0xa0 May 30 21:14:10 debian kernel: [ 76.506820] [<ffffffffa107d55e>] ? __rpm_callback+0x2e/0x70 May 30 21:14:10 debian kernel: [ 76.506822] [<ffffffffa0f76340>] ? pci_restore_standard_config+0x40/0x40 May 30 21:14:10 debian kernel: [ 76.506822] [<ffffffffa107d5bf>] ? rpm_callback+0x1f/0x70 May 30 21:14:10 debian kernel: [ 76.506824] [<ffffffffa0f76340>] ? pci_restore_standard_config+0x40/0x40 May 30 21:14:10 debian kernel: [ 76.506824] [<ffffffffa107dd79>] ? rpm_resume+0x499/0x6a0 May 30 21:14:10 debian kernel: [ 76.506825] [<ffffffffa107ed60>] ? pm_runtime_work+0x50/0x90 May 30 21:14:10 debian kernel: [ 76.506826] [<ffffffffa0c90374>] ? process_one_work+0x184/0x410 May 30 21:14:10 debian kernel: [ 76.506827] [<ffffffffa0c9064d>] ? worker_thread+0x4d/0x480 May 30 21:14:10 debian kernel: [ 76.506828] [<ffffffffa0c90600>] ? process_one_work+0x410/0x410 May 30 21:14:10 debian kernel: [ 76.506829] [<ffffffffa0c7bafa>] ? do_group_exit+0x3a/0xa0 May 30 21:14:10 debian kernel: [ 76.506830] [<ffffffffa0c965c7>] ? kthread+0xd7/0xf0 May 30 21:14:10 debian kernel: [ 76.506831] [<ffffffffa0c964f0>] ? kthread_park+0x60/0x60 May 30 21:14:10 debian kernel: [ 76.506833] [<ffffffffa12043b5>] ? ret_from_fork+0x25/0x30 May 30 21:14:10 debian kernel: [ 76.506833] Code: 48 81 ff ff ff 03 00 77 20 48 81 ff 00 00 01 00 76 05 0f b7 d7 ed c3 48 c7 c6 09 4b 62 a1 e8 1d ff ff ff b8 ff ff ff ff c3 8b 07 <c3> 0f 1f 40 00 48 81 fe ff ff 03 00 48 89 f2 77 1f 48 81 fe 00
I am seeing similar issues with yesterday's Rawhide installed to the SSD: May 31 04:05:42 localhost.localdomain NetworkManager[963]: <info> [1496196342.2298] device (wlp3s0): supplicant interface state: inactive -> scanning May 31 04:05:48 localhost.localdomain audit[1]: SERVICE_STOP pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=fprintd comm="systemd" exe="/usr/lib/syste May 31 04:05:57 localhost.localdomain kernel: NMI watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [systemd-logind:946] May 31 04:05:57 localhost.localdomain kernel: Modules linked in: fuse xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun nf_conntrack_netbios_ns nf_conntrack_broadcast xt_CT ip6t_rpfilte May 31 04:05:57 localhost.localdomain kernel: i2c_designware_core hp_wmi sparse_keymap intel_uncore iwlmvm intel_rapl_perf mac80211 snd_hda_intel snd_hda_codec snd_hda_core iwlwifi snd_hwd May 31 04:05:57 localhost.localdomain kernel: e1000e crc32c_intel mxm_wmi ttm ptp serio_raw pps_core drm nvme rtsx_pci nvme_core uas usb_storage i2c_hid wmi video May 31 04:05:57 localhost.localdomain kernel: irq event stamp: 153008 May 31 04:05:57 localhost.localdomain kernel: hardirqs last enabled at (153007): [<ffffffffb4981666>] _raw_spin_unlock_irqrestore+0x36/0x60 May 31 04:05:57 localhost.localdomain kernel: hardirqs last disabled at (153008): [<ffffffffb4979084>] __schedule+0xc4/0xb40 May 31 04:05:57 localhost.localdomain kernel: softirqs last enabled at (150688): [<ffffffffb4986e52>] __do_softirq+0x382/0x4ed May 31 04:05:57 localhost.localdomain kernel: softirqs last disabled at (150679): [<ffffffffb40b8ecf>] irq_exit+0x10f/0x120 May 31 04:05:57 localhost.localdomain kernel: CPU: 1 PID: 946 Comm: systemd-logind Not tainted 4.12.0-0.rc2.git3.1.fc27.x86_64 #1 May 31 04:05:57 localhost.localdomain kernel: Hardware name: HP Studio G3/80D4, BIOS N82 Ver. 01.16 04/14/2017 May 31 04:05:57 localhost.localdomain kernel: task: ffff9129c95d0000 task.stack: ffffb429044d4000 May 31 04:05:57 localhost.localdomain kernel: RIP: 0010:ioread32+0x19/0x40 May 31 04:05:57 localhost.localdomain kernel: RSP: 0018:ffffb429044d7ab0 EFLAGS: 00000292 ORIG_RAX: ffffffffffffff10 May 31 04:05:57 localhost.localdomain kernel: RAX: 00000000ffffffff RBX: ffff9129d2ef8800 RCX: 0000000000000018 May 31 04:05:57 localhost.localdomain kernel: RDX: 000009f731ac7af6 RSI: ffffb4290610a014 RDI: ffffb42906009400 May 31 04:05:57 localhost.localdomain kernel: RBP: ffffb429044d7ad0 R08: 0000000000000000 R09: 0000000000000001 May 31 04:05:57 localhost.localdomain kernel: R10: 0000000000000000 R11: 0000000000000000 R12: 00000000ffffffff May 31 04:05:57 localhost.localdomain kernel: R13: ffff9129d06ba600 R14: ffffffffffffffff R15: ffff9129d06b8000 May 31 04:05:57 localhost.localdomain kernel: FS: 00007f9d3c8fae80(0000) GS:ffff9129de000000(0000) knlGS:0000000000000000 May 31 04:05:57 localhost.localdomain kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 May 31 04:05:57 localhost.localdomain kernel: CR2: 00007fa501640550 CR3: 000000088977b000 CR4: 00000000003406e0 May 31 04:05:57 localhost.localdomain kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 May 31 04:05:57 localhost.localdomain kernel: DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 May 31 04:05:57 localhost.localdomain kernel: Call Trace: May 31 04:05:57 localhost.localdomain kernel: ? nv04_timer_read+0x3b/0x70 [nouveau] May 31 04:05:57 localhost.localdomain kernel: nvkm_timer_read+0xf/0x20 [nouveau] May 31 04:05:57 localhost.localdomain kernel: nvkm_pmu_reset+0x77/0x180 [nouveau] May 31 04:05:57 localhost.localdomain kernel: nvkm_pmu_preinit+0x12/0x20 [nouveau] May 31 04:05:57 localhost.localdomain kernel: nvkm_subdev_preinit+0x37/0x120 [nouveau] May 31 04:05:57 localhost.localdomain kernel: nvkm_device_init+0x60/0x270 [nouveau] May 31 04:05:57 localhost.localdomain kernel: nvkm_udevice_init+0x50/0x60 [nouveau] May 31 04:05:57 localhost.localdomain kernel: nvkm_object_init+0x3f/0x190 [nouveau] May 31 04:05:57 localhost.localdomain kernel: nvkm_object_init+0xa3/0x190 [nouveau] May 31 04:05:57 localhost.localdomain kernel: nvkm_object_init+0xa3/0x190 [nouveau] May 31 04:05:57 localhost.localdomain kernel: nvkm_client_resume+0xe/0x10 [nouveau] May 31 04:05:57 localhost.localdomain kernel: nvif_client_resume+0x17/0x20 [nouveau] May 31 04:05:57 localhost.localdomain kernel: nouveau_do_resume+0x40/0xe0 [nouveau] May 31 04:05:57 localhost.localdomain kernel: nouveau_pmops_runtime_resume+0x7d/0x130 [nouveau] May 31 04:05:57 localhost.localdomain kernel: ? pci_restore_standard_config+0x40/0x40 May 31 04:05:57 localhost.localdomain kernel: pci_pm_runtime_resume+0x7f/0xa0 May 31 04:05:57 localhost.localdomain kernel: __rpm_callback+0xb9/0x1e0 May 31 04:05:57 localhost.localdomain kernel: rpm_callback+0x24/0x80 May 31 04:05:57 localhost.localdomain kernel: ? pci_restore_standard_config+0x40/0x40 May 31 04:05:57 localhost.localdomain kernel: rpm_resume+0x512/0x870 May 31 04:05:57 localhost.localdomain kernel: __pm_runtime_resume+0x4e/0x90 May 31 04:05:57 localhost.localdomain kernel: nouveau_drm_ioctl+0x3d/0xc0 [nouveau] May 31 04:05:57 localhost.localdomain kernel: do_vfs_ioctl+0xa6/0x720 May 31 04:05:57 localhost.localdomain kernel: ? security_file_ioctl+0x43/0x60 May 31 04:05:57 localhost.localdomain kernel: SyS_ioctl+0x79/0x90 May 31 04:05:57 localhost.localdomain kernel: do_syscall_64+0x6c/0x1c0 May 31 04:05:57 localhost.localdomain kernel: entry_SYSCALL64_slow_path+0x25/0x25 May 31 04:05:57 localhost.localdomain kernel: RIP: 0033:0x7f9d3bc63707 May 31 04:05:57 localhost.localdomain kernel: RSP: 002b:00007ffdc4f4f818 EFLAGS: 00000246 ORIG_RAX: 0000000000000010 May 31 04:05:57 localhost.localdomain kernel: RAX: ffffffffffffffda RBX: 0000556118dda970 RCX: 00007f9d3bc63707 May 31 04:05:57 localhost.localdomain kernel: RDX: 0000000000000000 RSI: 000000000000641f RDI: 000000000000001a May 31 04:05:57 localhost.localdomain kernel: RBP: 00007ffdc4f4f850 R08: 0000000000000000 R09: 000000000000000f May 31 04:05:57 localhost.localdomain kernel: R10: 00007f9d3c718f20 R11: 0000000000000246 R12: 00007ffdc4f4f848 May 31 04:05:57 localhost.localdomain kernel: R13: 0000000000000002 R14: 0000000000000005 R15: 0000556118ddc7c0 May 31 04:05:57 localhost.localdomain kernel: Code: 5e ff ff ff b8 ff ff 00 00 5d c3 0f 1f 80 00 00 00 00 48 81 ff ff ff 03 00 77 0e 48 81 ff 00 00 01 00 76 08 0f b7 d7 ed c3 8b 07 <c3> 55 May 31 04:05:58 localhost.localdomain abrt-dump-journal-oops[934]: abrt-dump-journal-oops: Found oopses: 1 May 31 04:05:58 localhost.localdomain abrt-dump-journal-oops[934]: abrt-dump-journal-oops: Creating problem directories May 31 04:05:58 localhost.localdomain abrt-server[1653]: Can't find a meaningful backtrace for hashing in '.' May 31 04:05:58 localhost.localdomain abrt-server[1653]: Option 'DropNotReportableOopses' is not configured May 31 04:05:58 localhost.localdomain abrt-server[1653]: Preserving oops '.' because DropNotReportableOopses is 'no' May 31 04:05:58 localhost.localdomain dbus-daemon[898]: [system] Activating service name='org.freedesktop.problems' requested by ':1.57' (uid=0 pid=1670 comm="/usr/bin/python3 /usr/bin/abrt May 31 04:05:58 localhost.localdomain dbus-daemon[898]: [system] Successfully activated service 'org.freedesktop.problems' May 31 04:05:58 localhost.localdomain abrt-notification[1676]: System encountered a non-fatal error in ??() May 31 04:05:59 localhost.localdomain abrt-dump-journal-oops[934]: Reported 1 kernel oopses to Abrt May 31 04:06:19 localhost.localdomain realmd[1550]: quitting realmd service after timeout May 31 04:06:19 localhost.localdomain realmd[1550]: stopping service May 31 04:06:19 localhost.localdomain audit[1]: SERVICE_STOP pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=realmd comm="systemd" exe="/usr/lib/system May 31 04:06:25 localhost.localdomain kernel: NMI watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [systemd-logind:946] May 31 04:06:25 localhost.localdomain kernel: Modules linked in: fuse xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun nf_conntrack_netbios_ns nf_conntrack_broadcast xt_CT ip6t_rpfilte May 31 04:06:25 localhost.localdomain kernel: i2c_designware_core hp_wmi sparse_keymap intel_uncore iwlmvm intel_rapl_perf mac80211 snd_hda_intel snd_hda_codec snd_hda_core iwlwifi snd_hwd May 31 04:06:25 localhost.localdomain kernel: e1000e crc32c_intel mxm_wmi ttm ptp serio_raw pps_core drm nvme rtsx_pci nvme_core uas usb_storage i2c_hid wmi video May 31 04:06:25 localhost.localdomain kernel: irq event stamp: 153008 May 31 04:06:25 localhost.localdomain kernel: hardirqs last enabled at (153007): [<ffffffffb4981666>] _raw_spin_unlock_irqrestore+0x36/0x60 May 31 04:06:25 localhost.localdomain kernel: hardirqs last disabled at (153008): [<ffffffffb4979084>] __schedule+0xc4/0xb40 May 31 04:06:25 localhost.localdomain kernel: softirqs last enabled at (150688): [<ffffffffb4986e52>] __do_softirq+0x382/0x4ed May 31 04:06:25 localhost.localdomain kernel: softirqs last disabled at (150679): [<ffffffffb40b8ecf>] irq_exit+0x10f/0x120 May 31 04:06:25 localhost.localdomain kernel: CPU: 1 PID: 946 Comm: systemd-logind Tainted: G L 4.12.0-0.rc2.git3.1.fc27.x86_64 #1 May 31 04:06:25 localhost.localdomain kernel: Hardware name: HP Studio G3/80D4, BIOS N82 Ver. 01.16 04/14/2017 May 31 04:06:25 localhost.localdomain kernel: task: ffff9129c95d0000 task.stack: ffffb429044d4000 May 31 04:06:25 localhost.localdomain kernel: RIP: 0010:ioread32+0x19/0x40 May 31 04:06:25 localhost.localdomain kernel: RSP: 0018:ffffb429044d7ab0 EFLAGS: 00000296 ORIG_RAX: ffffffffffffff10 May 31 04:06:25 localhost.localdomain kernel: RAX: 00000000ffffffff RBX: ffff9129d2ef8800 RCX: 0000000000000018 May 31 04:06:25 localhost.localdomain kernel: RDX: 000009f731ac7af6 RSI: ffffb4290610a014 RDI: ffffb42906009410 May 31 04:06:25 localhost.localdomain kernel: RBP: ffffb429044d7ad0 R08: 0000000000000000 R09: 0000000000000001 May 31 04:06:25 localhost.localdomain kernel: R10: 0000000000000000 R11: 0000000000000000 R12: 00000000ffffffff May 31 04:06:25 localhost.localdomain kernel: R13: 00000000ffffffff R14: ffffffffffffffff R15: ffff9129d06b8000 May 31 04:06:25 localhost.localdomain kernel: FS: 00007f9d3c8fae80(0000) GS:ffff9129de000000(0000) knlGS:0000000000000000 May 31 04:06:25 localhost.localdomain kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 May 31 04:06:25 localhost.localdomain kernel: CR2: 00007fa501640550 CR3: 000000088977b000 CR4: 00000000003406e0 May 31 04:06:25 localhost.localdomain kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 May 31 04:06:25 localhost.localdomain kernel: DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 May 31 04:06:25 localhost.localdomain kernel: Call Trace: May 31 04:06:25 localhost.localdomain kernel: ? nv04_timer_read+0x51/0x70 [nouveau] May 31 04:06:25 localhost.localdomain kernel: nvkm_timer_read+0xf/0x20 [nouveau] May 31 04:06:25 localhost.localdomain kernel: nvkm_pmu_reset+0x77/0x180 [nouveau] May 31 04:06:25 localhost.localdomain kernel: nvkm_pmu_preinit+0x12/0x20 [nouveau] May 31 04:06:25 localhost.localdomain kernel: nvkm_subdev_preinit+0x37/0x120 [nouveau] May 31 04:06:25 localhost.localdomain kernel: nvkm_device_init+0x60/0x270 [nouveau] May 31 04:06:25 localhost.localdomain kernel: nvkm_udevice_init+0x50/0x60 [nouveau] May 31 04:06:25 localhost.localdomain kernel: nvkm_object_init+0x3f/0x190 [nouveau] May 31 04:06:25 localhost.localdomain kernel: nvkm_object_init+0xa3/0x190 [nouveau] May 31 04:06:25 localhost.localdomain kernel: nvkm_object_init+0xa3/0x190 [nouveau] May 31 04:06:25 localhost.localdomain kernel: nvkm_client_resume+0xe/0x10 [nouveau] May 31 04:06:25 localhost.localdomain kernel: nvif_client_resume+0x17/0x20 [nouveau] May 31 04:06:25 localhost.localdomain kernel: nouveau_do_resume+0x40/0xe0 [nouveau] May 31 04:06:25 localhost.localdomain kernel: nouveau_pmops_runtime_resume+0x7d/0x130 [nouveau] May 31 04:06:25 localhost.localdomain kernel: ? pci_restore_standard_config+0x40/0x40 May 31 04:06:25 localhost.localdomain kernel: pci_pm_runtime_resume+0x7f/0xa0 May 31 04:06:25 localhost.localdomain kernel: __rpm_callback+0xb9/0x1e0 May 31 04:06:25 localhost.localdomain kernel: rpm_callback+0x24/0x80 May 31 04:06:25 localhost.localdomain kernel: ? pci_restore_standard_config+0x40/0x40 May 31 04:06:25 localhost.localdomain kernel: rpm_resume+0x512/0x870 May 31 04:06:25 localhost.localdomain kernel: __pm_runtime_resume+0x4e/0x90 May 31 04:06:25 localhost.localdomain kernel: nouveau_drm_ioctl+0x3d/0xc0 [nouveau] May 31 04:06:25 localhost.localdomain kernel: do_vfs_ioctl+0xa6/0x720 May 31 04:06:25 localhost.localdomain kernel: ? security_file_ioctl+0x43/0x60 May 31 04:06:25 localhost.localdomain kernel: SyS_ioctl+0x79/0x90 May 31 04:06:25 localhost.localdomain kernel: do_syscall_64+0x6c/0x1c0 May 31 04:06:25 localhost.localdomain kernel: entry_SYSCALL64_slow_path+0x25/0x25 May 31 04:06:25 localhost.localdomain kernel: RIP: 0033:0x7f9d3bc63707 May 31 04:06:25 localhost.localdomain kernel: RSP: 002b:00007ffdc4f4f818 EFLAGS: 00000246 ORIG_RAX: 0000000000000010 May 31 04:06:25 localhost.localdomain kernel: RAX: ffffffffffffffda RBX: 0000556118dda970 RCX: 00007f9d3bc63707 May 31 04:06:25 localhost.localdomain kernel: RDX: 0000000000000000 RSI: 000000000000641f RDI: 000000000000001a May 31 04:06:25 localhost.localdomain kernel: RBP: 00007ffdc4f4f850 R08: 0000000000000000 R09: 000000000000000f May 31 04:06:25 localhost.localdomain kernel: R10: 00007f9d3c718f20 R11: 0000000000000246 R12: 00007ffdc4f4f848 May 31 04:06:25 localhost.localdomain kernel: R13: 0000000000000002 R14: 0000000000000005 R15: 0000556118ddc7c0 May 31 04:06:25 localhost.localdomain kernel: Code: 5e ff ff ff b8 ff ff 00 00 5d c3 0f 1f 80 00 00 00 00 48 81 ff ff ff 03 00 77 0e 48 81 ff 00 00 01 00 76 08 0f b7 d7 ed c3 8b 07 <c3> 55 May 31 04:06:26 localhost.localdomain abrt-dump-journal-oops[934]: abrt-dump-journal-oops: Found oopses: 1 May 31 04:06:26 localhost.localdomain abrt-dump-journal-oops[934]: abrt-dump-journal-oops: Creating problem directories May 31 04:06:26 localhost.localdomain abrt-server[1679]: Can't find a meaningful backtrace for hashing in '.' May 31 04:06:26 localhost.localdomain abrt-server[1679]: Option 'DropNotReportableOopses' is not configured May 31 04:06:26 localhost.localdomain abrt-server[1679]: Preserving oops '.' because DropNotReportableOopses is 'no' May 31 04:06:26 localhost.localdomain abrt-notification[1697]: System encountered a non-fatal error in ??() May 31 04:06:27 localhost.localdomain abrt-dump-journal-oops[934]: Reported 1 kernel oopses to Abrt May 31 04:06:35 localhost.localdomain kernel: INFO: rcu_sched self-detected stall on CPU May 31 04:06:35 localhost.localdomain kernel: 1-...: (64999 ticks this GP) idle=032/140000000000001/0 softirq=1686/1686 fqs=16215 May 31 04:06:35 localhost.localdomain kernel: (t=65000 jiffies g=1505 c=1504 q=0) May 31 04:06:35 localhost.localdomain kernel: NMI backtrace for cpu 1 May 31 04:06:35 localhost.localdomain kernel: CPU: 1 PID: 946 Comm: systemd-logind Tainted: G L 4.12.0-0.rc2.git3.1.fc27.x86_64 #1 May 31 04:06:35 localhost.localdomain kernel: Hardware name: HP Studio G3/80D4, BIOS N82 Ver. 01.16 04/14/2017 May 31 04:06:35 localhost.localdomain kernel: Call Trace: May 31 04:06:35 localhost.localdomain kernel: <IRQ> May 31 04:06:35 localhost.localdomain kernel: dump_stack+0x8e/0xcd May 31 04:06:35 localhost.localdomain kernel: nmi_cpu_backtrace+0x94/0xa0 May 31 04:06:35 localhost.localdomain kernel: ? irq_force_complete_move+0x130/0x130 May 31 04:06:35 localhost.localdomain kernel: nmi_trigger_cpumask_backtrace+0xea/0x120 May 31 04:06:35 localhost.localdomain kernel: arch_trigger_cpumask_backtrace+0x19/0x20 May 31 04:06:35 localhost.localdomain kernel: rcu_dump_cpu_stacks+0xb1/0xeb May 31 04:06:35 localhost.localdomain kernel: rcu_check_callbacks+0x83a/0xa60 May 31 04:06:35 localhost.localdomain kernel: ? trace_hardirqs_off+0xd/0x10 May 31 04:06:35 localhost.localdomain kernel: ? tick_sched_do_timer+0x60/0x60 May 31 04:06:35 localhost.localdomain kernel: update_process_times+0x2f/0x60 May 31 04:06:35 localhost.localdomain kernel: tick_sched_handle.isra.21+0x23/0x60 May 31 04:06:35 localhost.localdomain kernel: ? tick_sched_do_timer+0x44/0x60 May 31 04:06:35 localhost.localdomain kernel: tick_sched_timer+0x3d/0x70 May 31 04:06:35 localhost.localdomain kernel: __hrtimer_run_queues+0xf0/0x4e0 May 31 04:06:35 localhost.localdomain kernel: hrtimer_interrupt+0xb6/0x210 May 31 04:06:35 localhost.localdomain kernel: local_apic_timer_interrupt+0x38/0x60 May 31 04:06:35 localhost.localdomain kernel: smp_apic_timer_interrupt+0x38/0x50 May 31 04:06:35 localhost.localdomain kernel: apic_timer_interrupt+0x9d/0xb0 May 31 04:06:35 localhost.localdomain kernel: RIP: 0010:ioread32+0x19/0x40 May 31 04:06:35 localhost.localdomain kernel: RSP: 0018:ffffb429044d7ab0 EFLAGS: 00000296 ORIG_RAX: ffffffffffffff10 May 31 04:06:35 localhost.localdomain kernel: RAX: 00000000ffffffff RBX: ffff9129d2ef8800 RCX: 0000000000000018 May 31 04:06:35 localhost.localdomain kernel: RDX: 000009f731ac7af6 RSI: ffffb4290610a014 RDI: ffffb42906009410 May 31 04:06:35 localhost.localdomain kernel: RBP: ffffb429044d7ad0 R08: 0000000000000000 R09: 0000000000000001 May 31 04:06:35 localhost.localdomain kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffff9129d2ef8800 May 31 04:06:35 localhost.localdomain kernel: R13: ffff9129d06ba600 R14: ffffffffffffffff R15: ffff9129d06b8000 May 31 04:06:35 localhost.localdomain kernel: </IRQ> May 31 04:06:35 localhost.localdomain kernel: ? nv04_timer_read+0x25/0x70 [nouveau] May 31 04:06:35 localhost.localdomain kernel: nvkm_timer_read+0xf/0x20 [nouveau] May 31 04:06:35 localhost.localdomain kernel: nvkm_pmu_reset+0x77/0x180 [nouveau] May 31 04:06:35 localhost.localdomain kernel: nvkm_pmu_preinit+0x12/0x20 [nouveau] May 31 04:06:35 localhost.localdomain kernel: nvkm_subdev_preinit+0x37/0x120 [nouveau] May 31 04:06:35 localhost.localdomain kernel: nvkm_device_init+0x60/0x270 [nouveau] May 31 04:06:35 localhost.localdomain kernel: nvkm_udevice_init+0x50/0x60 [nouveau] May 31 04:06:35 localhost.localdomain kernel: nvkm_object_init+0x3f/0x190 [nouveau] May 31 04:06:35 localhost.localdomain kernel: nvkm_object_init+0xa3/0x190 [nouveau] May 31 04:06:35 localhost.localdomain kernel: nvkm_object_init+0xa3/0x190 [nouveau] May 31 04:06:35 localhost.localdomain kernel: nvkm_client_resume+0xe/0x10 [nouveau] May 31 04:06:35 localhost.localdomain kernel: nvif_client_resume+0x17/0x20 [nouveau] May 31 04:06:35 localhost.localdomain kernel: nouveau_do_resume+0x40/0xe0 [nouveau] May 31 04:06:35 localhost.localdomain kernel: nouveau_pmops_runtime_resume+0x7d/0x130 [nouveau] May 31 04:06:35 localhost.localdomain kernel: ? pci_restore_standard_config+0x40/0x40 May 31 04:06:35 localhost.localdomain kernel: pci_pm_runtime_resume+0x7f/0xa0 May 31 04:06:35 localhost.localdomain kernel: __rpm_callback+0xb9/0x1e0 May 31 04:06:35 localhost.localdomain kernel: rpm_callback+0x24/0x80 May 31 04:06:35 localhost.localdomain kernel: ? pci_restore_standard_config+0x40/0x40 May 31 04:06:35 localhost.localdomain kernel: rpm_resume+0x512/0x870 May 31 04:06:35 localhost.localdomain kernel: __pm_runtime_resume+0x4e/0x90 May 31 04:06:35 localhost.localdomain kernel: nouveau_drm_ioctl+0x3d/0xc0 [nouveau] May 31 04:06:35 localhost.localdomain kernel: do_vfs_ioctl+0xa6/0x720 May 31 04:06:35 localhost.localdomain kernel: ? security_file_ioctl+0x43/0x60 May 31 04:06:35 localhost.localdomain kernel: SyS_ioctl+0x79/0x90 May 31 04:06:35 localhost.localdomain kernel: do_syscall_64+0x6c/0x1c0 May 31 04:06:35 localhost.localdomain kernel: entry_SYSCALL64_slow_path+0x25/0x25 May 31 04:06:35 localhost.localdomain kernel: RIP: 0033:0x7f9d3bc63707 May 31 04:06:35 localhost.localdomain kernel: RSP: 002b:00007ffdc4f4f818 EFLAGS: 00000246 ORIG_RAX: 0000000000000010 May 31 04:06:35 localhost.localdomain kernel: RAX: ffffffffffffffda RBX: 0000556118dda970 RCX: 00007f9d3bc63707 May 31 04:06:35 localhost.localdomain kernel: RDX: 0000000000000000 RSI: 000000000000641f RDI: 000000000000001a May 31 04:06:35 localhost.localdomain kernel: RBP: 00007ffdc4f4f850 R08: 0000000000000000 R09: 000000000000000f May 31 04:06:35 localhost.localdomain kernel: R10: 00007f9d3c718f20 R11: 0000000000000246 R12: 00007ffdc4f4f848 May 31 04:06:35 localhost.localdomain kernel: R13: 0000000000000002 R14: 0000000000000005 R15: 0000556118ddc7c0 May 31 04:06:44 localhost.localdomain kernel: INFO: rcu_sched detected expedited stalls on CPUs/tasks: { 1-... } 67636 jiffies s: 685 root: 0x2/. May 31 04:06:44 localhost.localdomain kernel: blocking rcu_node structures: May 31 04:06:44 localhost.localdomain kernel: Task dump for CPU 1: May 31 04:06:44 localhost.localdomain kernel: systemd-logind R running task 11848 946 1 0x00000108 May 31 04:06:44 localhost.localdomain kernel: Call Trace: May 31 04:06:44 localhost.localdomain kernel: ? lock_acquire+0xa3/0x1f0 May 31 04:06:44 localhost.localdomain kernel: ? trace_hardirqs_on_thunk+0x1a/0x1c May 31 04:06:44 localhost.localdomain kernel: ? retint_kernel+0x10/0x10 May 31 04:06:44 localhost.localdomain kernel: ? ioread32+0x19/0x40 May 31 04:06:44 localhost.localdomain kernel: ? nv04_timer_read+0x51/0x70 [nouveau] May 31 04:06:44 localhost.localdomain kernel: ? nvkm_timer_read+0xf/0x20 [nouveau] May 31 04:06:44 localhost.localdomain kernel: ? nvkm_pmu_reset+0x77/0x180 [nouveau] May 31 04:06:44 localhost.localdomain kernel: ? nvkm_pmu_preinit+0x12/0x20 [nouveau] May 31 04:06:44 localhost.localdomain kernel: ? nvkm_subdev_preinit+0x37/0x120 [nouveau] May 31 04:06:44 localhost.localdomain kernel: ? nvkm_device_init+0x60/0x270 [nouveau] May 31 04:06:44 localhost.localdomain kernel: ? nvkm_udevice_init+0x50/0x60 [nouveau] May 31 04:06:44 localhost.localdomain kernel: ? nvkm_object_init+0x3f/0x190 [nouveau] May 31 04:06:44 localhost.localdomain kernel: ? nvkm_object_init+0xa3/0x190 [nouveau] May 31 04:06:44 localhost.localdomain kernel: ? nvkm_object_init+0xa3/0x190 [nouveau] May 31 04:06:44 localhost.localdomain kernel: ? nvkm_client_resume+0xe/0x10 [nouveau] May 31 04:06:44 localhost.localdomain kernel: ? nvif_client_resume+0x17/0x20 [nouveau] May 31 04:06:44 localhost.localdomain kernel: ? nouveau_do_resume+0x40/0xe0 [nouveau] May 31 04:06:44 localhost.localdomain kernel: ? nouveau_pmops_runtime_resume+0x7d/0x130 [nouveau] May 31 04:06:44 localhost.localdomain kernel: ? pci_restore_standard_config+0x40/0x40 May 31 04:06:44 localhost.localdomain kernel: ? pci_pm_runtime_resume+0x7f/0xa0 May 31 04:06:44 localhost.localdomain kernel: ? __rpm_callback+0xb9/0x1e0 May 31 04:06:44 localhost.localdomain kernel: ? rpm_callback+0x24/0x80 May 31 04:06:44 localhost.localdomain kernel: ? pci_restore_standard_config+0x40/0x40 May 31 04:06:44 localhost.localdomain kernel: ? rpm_resume+0x512/0x870 May 31 04:06:44 localhost.localdomain kernel: ? __pm_runtime_resume+0x4e/0x90 May 31 04:06:44 localhost.localdomain kernel: ? nouveau_drm_ioctl+0x3d/0xc0 [nouveau] May 31 04:06:44 localhost.localdomain kernel: ? do_vfs_ioctl+0xa6/0x720 May 31 04:06:44 localhost.localdomain kernel: ? security_file_ioctl+0x43/0x60 May 31 04:06:44 localhost.localdomain kernel: ? SyS_ioctl+0x79/0x90 May 31 04:06:44 localhost.localdomain kernel: ? do_syscall_64+0x6c/0x1c0 May 31 04:06:44 localhost.localdomain kernel: ? entry_SYSCALL64_slow_path+0x25/0x25 May 31 04:07:01 localhost.localdomain kernel: NMI watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [systemd-logind:946] May 31 04:07:01 localhost.localdomain kernel: Modules linked in: fuse xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun nf_conntrack_netbios_ns nf_conntrack_broadcast xt_CT ip6t_rpfilte May 31 04:07:01 localhost.localdomain kernel: i2c_designware_core hp_wmi sparse_keymap intel_uncore iwlmvm intel_rapl_perf mac80211 snd_hda_intel snd_hda_codec snd_hda_core iwlwifi snd_hwd May 31 04:07:01 localhost.localdomain kernel: e1000e crc32c_intel mxm_wmi ttm ptp serio_raw pps_core drm nvme rtsx_pci nvme_core uas usb_storage i2c_hid wmi video May 31 04:07:01 localhost.localdomain kernel: irq event stamp: 153008 May 31 04:07:01 localhost.localdomain kernel: hardirqs last enabled at (153007): [<ffffffffb4981666>] _raw_spin_unlock_irqrestore+0x36/0x60 May 31 04:07:01 localhost.localdomain kernel: hardirqs last disabled at (153008): [<ffffffffb4979084>] __schedule+0xc4/0xb40 May 31 04:07:01 localhost.localdomain kernel: softirqs last enabled at (150688): [<ffffffffb4986e52>] __do_softirq+0x382/0x4ed May 31 04:07:01 localhost.localdomain kernel: softirqs last disabled at (150679): [<ffffffffb40b8ecf>] irq_exit+0x10f/0x120 May 31 04:07:01 localhost.localdomain kernel: CPU: 1 PID: 946 Comm: systemd-logind Tainted: G L 4.12.0-0.rc2.git3.1.fc27.x86_64 #1
I'm not seeing how this could be considered to be remotely the same issue. Your logs don't even indicate that you have xfs loaded. If you're having an issue that's not related to XFS corruption on that specific model of SSD, I suggest that you open a separate bug.
I will do that. However I have been chasing this issue for a few months now and initially thought it could be related to my problems. I agree that the logs don't indicate filesystem issues so I will file another bug for my specific problems. Before that I will try an install on an XFS partition to see if I do see the same corruption. The SSD is an Intel 600P 512 GB, I have also notified Intel of my issue and am following up on it with them.
(In reply to Sunil Amitkumar Janki from comment #45) > > The SSD is an Intel 600P 512 GB, I have also notified Intel of my issue and > am following up on it with them. I think you'd be better off taking your new issue directly to the nouveau mailing list, right? This doesn't appear at all related the Intel 600p or NVMe.
I tested it again after doing the latest firmware upgrade: XFS+LVM: Anaconda crashed and the journal contained a few errors related to XFS async work or whatever XFS w/o LVM: Installation worked, no data loss by now. Kind of fixed the problem for me (so far, let's see) but the first trial outcome was not exactly inspiring confidence in this SSD.
TL;DR Just for the record, Intel SSD 600p seems to corrupt also EFI volumes (no LVM involved here). I am experiencing data loss and corrupt files on System EFI volume (that's not on LVM at all), my Fedora 25 installation went fine onto Intel 600p 256GB NVMe perhaps because I used no-LVM and ext4 for the boot/root volumes, but after applying update of grub2 files on EFI volume were all corrupted, fsck signals incorrect FAT entries and system is not bootable. This is tracked under different ticket: https://bugzilla.redhat.com/show_bug.cgi?id=1413191 but I believe Intel needs to fix both with an firmware update according to the thread here. Good luck to us!
(In reply to Lukas Zapletal from comment #48) > TL;DR Just for the record, Intel SSD 600p seems to corrupt also EFI volumes > (no LVM involved here). > > I am experiencing data loss and corrupt files on System EFI volume (that's > not on LVM at all), my Fedora 25 installation went fine onto Intel 600p > 256GB NVMe perhaps because I used no-LVM and ext4 for the boot/root volumes, > but after applying update of grub2 files on EFI volume were all corrupted, > fsck signals incorrect FAT entries and system is not bootable. > > This is tracked under different ticket: > https://bugzilla.redhat.com/show_bug.cgi?id=1413191 but I believe Intel > needs to fix both with an firmware update according to the thread here. Good > luck to us! Lukas: is this after applying the latest firmware?
I haven't updated, see above - the fixed firmware haven't been released yet.
(In reply to Lukas Zapletal from comment #50) > I haven't updated, see above - the fixed firmware haven't been released yet. Intel 600p old firmware (109C) was the offender. The new firmware (121C) has been released publicly and is downloadable through https://downloadcenter.intel.com/download/26491/Intel-SSD-Firmware-Update-Tool?product=94922 or https://downloadcenter.intel.com/download/26776/Intel-Solid-State-Drive-Toolbox?product=94922
Correction: links were wrong. https://downloadcenter.intel.com/download/26491/Intel-SSD-Firmware-Update-Tool?product=94922 or https://downloadcenter.intel.com/download/26837/Intel-Solid-State-Drive-Toolbox?product=94922
Brad, I see you are in charge working with Intel on the Series 6 SSD NVMe XFS/LVM data corruption in bug 1428042. Can you please confirm if firmware 121C really solves the issue? Because I have an impression it may have not. Thank you.
I'm here to report back some positive progress. I applied 121C firmware to my 600p NVMe SSD. BTW, the bootable ISO image does not work with UEFI machines, at least not with mine. I ran the firmware update from Windows 10 instead. The flashing went extremely fast; less than a second. After the firmware update, I performed a fresh install of Fedora 25 with LVM+XFS as I originally did. I have since been able to boot up, reboot, update, reboot several times without the data corruption issue I had before. This is looking pretty positive after 6 months of waiting!! Thanks Intel; but I really wished you guys would have fixed it sooner!
Installed firmware 121C from the issdfut_2.2.2 package. Bad news. Running CentOS 7.3 installer, install died with XFS errors, best I can tell, the SSD disappeared, became unwritable, etc. (the installer image does not have enough tools available for more debugging, i.e. the "nvme" command is missing). K.O.
(In reply to Konstantin Olchanski from comment #55) > Installed firmware 121C from the issdfut_2.2.2 package. Bad news. > > Running CentOS 7.3 installer, install died with XFS errors, best I can tell, > the SSD disappeared, became unwritable, etc. (the installer image does not > have enough tools available for more debugging, i.e. the "nvme" command is > missing). > > K.O. There is an "updates image" mechanism to add and replace stuff in the installation environment. It is mainly used for development, debugging and hotfixing, but it can also be used to add content of arbitrary RPMs to the installation environment. I guess it could be used in this case to add additional debugging tools not included in the installation image by default. Just a few disclaimers though: - the makeupdates script does not depsolve, any dependencies have to be manually included as well - the RPMs are just unpacked, so RPM scriptlets have no effect More information about the updates image mechanism: https://fedoraproject.org/wiki/Anaconda/Updates#create-images (note the -a option of the makeupdates script, which can be used to add RPMs to the updates image)
(In reply to Konstantin Olchanski from comment #55) > Installed firmware 121C from the issdfut_2.2.2 package. Bad news. > > Running CentOS 7.3 installer, install died with XFS errors, best I can tell, > the SSD disappeared, became unwritable, etc. (the installer image does not > have enough tools available for more debugging, i.e. the "nvme" command is > missing). > > K.O. I don't know for sure, but your case might involve another factor. I've been running the 121C firmware with Fedora 25 installation for 2-1/2 months now for my daily workstation laptop (ThinkPad P50) and have had no problems.
(In reply to Konstantin Olchanski from comment #55) > Installed firmware 121C from the issdfut_2.2.2 package. Bad news. > > Running CentOS 7.3 installer, install died with XFS errors... > Running Ubuntu 17.04 installer, completed successfully. Default install, ext4 rootfs, EFI boot. Oh, well. K.O.
(In reply to Bond Masuda from comment #57) > (In reply to Konstantin Olchanski from comment #55) > > Installed firmware 121C from the issdfut_2.2.2 package. Bad news. > > Running CentOS 7.3 installer, install died with XFS errors ... > > I don't know for sure, but your case might involve another factor. I've been > running the 121C firmware with Fedora 25 installation for 2-1/2 months now > for my daily workstation laptop (ThinkPad P50) and have had no problems. Our information is compatible. One machine with this SSD is running just fine, el7 OS installed by cloning instead of running the installer (installer fails, cloning via mkfs + rsync + magic voodoo to make it bootable is no problem). I have 5 of these SSDs purchased by mistake (was supposed to be Kingston SATA SSDs). With latest firmware, I see crash of CentOS-7.3 installer on two machines out of two machines tried. Ubuntu-17.04 is success 2 out of 2 tried. Conclusion: do not buy Intel other than CPU and NIC. K.O.
considering this is now fixed by Intel firmware update, and it was not a Linux/Fedora software issue, i'm going to close this ticket.
The needinfo request[s] on this closed bug have been removed as they have been unresolved for 1000 days