RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1185734 - kernel panic rebooting server with iSCSI partition mounted
Summary: kernel panic rebooting server with iSCSI partition mounted
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: iscsi-initiator-utils
Version: 7.1
Hardware: Unspecified
OS: Unspecified
high
medium
Target Milestone: rc
: 7.6
Assignee: Chris Leech
QA Contact: Martin Hoyer
URL:
Whiteboard:
Depends On:
Blocks: 1295577 1313485 1507952 1507957 1609081
TreeView+ depends on / blocked
 
Reported: 2015-01-26 07:58 UTC by Bruno Goncalves
Modified: 2023-03-08 07:27 UTC (History)
10 users (show)

Fixed In Version: iscsi-initiator-utils-6.2.0.874-10.el7
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2018-10-30 11:40:49 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)
RHEL-7.2 systemctl show mnt-os_reboot.mount (2.26 KB, text/plain)
2016-02-11 14:17 UTC, Martin Hoyer 		no flags Details
View All


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHEA-2018:3281 0 None None None 2018-10-30 11:41:12 UTC

Description Bruno Goncalves 2015-01-26 07:58:24 UTC 
Description of problem:
Kernel panic when rebooting a server with mounted iSCSI partition and performing IO on it.

Version-Release number of selected component (if applicable):
systemd-208-20.el7
kernel 3.10.0-224.el7
iscsi-initiator-utils-6.2.0.873-28.el7


How reproducible:
70%

Steps to Reproduce:
1.Mount iSCSI device
echo "UUID="0fc1f3be-9779-4cf5-a2c3-c434e1e6a1ee"	/root/os_reboot	xfs	_netdev	0 2" >> /etc/fstab

mount -a

2. Start IO on it

dd if=/dev/urandom of=/root/os_reboot/test.img &

3. reboot server

Actual results:
[    
  OK     
] Started Show Plymouth Reboot Screen.  
[    
  OK     
] Started Restore /run/initramfs.  
2015-01-25 20:16:57,550 rhts_task controller_disconnected: ERROR Connection to controller was lost! reason=[Failure instance: Traceback (failure with no frames): <class 'twisted.internet.error.ConnectionDone'>: Connection was closed cleanly.  
]  
[    
  OK     
] Stopped Dynamic System Tuning Daemon.  
[    
  OK     
] Stopped D-Bus System Message Bus.  
[    
  OK     
] Stopped Login Service.  
2015-01-25 20:16:59,031 rhts_task on_exit: INFO quitting...  
[    
  OK     
] Stopped Open-iSCSI.  
         Stopping Device-Mapper Multipath Device Controller...  
         Stopping iSCSI UserSpace I/O driver...  
[    
  OK     
] Stopped iSCSI UserSpace I/O driver.  
[    
  OK     
] Stopped Postfix Mail Transport Agent.  
[    
  OK     
] Stopped target Network.  
         Stopping Network [   87.600809]  connection1:0: detected conn error (1011) 
Manager...  
[   88.625007] bnx2i: (ens1f0) chip cleanup for 1 active connections 
[    
  OK     
] Stopped Device-Mapper Multipath Device Controller.  
[   93.790952]  connection2:0: detected conn error (1011) 
[   
    
*        
] A stop job is running for Network Manager[   94.816263] bnx2i: (ens1f1) chip cleanup for 1 active connections 
   
[      
*   
    
*       
] A stop job is running for Network Manager   
[    
*      
*   
    
*      
] A stop job is running for Network Manager   
[     
*      
*   
    
*     
] A stop job is running for Network Manager   
[      
*      
*   
    
*    
] A stop job is running for Network Manager   
[       
*      
*   
    
*   
] A stop job is running for Network Manager   
[        
*      
*   
] A stop job is running for Network Manager   
[         
*   
] A stop job is running for Network Manager   
[        
*      
*   
] A stop job is running for Network Manager   
[       
*      
*   
    
*   
] A stop job is running for Network Manager   
[    
  OK     
] Stopped Network Manager.  
[    
  OK     
] Stopped target Basic System.  
[    
  OK     
] Stopped target Slices.  
[    
  OK     
] Removed slice User and Session Slice.  
[    
  OK     
] Stopped target Paths.  
[    
  OK     
] Stopped target Timers.  
[    
  OK     
] Stopped target Sockets.  
[    
  OK     
] Closed RPCbind Server Activation Socket.  
[    
  OK     
] Closed Open-iSCSI iscsid Socket.  
[    
  OK     
] Closed Open-iSCSI iscsiuio Socket.  
[    
  OK     
] Closed D-Bus System Message Bus Socket.  
[    
  OK     
] Stopped target System Initialization.  
         Stopping Setup Virtual Console...  
[    
  OK     
] Stopped Setup Virtual Console.  
[    
  OK     
] Stopped target Encrypted Volumes.  
         Stopping Load/Save Random Seed...  
         Stopping Apply Kernel Variables...  
[    
  OK     
] Stopped Apply Kernel Variables.  
         Stopping Update UTMP about System Reboot/Shutdown...  
[    
  OK     
] Stopped target Swap.  
         Deactivating swap /dev/dm-1...  
         Deactivating swap /dev/dm-1...  
         Deactivating swap /dev/dm-1...  
         Deactivating swap /dev/dm-1...  
         Deactivating swap /dev/dm-1...  
         Deactivating swap /dev/dm-1...  
[    
  OK     
] Stopped Load/Save Random Seed.  
[    
  OK     
] Stopped Update UTMP about System Reboot/Shutdown.  
         Stopping Security Auditing Service...  
[  102.929054] type=1305 audit(1422213439.028:94): audit_pid=0 old=856 auid=4294967295 ses=4294967295 subj=system_u:system_r:auditd_t:s0 res=1 
[    
  OK     
] Deactivated swap /dev/rhel_storageqe-85/swap.  
[    
  OK     
] Deactivated swap /dev/disk/by-uuid/8d8aeb8e-13c5-4f5...4-a955145dcb79.  
[    
  OK     
] Deactivated swap /dev/disk/by-id/dm-uuid-LVM-we55Obh...f8vJMUX1MtC2SU.  
[    
  OK     
] Deactivated swap /dev/disk/by-id/dm-name-rhel_storageqe--85-swap.  
[    
  OK     
] [  102.991273] type=1131 audit(1422213439.090:95): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg=' comm="auditd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' 
Deactivated swap /dev/mapper/rhel_storageqe--85-swap.  
[    
  OK     
] Deactivated swap /dev/dm-1.  
[    
  OK     
] Stopped Security Auditing Service.  
[  103.079204] type=1131 audit(1422213439.178:96): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg=' comm="systemd-tmpfiles-setup" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' 
         Stopping Create Volatil[  103.172214] type=1131 audit(1422213439.271:97): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg=' comm="rhel-import-state" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' 
e Files and Directories...  
[    
  OK     
] Stopped Create Volatile Files and Directories.  
         Stopping Import network configuration from initramfs...  
[    
  OK     
] Stopped Import network configuration from initramfs.  
[ [3[  103.264089] type=1131 audit(1422213439.363:98): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg=' comm="rhel-readonly" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' 
2m  OK     
] Stopped target Local File Systems.  
         Unmounting /home...  
         Unmounting /boot...  
         Stopping Configure read-only root support...  
[    
  OK     
] Stopped Configure read-only root support.  
[  103.461750] type=1131 audit(1422213439.561:99): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg=' comm="systemd-remount-fs" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' 
         Stopping Monitoring of LVM2 mirrors, snapshots etc. ...ress polling...  
[    
  OK     
] Unmounted /home.  
[    
  OK     
] Unmounted /boot.  
[    
  OK     
] Reached target Unmount All Filesystems.  
[    
  OK     
] Stopped target Local File Systems (Pre).  
         Stopping Remount Root and Kernel File Systems...  
[    
  OK     
] Stopped Remount Root and Kernel File Systems.  
[  103.563541] type=1131 audit(1422213439.663:100): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg=' comm="lvm2-monitor" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' 
[    
  OK     
] Reached target Shutdown.  
[  104.049099] type=1131 audit(1422213440.148:101): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg=' comm="lvm2-lvmetad" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' 
[  104.144945] hpwdt: Unexpected close, not stopping watchdog! 
[  105.527488] systemd-journald[674]: Received SIGTERM 
Sending SIGTERM to remaining processes...  
Sending SIGKILL to remaining processes...  
Hardware watchdog 'HP iLO2+ HW Watchdog Timer', version 0  
Unmounting file systems.  
[  247.048359] INFO: task systemd-shutdow:1 blocked for more than 120 seconds. 
[  247.083126] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. 
[  247.121821] systemd-shutdow D ffff88043f413680     0     1      0 0x00000000 
[  247.153529]  ffff88042955bc80 0000000000000086 ffff880429550000 ffff88042955bfd8 
[  247.186830]  ffff88042955bfd8 ffff88042955bfd8 ffff880429550000 ffff88042955bda8 
[  247.220117]  ffff88042955bdb0 7fffffffffffffff ffff880429550000 ffff8800363c1800 
[  247.253429] Call Trace: 
[  247.264372]  [<ffffffff81609269>] schedule+0x29/0x70 
[  247.286639]  [<ffffffff816071b9>] schedule_timeout+0x209/0x2d0 
[  247.313319]  [<ffffffff8108cfe6>] ? __queue_work+0x136/0x320 
[  247.338856]  [<ffffffff8108d29a>] ? __queue_delayed_work+0xaa/0x1a0 
[  247.367055]  [<ffffffff81609766>] wait_for_completion+0x116/0x170 
[  247.394432]  [<ffffffff810a94d0>] ? wake_up_state+0x20/0x20 
[  247.419467]  [<ffffffff811f020e>] writeback_inodes_sb_nr+0x8e/0xd0 
[  247.447232]  [<ffffffff811f0275>] writeback_inodes_sb+0x25/0x30 
[  247.473870]  [<ffffffff811f6acd>] sync_filesystem+0x2d/0xa0 
[  247.499012]  [<ffffffff811ca294>] do_remount_sb+0x54/0x1a0 
[  247.523811]  [<ffffffff811e8519>] do_mount+0x6a9/0xa40 
[  247.546481]  [<ffffffff811d6c25>] ? getname_flags+0xb5/0x190 
[  247.572354]  [<ffffffff811887e7>] ? do_munmap+0x2b7/0x410 
[  247.599158]  [<ffffffff811e8946>] SyS_mount+0x96/0xf0 
[  247.624411]  [<ffffffff81613de9>] system_call_fastpath+0x16/0x1b 
[  247.653011] INFO: task kworker/u128:3:484 blocked for more than 120 seconds. 
[  247.686231] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. 
[  247.721494] kworker/u128:3  D ffff88043f473680     0   484      2 0x00000000 
[  247.753279] Workqueue: writeback bdi_writeback_workfn (flush-253:3) 
[  247.783534]  ffff88042332b750 0000000000000046 ffff880424fcf1c0 ffff88042332bfd8 
[  247.819205]  ffff88042332bfd8 ffff88042332bfd8 ffff880424fcf1c0 ffff88043f473f48 
[  247.852455]  ffff8804275007e0 ffff8804226c9270 ffff880427500810 ffff8804275007e0 
[  247.885415] Call Trace: 
[  247.896377]  [<ffffffff8160956d>] io_schedule+0x9d/0x130 
[  247.920248]  [<ffffffff812ac3c5>] get_request+0x1b5/0x780 
[  247.944603]  [<ffffffff81098230>] ? wake_up_bit+0x30/0x30 
[  247.968937]  [<ffffffff812ae6a6>] blk_queue_bio+0xc6/0x390 
[  247.993610]  [<ffffffffa0002735>] dm_request+0x145/0x1a0 [dm_mod] 
[  248.021168]  [<ffffffff812aa502>] generic_make_request+0xe2/0x130 
[  248.048552]  [<ffffffff812aa5c1>] submit_bio+0x71/0x150 
[  248.072023]  [<ffffffffa03d3813>] xfs_submit_ioend_bio.isra.12+0x33/0x40 [xfs] 
[  248.105741]  [<ffffffffa03d390f>] xfs_submit_ioend+0xef/0x130 [xfs] 
[  248.137029]  [<ffffffffa03d46ca>] xfs_vm_writepage+0x36a/0x5d0 [xfs] 
[  248.168337]  [<ffffffff81160cd3>] __writepage+0x13/0x50 
[  248.191834]  [<ffffffff811617f1>] write_cache_pages+0x251/0x4d0 
[  248.218410]  [<ffffffff81160cc0>] ? global_dirtyable_memory+0x70/0x70 
[  248.247365]  [<ffffffff81161abd>] generic_writepages+0x4d/0x80 
[  248.273583]  [<ffffffffa03d3ea3>] xfs_vm_writepages+0x43/0x50 [xfs] 
[  248.301722]  [<ffffffff81162b6e>] do_writepages+0x1e/0x40 
[  248.326111]  [<ffffffff811efe90>] __writeback_single_inode+0x40/0x220 
[  248.355023]  [<ffffffff811f090e>] writeback_sb_inodes+0x25e/0x420 
[  248.382451]  [<ffffffff811f124f>] wb_writeback+0xff/0x2f0 
[  248.406756]  [<ffffffff810925f6>] ? set_worker_desc+0x86/0xb0 
[  248.432603]  [<ffffffff811f2ac5>] bdi_writeback_workfn+0x115/0x450 
[  248.460385]  [<ffffffff8108f0ab>] process_one_work+0x17b/0x470 
[  248.486196]  [<ffffffff8108fe8b>] worker_thread+0x11b/0x400 
[  248.511370]  [<ffffffff8108fd70>] ? rescuer_thread+0x400/0x400 
[  248.537604]  [<ffffffff8109726f>] kthread+0xcf/0xe0 
[  248.559602]  [<ffffffff810971a0>] ? kthread_create_on_node+0x140/0x140 
[  248.589135]  [<ffffffff81613d3c>] ret_from_fork+0x7c/0xb0 
[  248.614083]  [<ffffffff810971a0>] ? kthread_create_on_node+0x140/0x140 
[  248.646002] INFO: task xfsaild/dm-3:2129 blocked for more than 120 seconds. 
[  248.680075] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. 
[  248.716515] xfsaild/dm-3    D ffff88043f573680     0  2129      2 0x00000080 
[  248.747793]  ffff8804267b7a68 0000000000000046 ffff8804160b16c0 ffff8804267b7fd8 
[  248.781258]  ffff8804267b7fd8 ffff8804267b7fd8 ffff8804160b16c0 ffff88043f573f48 
[  248.815287]  ffff8804275007e0 ffff880424be3e38 ffff880427500810 ffff8804275007e0 
[  248.848672] Call Trace: 
[  248.859652]  [<ffffffff8160956d>] io_schedule+0x9d/0x130 
[  248.883328]  [<ffffffff812ac3c5>] get_request+0x1b5/0x780 
[  248.907591]  [<ffffffffa0432d77>] ? xfs_mount_validate_sb+0x337/0x4a0 [xfs] 
[  248.939020]  [<ffffffff81098230>] ? wake_up_bit+0x30/0x30 
[  248.963352]  [<ffffffff812ae6a6>] blk_queue_bio+0xc6/0x390 
[  248.987965]  [<ffffffffa0002735>] dm_request+0x145/0x1a0 [dm_mod] 
[  249.015399]  [<ffffffff812aa502>] generic_make_request+0xe2/0x130 
[  249.042813]  [<ffffffff812aa5c1>] submit_bio+0x71/0x150 
[  249.066303]  [<ffffffffa03d91cb>] _xfs_buf_ioapply+0x2db/0x420 [xfs] 
[  249.094982]  [<ffffffffa03daae5>] ? xfs_bdstrat_cb+0x55/0xb0 [xfs] 
[  249.122542]  [<ffffffffa03daa36>] xfs_buf_iorequest+0x46/0xa0 [xfs] 
[  249.153598]  [<ffffffffa03daae5>] xfs_bdstrat_cb+0x55/0xb0 [xfs] 
[  249.183756]  [<ffffffffa03dacc3>] __xfs_buf_delwri_submit+0x183/0x230 [xfs] 
[  249.215601]  [<ffffffffa03db7ef>] ? xfs_buf_delwri_submit_nowait+0x2f/0x50 [xfs] 
[  249.248874]  [<ffffffffa043d2f0>] ? xfs_trans_ail_cursor_first+0x90/0x90 [xfs] 
[  249.280936]  [<ffffffffa03db7ef>] xfs_buf_delwri_submit_nowait+0x2f/0x50 [xfs] 
[  249.313511]  [<ffffffffa043d530>] xfsaild+0x240/0x5e0 [xfs] 
[  249.338664]  [<ffffffffa043d2f0>] ? xfs_trans_ail_cursor_first+0x90/0x90 [xfs] 
[  249.371079]  [<ffffffff8109726f>] kthread+0xcf/0xe0 
[  249.393020]  [<ffffffff810971a0>] ? kthread_create_on_node+0x140/0x140 
[  249.422406]  [<ffffffff81613d3c>] ret_from_fork+0x7c/0xb0 
[  249.446684]  [<ffffffff810971a0>] ? kthread_create_on_node+0x140/0x140 
[  249.476609] [sched_delayed] sched: RT throttling activated 


Additional info:
#
# /etc/fstab
# Created by anaconda on Sun Jan 25 18:05:04 2015
#
# Accessible filesystems, by reference, are maintained under '/dev/disk'
# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info
#
/dev/mapper/rhel_storageqe--85-root /                       xfs     defaults        0 0
UUID=94f7b292-9f4e-4717-9d38-5efcb5418d91 /boot                   xfs     defaults        0 0
/dev/mapper/rhel_storageqe--85-home /home                   xfs     defaults        0 0
/dev/mapper/rhel_storageqe--85-swap swap                    swap    defaults        0 0
UUID=0fc1f3be-9779-4cf5-a2c3-c434e1e6a1ee       /root/os_reboot xfs     _netdev 0 2
Comment 2 Bruno Goncalves 2015-02-04 13:46:28 UTC 
The problem is still reproducible with iscsi-initiator-utils-6.2.0.873-29.el7


# cat /etc/fstab

#
# /etc/fstab
# Created by anaconda on Wed Feb  4 10:52:42 2015
#
# Accessible filesystems, by reference, are maintained under '/dev/disk'
# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info
#
/dev/mapper/rhel_hp--nehalem--01-root /                       xfs     defaults        0 0
UUID=8d4d9db0-e5b9-45a5-9ddb-9e730f524237 /boot                   xfs     defaults        0 0
/dev/mapper/rhel_hp--nehalem--01-home /home                   xfs     defaults        0 0
/dev/mapper/rhel_hp--nehalem--01-swap swap                    swap    defaults        0 0
UUID=761abd20-2990-49cf-95cc-b7e16f8b5e50       /root/os_reboot xfs     _netdev 0 2

# dd if=/dev/urandom of=/root/os_reboot/test.img &

# reboot

##############################################################################

[-- MARK -- Wed Feb  4 12:00:00 2015] 
[   65.413398] XFS (dm-3): Mounting V4 Filesystem 
[   65.479579] XFS (dm-3): Ending clean mount 
2015-02-04 07:00:32,488 beah buildProtocol: INFO TaskListener: Connected.  Address: UNIXAddress('') 
2015-02-04 07:00:32,489 beah proc_evt_rebooting: INFO System reboot triggered by the task 
2015-02-04 07:00:32,525 backend linfo: INFO BackendFactory: Lost connection.  Reason: [Failure instance: Traceback (failure with no frames): <class 'twisted.internet.error.ConnectionLost'>: Connection to the other side was lost in a non-clean fashion: Connection lost. 
] 
[   95.909935] device-mapper: multipath: Failing path 8:16. 
[   95.941654] device-mapper: multipath: Failing path 8:32. 
2015-02-04 07:00:32,667 backend set_controller: INFO Connection to controller lost.  
2015-02-04 07:00:32,668 backend linfo: INFO BackendFactory: Lost connection.  Reason: [Failure instance: Traceback (failure with no frames): <class 'twisted.internet.error.ConnectionLost'>: Connection to the other side was lost in a non-clean fashion: Connection lost.  
]  
2015-02-04 07:00:32,668 backend set_controller: INFO Connection to controller lost.  
[    
  OK     
] Started Show Plymouth Reboot Screen.  
[    
  OK     
] Stopped The Beaker backend server..  
[    
  OK     
] Stopped target System Time Synchronized.  
         Stopping Wait for chrony to synchronize system clock...  
[    
  OK     
] Stopped Wait for chrony to synchronize system clock.  
         Stopping NTP client/server...  
[    
  OK     
] Stopped NTP client/server.  
[    
  OK     
] Stopped Logout off all iSCSI sessions on shutdown.  
         Stopping Open-iSCSI...  
[    
  OK     
] Started Restore /run/initramfs.  
2015-02-04 07:00:32,526 rhts_task controller_disconnected: ERROR Connection to controller was lost! reason=[Failure instance: Traceback (failure with no frames): <class 'twisted.internet.error.ConnectionDone'>: Connection was closed cleanly.  
]  
[    
  OK     
] Stopped Dynamic System Tuning Daemon.  
[    
  OK     
] Stopped D-Bus System Message Bus.  
2015-02-04 07:00:33,688 rhts_task on_exit: INFO quitting...  
[    
  OK     
] Stopped Login Service.  
[   97.910106] end_request: I/O error, dev dm-3, sector 169984 
[    
  OK     
[   97.942700] end_request: I/O error, dev dm-3, sector 168960 
] Stopped Open-i[   97.975075] end_request: I/O error, dev dm-3, sector 167936 
SCSI.  
        [   98.008147] end_request: I/O error, dev dm-3, sector 166912 
 Stopping Device[   98.041050] end_request: I/O error, dev dm-3, sector 165888 
-Mapper Multipat[   98.073574] end_request: I/O error, dev dm-3, sector 164864 
h Device Control[   98.106163] end_request: I/O error, dev dm-3, sector 163840 
ler...  
[   98.136255] end_request: I/O error, dev dm-3, sector 162816 
[   98.167087] end_request: I/O error, dev dm-3, sector 161792 
[   98.197339] end_request: I/O error, dev dm-3, sector 160768 
[   98.229149] Buffer I/O error on device dm-3, logical block 21259 
[   98.262711] lost page write due to I/O error on dm-3 
[   98.291167] Buffer I/O error on device dm-3, logical block 21260 
[   98.326277] lost page write due to I/O error on dm-3 
[   98.354657] Buffer I/O error on device dm-3, logical block 21261 
[   98.389698] lost page write due to I/O error on dm-3 
[   98.416488] Buffer I/O error on device dm-3, logical block 21262 
[   98.449590] lost page write due to I/O error on dm-3 
[   98.476641] Buffer I/O error on device dm-3, logical block 21263 
[   98.511004] lost page write due to I/O error on dm-3 
[   98.538961] Buffer I/O error on device dm-3, logical block 21264 
[   98.573326] lost page write due to I/O error on dm-3 
[   98.600169] Buffer I/O error on device dm-3, logical block 16384 
[   98.602952] Buffer I/O error on device dm-3, logical block 21265 
[   98.602953] lost page write due to I/O error on dm-3 
[   98.602955] Buffer I/O error on device dm-3, logical block 21266 
[   98.602956] lost page write due to I/O error on dm-3 
[   98.755504] lost page write due to I/O error on dm-3 
[   98.789153] XFS (dm-3): metadata I/O error: block 0xa00048 ("xlog_iodone") error 5 numblks 64 
[   98.838477] XFS (dm-3): xfs_do_force_shutdown(0x2) called from line 1172 of file fs/xfs/xfs_log.c.  Return address = 0xffffror 5 returned. 
[  103.008756] XFS (dm-3): Please umount the filesystem and rectify the problem(s) 
[   
    
*        
] (1 of 2) A stop job is running for Postfix Mail Transport Agent[  103.077362] blk_update_request: 52 callbacks suppressed 
[  103.107453] end_request: I/O error, dev dm-3, sector 20971392 
   
[    
  OK  [  103.141531] end_request: I/O error, dev dm-3, sector 20971392 
   
] Stopped Device-Mapper Multipath Device Controller.  
[    
  OK     
] Stopped Postfix Mail Transport Agent.  
[    
  OK     
] Stopped target Network.  
         Stopping LSB: Bring up/down networking...  
[    
  OK     
] Stopped LSB: Bring up/down networking.  
         Stopping Network Manager...  
[    
  OK     
] Stopped Network Manager.  
[    
  OK     
] Stopped target Basic System.  
[    
  OK     
] Stopped target Slices.  
[    
  OK     
] Removed slice User and Session Slice.  
[    
  OK     
] Stopped target Paths.  
[    
  OK     
] Stopped target Timers.  
[    
  OK     
] Stopped target Sockets.  
[    
  OK     
] Closed RPCbind Server Activation Socket.  
[    
  OK     
] Closed Open-iSCSI iscsiuio Socket.  
[    
  OK     
] Closed Open-iSCSI iscsid Socket.  
[    
  OK     
] Closed D-Bus System Message Bus Socket.  
[    
  OK     
] Stopped target System Initialization.  
         Stopping Setup Virtual Console...  
[    
  OK     
] Stopped Setup Virtual Console.  
[    
  OK     
] Stopped target Encrypted Volumes.  
         Stopping Load/Save Random Seed...  
         Stopping Apply Kernel Variables...  
[    
  OK     
] Stopped Apply Kernel Variables.  
         Stopping Update UTMP about System Reboot/Shutdown...  
[    
  OK     
] Stopped target Swap.  
         Deactivating swap /dev/dm-1...  
         Deactivating swap /dev/dm-1...  
         Deactivating swap /dev/dm-1...  
         Deactivating swap /dev/dm-1...  
         Deactivating swap /dev/dm-1...  
         Deactivating swap /dev/dm-1...  
[    
  OK     
] Stopped Load/Save Random Seed.  
[    
  OK     
] Stopped Update UTMP about System Reboot/Shutdown.  
         Stopping Security Auditing Service...  
[  108.073433] audit_printk_skb: 9765 callbacks suppressed 
[    
  OK     
[  108.102760] type=1305 audit(1423051244.785:95): audit_pid=0 old=1075 auid=4294967295 ses=4294967295 subj=system_u:system_r:auditd_t:s0 res=1 
] Deactivated sw[  108.173983] type=1131 audit(1423051244.887:96): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg=' comm="auditd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' 
ap /dev/rhel_hp-nehalem-01/swap.  
[    
  OK     
] Deactivated swap /dev/disk/by-uuid/6a7df2b3[  108.283829] type=1131 audit(1423051244.996:97): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg=' comm="systemd-tmpfiles-setup" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' 
-849b-4f9...e-e4[  108.404463] type=1131 audit(1423051245.117:98): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg=' comm="rhel-import-state" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' 
26b140802e.  
[    
  OK     
] Deactivated swap /dev/disk/by-id/dm-uuid-LVM-OfnGdRM...iXYEf61dEi9xRU.  
[    
  OK     
] Deactivated swap /dev[  108.517737] type=1131 audit(1423051245.230:99): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg=' comm="rhel-readonly" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' 
/disk/by-id/dm-name-rhel_hp--nehalem--01-swap.  
[    
  OK     
] Deactivated swap /dev/mapper/rhel_hp--nehalem--01-swap.  
[    
  OK     
] Deactivated swap /dev/dm-1.  
[    
  OK     
] Stopped Security Auditing Service.  
         Stopping Create Volatile Files and Directories...  [  104967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg=' comm="systemd-remount-fs" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' 

[    
  OK     
] Stopped Create Volatile Files and Directories.  
         Stopping Import network configuration from initramfs...  
[    
  OK     
] Stopped Import network configuration from initramfs.  
[    
  OK     
] Stopped target Local File Systems.  
         Unmounting /home...  
         Unmounting /boot...  
         Stopping Configure read-only root support...  
[    
  OK     
] Stopped Configure read-only root support.  
         Stopping Monitoring of LVM2 mirrors, snapshots etc. ...ress polling...  
[    
  OK     
] Unmounted /home.  
[    
  OK     
] Unmounted /boot.  
[    
  OK     
] Reached target Unmount All Filesystems.  
[    
  OK     
] Stopped target Local File Systems (Pre).  
         Stopping Remount Root and Kernel File Systems...  
[    
  OK     
] St[  110.719304] type=1131 audit(1423051247.432:101): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg=' comm="lvm2-monitor" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' 
opped Remount Root and Kernel File Systems.  
[    
  OK     
] Reached target Shutdown.  
[  111.231388] type=1131 audit(1423051247.945:102): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg=' comm="lvm2-lvmetad" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' 
[  111.359459] hpwdt: Unexpected close, not stopping watchdog! 
[  111.453564] systemd-journald[849]: Received SIGTERM 
Sending SIGTERM to remaining processes...  
Sending SIGKILL to remaining pro[  111.519427] XFS (dm-3): xfs_log_force: error 5 returned. 
cesses...  
Hardware watchdog 'HP iLO2+ HW Watchdog Timer', version 0  
Unmounting file systems.  
[  125.678445] XFS (dm-3): xfs_log_force: error 5 returned. 
[  155.747712] XFS (dm-3): xfs_log_force: error 5 returned. 
[  185.817059] XFS (dm-3): xfs_log_force: error 5 returned. 
[  215.886381] XFS (dm-3): xfs_log_force: error 5 returned. 
[  245.955603] XFS (dm-3): xfs_log_force: error 5 returned. 
[  276.024869] XFS (dm-3): xfs_log_force: error 5 returned. 
[  306.093920] XFS (dm-3): xfs_log_force: error 5 returned. 
[  336.163321] XFS (dm-3): xfs_log_force: error 5 returned. 
[-- MARK -- Wed Feb  4 12:05:00 2015] 
[  366.232476] XFS (dm-3): xfs_log_force: error 5 returned. 
[  396.301847] XFS (dm-3): xfs_log_force: error 5 returned. 
[  426.371158] XFS (dm-3): xfs_log_force: error 5 returned. 
[  456.440424] XFS (dm-3): xfs_log_force: error 5 returned. 
[  486.509753] XFS (dm-3): xfs_log_force: error 5 returned. 
[  516.579029] XFS (dm-3): xfs_log_force: error 5 returned. 
[  546.648313] XFS (dm-3): xfs_log_force: error 5 returned. 
[  576.717564] XFS (dm-3): xfs_log_force: error 5 returned. 
[  606.786632] XFS (dm-3): xfs_log_force: error 5 returned. 
[  636.856122] XFS (dm-3): xfs_log_force: error 5 returned. 
[-- MARK -- Wed Feb  4 12:10:00 2015] 
[  666.925366] XFS (dm-3): xfs_log_force: error 5 returned. 
[  696.994634] XFS (dm-3): xfs_log_force: error 5 returned. 
[  702.507267] Kernel panic - not syncing: An NMI occurred. Depending on your system the reason for the NMI is logged in any one of the following resources: 
[  702.507267] 1. Integrated Management Log (IML) 
[  702.507267] 2. OA Syslog 
[  702.507267] 3. OA Forward Progress Log 
[  702.507267] 4. iLO Event Log 
[  702.662624] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 3.10.0-229.el7.x86_64 #1 
[  702.703551] Hardware name: HP ProLiant DL580 G7, BIOS P65 12/01/2011 
[  702.740997]  ffffffffa009d2d8 7274911a29f17ecf ffff88013b805de0 ffffffff81603f36 
[  702.784204]  ffff88013b805e60 ffffffff815fd7da ffffc90000000008 ffff88013b805e70 
[  702.827520]  ffff88013b805e10 7274911a29f17ecf 0000000000000000 ffffc90011d10072 
[  702.870903] Call Trace: 
[  702.885491]  <NMI>  [<ffffffff81603f36>] dump_stack+0x19/0x1b 
[  702.918568]  [<ffffffff815fd7da>] panic+0xd8/0x1e7 
[  702.945559]  [<ffffffffa009c8ed>] hpwdt_pretimeout+0xdd/0xe0 [hpwdt] 
[  702.981504]  [<ffffffff8160c719>] nmi_handle.isra.0+0x69/0xb0 
[  703.014527]  [<ffffffff8160c886>] do_nmi+0x126/0x340 
[  703.043810]  [<ffffffff8160bb71>] end_repeat_nmi+0x1e/0x2e 
[  703.075070]  [<ffffffff8133c992>] ? intel_idle+0xd2/0x160 
[  703.107015]  [<ffffffff8133c992>] ? intel_idle+0xd2/0x160 
[  703.136386]  [<ffffffff8133c992>] ? intel_idle+0xd2/0x160 
[  703.167933]  <<EOE>>  [<ffffffff814a9c30>] cpuidle_enter_state+0x40/0xc0 
[  703.205673]  [<ffffffff814a9d75>] cpuidle_idle_call+0xc5/0x200 
[  703.238686]  [<ffffffff8101d14e>] arch_cpu_idle+0xe/0x30 
[  703.268415]  [<ffffffff810c67d1>] cpu_startup_entry+0xf1/0x290 
[  703.303039]  [<ffffffff815f2457>] rest_init+0x77/0x80 
[  703.331414]  [<ffffffff81a45057>] start_kernel+0x429/0x44a 
[  703.363980]  [<ffffffff81a44a37>] ? repair_env_string+0x5c/0x5c 
[  703.398283]  [<ffffffff81a44120>] ? early_idt_handlers+0x120/0x120 
[  703.434864]  [<ffffffff81a445ee>] x86_64_start_reservations+0x2a/0x2c 
[  703.471815]  [<ffffffff81a44742>] x86_64_start_kernel+0x152/0x175 
[  703.506685] drm_kms_helper: panic occurred, switching back to text console 
[  703.704531] ------------[ cut here ]------------ 
[  703.733644] WARNING: at kernel/rcutree.c:502 rcu_eqs_exit_common.isra.32+0xd4/0xe0() 
[  703.779165] Modules linked in: dm_service_time iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi coretemp kvm_intel kvm ipmi_si iTCO_wdt crc32c_intel iTCO_vendor_support hpwdt hpilo pcspkr i7core_edac serio_raw ipmi_msghandler acpi_power_meter shpchp lpc_ich edac_core mfd_core pcc_cpufreq dm_multipath xfs libcrc32c sr_mod cdrom sd_mod radeon crc_t10dif ata_generic i2c_algo_bit crct10dif_common pata_acpi drm_kms_helper ttm drm ata_piix libata i2c_core hpsa netxen_nic dm_mirror dm_region_hash dm_log dm_mod 
[  704.044237] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 3.10.0-229.el7.x86_64 #1 
[  704.083954] Hardware name: HP ProLiant DL580 G7, BIOS P65 12/01/2011 
[  704.118198]  0000000000000000 7274911a29f17ecf ffff88013b803ee0 ffffffff81603f36 
[  704.159322]  ffff88013b803f18 ffffffff8106e28b ffff88013b80dc60 0000000000000001 
[  704.200394]  0000000000000000 0000000000000000 ffff88013b805ef8 ffff88013b803f28 
[  704.241946] Call Trace: 
[  704.257788]  <IRQ>  [<ffffffff81603f36>] dump_stack+0x19/0x1b 
[  704.290012]  [<ffffffff8106e28b>] warn_slowpath_common+0x6b/0xb0 
[  704.324968]  [<ffffffff8106e3da>] warn_slowpath_null+0x1a/0x20 
[  704.358552]  [<ffffffff811127f4>] rcu_eqs_exit_common.isra.32+0xd4/0xe0 
[  704.397608]  [<ffffffff81114a25>] rcu_irq_enter+0x65/0x90 
[  704.428606]  [<ffffffff81077d27>] irq_enter+0x17/0xa0 
[  704.458097]  [<ffffffff81616345>] smp_apic_timer_interrupt+0x35/0x60 
[  704.495007]  [<ffffffff81614a1d>] apic_timer_interrupt+0x6d/0x80 
[  704.529937]  <EOI>  <NMI>  [<ffffffff815fd8a5>] ? panic+0x1a3/0x1e7 
[  704.564614]  [<ffffffffa009c8ed>] hpwdt_pretimeout+0xdd/0xe0 [hpwdt] 
[  704.602138]  [<ffffffff8160c719>] nmi_handle.isra.0+0x69/0xb0 
[  704.633171]  [<ffffffff8160c886>] do_nmi+0x126/0x340 
[  704.661110]  [<ffffffff8160bb71>] end_repeat_nmi+0x1e/0x2e 
[  704.691184]  [<ffffffff8133c992>] ? intel_idle+0xd2/0x160 
[  704.721899]  [<ffffffff8133c992>] ? intel_idle+0xd2/0x160 
[  704.752041]  [<ffffffff8133c992>] ? intel_idle+0xd2/0x160 
[  704.784578]  <<EOE>>  [<ffffffff814a9c30>] cpuidle_enter_state+0x40/0xc0 
[  704.822995]  [<ffffffff814a9d75>] cpuidle_idle_call+0xc5/0x200 
[  704.858001]  [<ffffffff8101d14e>] arch_cpu_idle+0xe/0x30 
[  704.888346]  [<ffffffff810c67d1>] cpu_startup_entry+0xf1/0x290 
[  704.921929]  [<ffffffff815f2457>] rest_init+0x77/0x80 
[  704.951261]  [<ffffffff81a45057>] start_kernel+0x429/0x44a 
[  704.984471]  [<ffffffff81a44a37>] ? repair_env_string+0x5c/0x5c 
[  705.017186]  [<ffffffff81a44120>] ? early_idt_handlers+0x120/0x120 
[  705.051461]  [<ffffffff81a445ee>] x86_64_start_reservations+0x2a/0x2c 
[  705.087827]  [<ffffffff81a44742>] x86_64_start_kernel+0x152/0x175 
[  705.123154] ---[ end trace 4b55dd22c8a6220d ]--- 
[  705.148648] ------------[ cut here ]------------ 
[  705.175578] WARNING: at kernel/rcutree.c:363 rcu_eqs_enter_common.isra.31+0xd5/0xf0() 
[  705.220057] Modules linked in: dm_service_time iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi coretemp kvm_intel kvm ipmi_si iTCO_wdt crc32c_intel iTCO_vendor_support hpwdt hpilo pcspkr i7core_edac serio_raw ipmi_msghandler acpi_power_meter shpchp lpc_ich edac_core mfd_core pcc_cpufreq dm_multipath xfs libcrc32c sr_mod cdrom sd_mod radeon crc_t10dif ata_generic i2c_algo_bit crct10dif_common pata_acpi drm_kms_helper ttm drm ata_piix libata i2c_core hpsa netxen_nic dm_mirror dm_region_hash dm_log dm_mod 
[  705.478891] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G        W   --------------   3.10.0-229.el7.x86_64 #1 
[  705.533288] Hardware name: HP ProLiant DL580 G7, BIOS P65 12/01/2011 
[  705.570328]  0000000000000000 7274911a29f17ecf ffff88013b803ee0 ffffffff81603f36 
[  705.612930]  ffff88013b803f18 ffffffff8106e28b ffff88013b80dc60 0000000000000006 
[  705.654857]  0000000000000000 0000000000000000 ffff88013b805ef8 ffff88013b803f28 
[  705.695995] Call Trace: 
[  705.709259]  <IRQ>  [<ffffffff81603f36>] dump_stack+0x19/0x1b 
[  705.741218]  [<ffffffff8106e28b>] warn_slowpath_common+0x6b/0xb0 
[  705.774503]  [<ffffffff8106e3da>] warn_slowpath_null+0x1a/0x20 
[  705.809719]  [<ffffffff81112665>] rcu_eqs_enter_common.isra.31+0xd5/0xf0 
[  705.848082]  [<ffffffff811148dd>] rcu_irq_exit+0x5d/0x90 
[  705.878994]  [<ffffffff81077e40>] irq_exit+0x90/0x120 
[  705.908035]  [<ffffffff81616355>] smp_apic_timer_interrupt+0x45/0x60 
[  705.944631]  [<ffffffff81614a1d>] apic_timer_interrupt+0x6d/0x80 
[  705.979628]  <EOI>  <NMI>  [<ffffffff815fd8a5>] ? panic+0x1a3/0x1e7 
[  706.016676]  [<ffffffffa009c8ed>] hpwdt_pretimeout+0xdd/0xe0 [hpwdt] 
[  706.054138]  [<ffffffff8160c719>] nmi_handle.isra.0+0x69/0xb0 
[  706.087316]  [<ffffffff8160c886>] do_nmi+0x126/0x340 
[  706.115196]  [<ffffffff8160bb71>] end_repeat_nmi+0x1e/0x2e 
[  706.147468]  [<ffffffff8133c992>] ? intel_idle+0xd2/0x160 
[  706.178284]  [<ffffffff8133c992>] ? intel_idle+0xd2/0x160 
[  706.209749]  [<ffffffff8133c992>] ? intel_idle+0xd2/0x160 
[  706.240184]  <<EOE>>  [<ffffffff814a9c30>] cpuidle_enter_state+0x40/0xc0 
[  706.279347]  [<ffffffff814a9d75>] cpuidle_idle_call+0xc5/0x200 
[  706.310855]  [<ffffffff8101d14e>] arch_cpu_idle+0xe/0x30 
[  706.341515]  [<ffffffff810c67d1>] cpu_startup_entry+0xf1/0x290 
[  706.375406]  [<ffffffff815f2457>] rest_init+0x77/0x80 
[  706.404426]  [<ffffffff81a45057>] start_kernel+0x429/0x44a 
[  706.436442]  [<ffffffff81a44a37>] ? repair_env_string+0x5c/0x5c 
[  706.471037]  [<ffffffff81a44120>] ? early_idt_handlers+0x120/0x120 
[  706.506437]  [<ffffffff81a445ee>] x86_64_start_reservations+0x2a/0x2c 
[  706.544000]  [<ffffffff81a44742>] x86_64_start_kernel+0x152/0x175 
[  706.578529] ---[ end trace 4b55dd22c8a6220e ]--- 
[  706.605489] ------------[ cut here ]------------ 
[  706.631775] WARNING: at kernel/time/tick-broadcast.c:600 tick_handle_oneshot_broadcast+0x1c1/0x1f0() 
[  706.684108] Modules linked in: dm_service_time iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi coretemp kvm_intel kvm ipmi_si iTCO_wdt crc32c_intel iTCO_vendor_support hpwdt hpilo pcspkr i7core_edac serio_raw ipmi_msghandler acpi_power_meter shpchp lpc_ich edac_core mfd_core pcc_cpufreq dm_multipath xfs libcrc32c sr_mod cdrom sd_mod radeon crc_t10dif ata_generic i2c_algo_bit crct10dif_common pata_acpi drm_kms_helper ttm drm ata_piix libata i2c_core hpsa netxen_nic dm_mirror dm_region_hash dm_log dm_mod 
[  706.942429] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G        W   --------------   3.10.0-229.el7.x86_64 #1 
[  706.997225] Hardware name: HP ProLiant DL580 G7, BIOS P65 12/01/2011 
[  707.032165]  0000000000000000 7274911a29f17ecf ffff88013b803e18 ffffffff81603f36 
[  707.075484]  ffff88013b803e50 ffffffff8106e28b 7fffffffffffffff 000000a3579dbe08 
[  707.116373]  000000000000da00 0000000000000000 ffffffff81922840 ffff88013b803e60 
[  707.158792] Call Trace: 
[  707.171874]  <IRQ>  [<ffffffff81603f36>] dump_stack+0x19/0x1b 
[  707.205633]  [<ffffffff8106e28b>] warn_slowpath_common+0x6b/0xb0 
[  707.238056]  [<ffffffff8106e3da>] warn_slowpath_null+0x1a/0x20 
[  707.272505]  [<ffffffff810cf231>] tick_handle_oneshot_broadcast+0x1c1/0x1f0 
[  707.312080]  [<ffffffff810164a5>] timer_interrupt+0x15/0x20 
[  707.345022]  [<ffffffff8110b2de>] handle_irq_event_percpu+0x3e/0x1e0 
[  707.381589]  [<ffffffff8110b4bd>] handle_irq_event+0x3d/0x60 
[  707.416554]  [<ffffffff8110e157>] handle_edge_irq+0x77/0x130 
[  707.449658]  [<ffffffff81015c9f>] handle_irq+0xbf/0x150 
[  707.479062]  [<ffffffff81077d27>] ? irq_enter+0x17/0xa0 
[  707.507208]  [<ffffffff8161626f>] do_IRQ+0x4f/0xf0 
[  707.534206]  [<ffffffff8160b4ed>] common_interrupt+0x6d/0x6d 
[  707.564697]  <EOI>  <NMI>  [<ffffffff815fd8a5>] ? panic+0x1a3/0x1e7 
[  707.599913]  [<ffffffffa009c8ed>] hpwdt_pretimeout+0xdd/0xe0 [hpwdt] 
[  707.635420]  [<ffffffff8160c719>] nmi_handle.isra.0+0x69/0xb0 
[  707.666413]  [<ffffffff8160c886>] do_nmi+0x126/0x340 
[  707.694371]  [<ffffffff8160bb71>] end_repeat_nmi+0x1e/0x2e 
[  707.724212]  [<ffffffff8133c992>] ? intel_idle+0xd2/0x160 
[  707.753907]  [<ffffffff8160bb2e>] ? first_nmi+0x1e/0x1e 
[  707.782320]  [<ffffffff815fd8a5>] ? panic+0x1a3/0x1e7 
[  707.811558]  <<EOE>>  
[  707.823324] ---[ end trace 4b55dd22c8a6220f ]---
Comment 5 Chris Leech 2016-02-11 06:01:18 UTC 
Can we rested this against 7.2?

The systemd ordering of remote mount points with the _netdev option set in fstab should get set correctly without needing to reload the systemd fstab generator, but I'm not sure something didn't get messed up in the systemd-208 backport in 7.1.

After editing the fstab and mounting, "systemctl show mnt-os_reboot.mount | grep remote-fs" should show somthing.  It does with systemd-219 (7.2) but with systemd-208 (7.1) I'm still seeing local-fs ordering.
Comment 7 Martin Hoyer 2016-02-11 14:17:33 UTC 
Created attachment 1123163 [details]
RHEL-7.2 systemctl show mnt-os_reboot.mount

It is reproducible with RHEL-7.2
iSCSI device in /etc/fstab:
UUID=d8abde37-c9ff-4a3f-bce3-d9cde20b0d46 /root/os_reboot xfs _netdev 0 2
'systemctl show mnt-os_reboot.mount | grep remote-fs' command returns nothing. 
systemd-219-19.el7.x86_64
kernel-3.10.0-327.el7
Comment 9 Rob Evers 2016-07-21 19:26:19 UTC 
Deferring to rhel7.4.  We can re-evaluate if progress is made.
Comment 11 sandeep 2017-11-06 13:43:33 UTC 
We (HPE) have Seen similar crash on UEK7.4 kernel.

If you have access to UEK bugs then please take a look at,

UEK Bug 16120 - HPE MSA2050 Storage_OL 7.4 : UEK 4.1.12-94.3.9.el7uek.x86_64 PANIC on Long Duration IO
Comment 12 Trinh Dao 2018-01-23 20:55:08 UTC 
Chris, any update on this bug?
Comment 14 Trinh Dao 2018-08-06 18:40:40 UTC 
RH, 
This bug has been opened since 2015.  we are running out of time to get fix in RHEL7.6. Do we have a root cause yet?
Comment 15 Chris Leech 2018-08-29 17:03:01 UTC 
I went back and reproduced this on 7.2 for comparison, taking a closer look at current 7.6 builds again now as my first attempt didn't reproduce it (but I might know why).

Looking way back on comment #7, if I add the fstab entry, mount, start traffic and reboot I can trigger this.  This is caused by systemd not recreating generated rules when configuration files change automatically.  After a "systemctl daemon-reload" or a reboot with the fstab entry in place I cannot reproduce this.  This has been pointed out to the systemd developers in the past and they don't plan on changing this, systemd controlled behaviour does not change with configuration file edits without a reload.

But, we also have the "iscsid.safe_logout" setting now as a safety net for these types of issues.  It wasn't on by default when first introduced, but I'd be willing to change that now as we haven't seen any problems with it.  It will cause iscsid to refuse to shutdown sessions with mounted filesystems.
Comment 17 Chris Leech 2018-08-30 15:59:38 UTC 
I haven't been able to reproduce this with recent el7 builds, not ruling it out but it's certainly not as easy.

I have gone ahead and changed the safe_logout checks to enabled by default, and I'm going to send this to QA.
Comment 19 Trinh Dao 2018-09-10 17:48:03 UTC 
RH, Is this fix going in RHEL7.6 Snapshot-3 or Snapshot-4?

thank you,
trinh
Comment 20 Martin Hoyer 2018-09-12 14:42:22 UTC 
Tested with iscsi-initiator-utils-6.2.0.874-10.el7
Panic not reproducible, safe_logout enabled by default.
No regression found.
Comment 21 Trinh Dao 2018-09-13 17:13:07 UTC 
Sandeep, 
please test with the latest RHEL7.6 Snapshot-3 to see if you can still reproduce this bug?

Thank you, 
trinh
Comment 22 Trinh Dao 2018-09-19 19:32:58 UTC 
Sandeep, 
any update on RHEL7.6 snapshot-3 test result?

thank you, 
trinh
Comment 23 Trinh Dao 2018-10-02 14:34:25 UTC 
Sandeep, 
I need test update. Did any RHEL7.6 build or latest snapshot-5 fix your bug?

thank you,
trinh
Comment 24 Trinh Dao 2018-10-02 14:36:14 UTC 
I received email from Paul Hinchman, HPE cannot reproduce this issue on RHEL.  I am clearing the flag and mark HPE verified.
Comment 26 errata-xmlrpc 2018-10-30 11:40:49 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2018:3281
Comment 27 Jean-Francois Remy 2019-01-21 22:40:12 UTC 
The use of safe_logout set to yes has some side effects that should probably mentioned.
If there are multiple iscsi connections to a drive under a multipath device, then it becomes impossible to logout any of the paths since iscsid will see the device as "in use" by device mapper.
Note You need to log in before you can comment on or make changes to this bug.