Bug 1237136 - BUG: unable to handle kernel NULL pointer dereference at <addr> in function __blkg_lookup
Summary: BUG: unable to handle kernel NULL pointer dereference at <addr> in function _...
Keywords:
Status: CLOSED NEXTRELEASE
Alias: None
Product: Fedora
Classification: Fedora
Component: kernel
Version: rawhide
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: Kernel Maintainer List
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
: 1237137 (view as bug list)
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2015-06-30 13:03 UTC by Richard W.M. Jones
Modified: 2015-09-24 05:07 UTC (History)
11 users (show)

Fixed In Version: 4.2.1-300.fc23
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2015-09-24 05:07:58 UTC
Type: Bug
Embargoed:


Attachments (Terms of Use)

Description Richard W.M. Jones 2015-06-30 13:03:14 UTC
Description of problem:

# lvm lvremove -f /dev/VG
  /run/lvm/lvmetad.socket: connect failed: No such file or directory
  WARNING: Failed to connect to lvmetad. Falling back to internal scanning.
[ 2932.455300] BUG: unable to handle kernel NULL pointer dereference at 0000000000000060
[ 2932.455300] IP: [<ffffffff8115e9ea>] css_next_child+0x12a/0x150
[ 2932.455300] PGD 0 
[ 2932.455300] Oops: 0000 [#6] SMP 
[ 2932.455300] Modules linked in: fat xfs nls_utf8 isofs btrfs xor raid6_pq kvm_amd kvm snd_pcsp snd_pcm snd_timer snd soundcore serio_raw ata_generic pata_acpi libcrc32c crc8 crc_itu_t crc_ccitt virtio_pci virtio_mmio virtio_input virtio_balloon virtio_scsi sym53c8xx scsi_transport_spi megaraid_sas megaraid_mbox megaraid_mm megaraid ideapad_laptop rfkill sparse_keymap video virtio_net virtio_gpu ttm drm_kms_helper drm virtio_console virtio_rng virtio_blk virtio_ring virtio crc32
[ 2932.455300] CPU: 0 PID: 6720 Comm: lvm Tainted: G      D         4.2.0-0.rc0.git2.1.fc23.x86_64 #1
[ 2932.455300] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.8.1-20150619_004852- 04/01/2014
[ 2932.455300] task: ffff88001b9b8000 ti: ffff88001856c000 task.ti: ffff88001856c000
[ 2932.455300] RIP: 0010:[<ffffffff8115e9ea>]  [<ffffffff8115e9ea>] css_next_child+0x12a/0x150
[ 2932.455300] RSP: 0018:ffff88001856f9f8  EFLAGS: 00000046
[ 2932.455300] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000002
[ 2932.455300] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
[ 2932.455300] RBP: ffff88001856fa28 R08: 0000000000000000 R09: 0000000000000001
[ 2932.455300] R10: ffff88001b9b8000 R11: 0000000000000000 R12: 0000000000000000
[ 2932.455300] R13: ffff88001b870000 R14: ffff88001b870a58 R15: 0000000000000000
[ 2932.455300] FS:  00007f8d2b2bc880(0000) GS:ffff88001ec00000(0000) knlGS:0000000000000000
[ 2932.455300] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 2932.455300] CR2: 0000000000000060 CR3: 000000001b7df000 CR4: 00000000000007f0
[ 2932.455300] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 2932.455300] DR3: 0000000000000000 DR6: 0000000000000000 DR7: 0000000000000000
[ 2932.455300] Stack:
[ 2932.455300]  ffff880019d50538 00000000f0a415c8 0000000000000000 0000000000000000
[ 2932.455300]  0000000000000000 ffff88001b870000 ffff88001856fa48 ffffffff81160710
[ 2932.455300]  ffff88001b870000 ffff880019d50600 ffff88001856fa88 ffffffff8140f510
[ 2932.455300] Call Trace:
[ 2932.455300]  [<ffffffff81160710>] css_next_descendant_post+0x70/0xd0
[ 2932.455300]  [<ffffffff8140f510>] blk_throtl_drain+0x90/0x220
[ 2932.455300]  [<ffffffff8140f4b6>] ? blk_throtl_drain+0x36/0x220
[ 2932.455300]  [<ffffffff813e8068>] ? blk_queue_bypass_start+0x68/0xb0
[ 2932.455300]  [<ffffffff8140c35d>] blkcg_drain_queue+0x2d/0x60
[ 2932.455300]  [<ffffffff813e6689>] __blk_drain_queue+0xc9/0x1a0
[ 2932.455300]  [<ffffffff813e8068>] ? blk_queue_bypass_start+0x68/0xb0
[ 2932.455300]  [<ffffffff813e8072>] blk_queue_bypass_start+0x72/0xb0
[ 2932.455300]  [<ffffffff8140a1c9>] blkcg_deactivate_policy+0x39/0x130
[ 2932.455300]  [<ffffffff8140f7f8>] blk_throtl_exit+0x38/0x50
[ 2932.455300]  [<ffffffff8140c3c9>] blkcg_exit_queue+0x39/0x40
[ 2932.455300]  [<ffffffff813ee73e>] blk_release_queue+0x1e/0xc0
[ 2932.455300]  [<ffffffff8141f16a>] kobject_release+0x7a/0x190
[ 2932.455300]  [<ffffffff8141f01f>] kobject_put+0x2f/0x60
[ 2932.455300]  [<ffffffff813e9e5b>] blk_cleanup_queue+0x18b/0x280
[ 2932.455300]  [<ffffffff8169df9e>] cleanup_mapped_device+0xde/0x110
[ 2932.455300]  [<ffffffff816a0ea8>] __dm_destroy+0x1a8/0x2c0
[ 2932.455300]  [<ffffffff816a23f3>] dm_destroy+0x13/0x20
[ 2932.455300]  [<ffffffff816a7e7d>] dev_remove+0x10d/0x170
[ 2932.455300]  [<ffffffff816a7d70>] ? dev_suspend+0x280/0x280
[ 2932.455300]  [<ffffffff816a8582>] ctl_ioctl+0x282/0x550
[ 2932.455300]  [<ffffffff81104e6d>] ? trace_hardirqs_on+0xd/0x10
[ 2932.455300]  [<ffffffff816a8863>] dm_ctl_ioctl+0x13/0x20
[ 2932.455300]  [<ffffffff8128316e>] do_vfs_ioctl+0x2ee/0x550
[ 2932.455300]  [<ffffffff81859791>] ? __schedule+0x281/0x980
[ 2932.455300]  [<ffffffff8128f706>] ? __fget_light+0x66/0x90
[ 2932.455300]  [<ffffffff81283449>] SyS_ioctl+0x79/0x90
[ 2932.455300]  [<ffffffff8186062e>] entry_SYSCALL_64_fastpath+0x12/0x76
[ 2932.455300] Code: 49 39 4d 28 0f 87 69 ff ff ff 48 8b 40 50 48 89 45 d0 4c 8b 6d d0 e8 36 69 fc ff 4d 39 ec 49 8d 45 b0 75 d8 31 c0 e9 48 ff ff ff <49> 8b 44 24 60 48 89 45 d0 48 8b 5d d0 e8 14 69 fc ff 48 8d 43 
[ 2932.455300] RIP  [<ffffffff8115e9ea>] css_next_child+0x12a/0x150
[ 2932.455300]  RSP <ffff88001856f9f8>
[ 2932.455300] CR2: 0000000000000060
[ 2932.455300] ---[ end trace 4cbbdef4e1a3ff81 ]---
[ 2932.455300] BUG: sleeping function called from invalid context at include/linux/sched.h:2729
[ 2932.455300] in_atomic(): 1, irqs_disabled(): 1, pid: 6720, name: lvm
[ 2932.455300] INFO: lockdep is turned off.
[ 2932.455300] irq event stamp: 0
[ 2932.455300] hardirqs last  enabled at (0): [<          (null)>]           (null)
[ 2932.455300] hardirqs last disabled at (0): [<ffffffff810a8b96>] copy_process.isra.28.part.29+0x5f6/0x1c40
[ 2932.455300] softirqs last  enabled at (0): [<ffffffff810a8b96>] copy_process.isra.28.part.29+0x5f6/0x1c40
[ 2932.455300] softirqs last disabled at (0): [<          (null)>]           (null)
[ 2932.455300] CPU: 0 PID: 6720 Comm: lvm Tainted: G      D         4.2.0-0.rc0.git2.1.fc23.x86_64 #1
[ 2932.455300] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.8.1-20150619_004852- 04/01/2014
[ 2932.455300]  0000000000000000 00000000f0a415c8 ffff88001856f678 ffffffff81856f03
[ 2932.455300]  0000000000000000 ffff88001b9b8000 ffff88001856f6a8 ffffffff810d9f91
[ 2932.455300]  ffff88001856f708 ffffffff81c79335 0000000000000aa9 0000000000000000
[ 2932.455300] Call Trace:
[ 2932.455300]  [<ffffffff81856f03>] dump_stack+0x4c/0x65
[ 2932.455300]  [<ffffffff810d9f91>] ___might_sleep+0x181/0x240
[ 2932.455300]  [<ffffffff810da09d>] __might_sleep+0x4d/0x90
[ 2932.455300]  [<ffffffff810be9b3>] exit_signals+0x33/0x150
[ 2932.455300]  [<ffffffff810ae8e9>] do_exit+0xc9/0xc30
[ 2932.455300]  [<ffffffff8111c271>] ? kmsg_dump+0x151/0x1a0
[ 2932.455300]  [<ffffffff8111c154>] ? kmsg_dump+0x34/0x1a0
[ 2932.455300]  [<ffffffff8102070e>] oops_end+0x9e/0xd0
[ 2932.455300]  [<ffffffff810701a5>] no_context+0x135/0x390
[ 2932.455300]  [<ffffffff81070488>] __bad_area_nosemaphore+0x88/0x200
[ 2932.455300]  [<ffffffff810dcee7>] ? finish_task_switch+0x57/0x230
[ 2932.455300]  [<ffffffff81070613>] bad_area_nosemaphore+0x13/0x20
[ 2932.455300]  [<ffffffff81070915>] __do_page_fault+0xd5/0x470
[ 2932.455300]  [<ffffffff810dcf26>] ? finish_task_switch+0x96/0x230
[ 2932.455300]  [<ffffffff810dcee7>] ? finish_task_switch+0x57/0x230
[ 2932.455300]  [<ffffffff81070cdf>] do_page_fault+0x2f/0x80
[ 2932.455300]  [<ffffffff81861277>] ? native_iret+0x7/0x7
[ 2932.455300]  [<ffffffff81862848>] page_fault+0x28/0x30
[ 2932.455300]  [<ffffffff8115e9ea>] ? css_next_child+0x12a/0x150
[ 2932.455300]  [<ffffffff8115e8ec>] ? css_next_child+0x2c/0x150
[ 2932.455300]  [<ffffffff81160710>] css_next_descendant_post+0x70/0xd0
[ 2932.455300]  [<ffffffff8140f510>] blk_throtl_drain+0x90/0x220
[ 2932.455300]  [<ffffffff8140f4b6>] ? blk_throtl_drain+0x36/0x220
[ 2932.455300]  [<ffffffff813e8068>] ? blk_queue_bypass_start+0x68/0xb0
[ 2932.455300]  [<ffffffff8140c35d>] blkcg_drain_queue+0x2d/0x60
[ 2932.455300]  [<ffffffff813e6689>] __blk_drain_queue+0xc9/0x1a0
[ 2932.455300]  [<ffffffff813e8068>] ? blk_queue_bypass_start+0x68/0xb0
[ 2932.455300]  [<ffffffff813e8072>] blk_queue_bypass_start+0x72/0xb0
[ 2932.455300]  [<ffffffff8140a1c9>] blkcg_deactivate_policy+0x39/0x130
[ 2932.455300]  [<ffffffff8140f7f8>] blk_throtl_exit+0x38/0x50
[ 2932.455300]  [<ffffffff8140c3c9>] blkcg_exit_queue+0x39/0x40
[ 2932.455300]  [<ffffffff813ee73e>] blk_release_queue+0x1e/0xc0
[ 2932.455300]  [<ffffffff8141f16a>] kobject_release+0x7a/0x190
[ 2932.455300]  [<ffffffff8141f01f>] kobject_put+0x2f/0x60
[ 2932.455300]  [<ffffffff813e9e5b>] blk_cleanup_queue+0x18b/0x280
[ 2932.455300]  [<ffffffff8169df9e>] cleanup_mapped_device+0xde/0x110
[ 2932.455300]  [<ffffffff816a0ea8>] __dm_destroy+0x1a8/0x2c0
[ 2932.455300]  [<ffffffff816a23f3>] dm_destroy+0x13/0x20
[ 2932.455300]  [<ffffffff816a7e7d>] dev_remove+0x10d/0x170
[ 2932.455300]  [<ffffffff816a7d70>] ? dev_suspend+0x280/0x280
[ 2932.455300]  [<ffffffff816a8582>] ctl_ioctl+0x282/0x550
[ 2932.455300]  [<ffffffff81104e6d>] ? trace_hardirqs_on+0xd/0x10
[ 2932.455300]  [<ffffffff816a8863>] dm_ctl_ioctl+0x13/0x20
[ 2932.455300]  [<ffffffff8128316e>] do_vfs_ioctl+0x2ee/0x550
[ 2932.455300]  [<ffffffff81859791>] ? __schedule+0x281/0x980
[ 2932.455300]  [<ffffffff8128f706>] ? __fget_light+0x66/0x90
[ 2932.455300]  [<ffffffff81283449>] SyS_ioctl+0x79/0x90
[ 2932.455300]  [<ffffffff8186062e>] entry_SYSCALL_64_fastpath+0x12/0x76
[ 2932.455300] note: lvm[6720] exited with preempt_count 2

Version-Release number of selected component (if applicable):

kernel 4.2.0-0.rc0.git2.1.fc23

How reproducible:

Unknown.

Steps to Reproduce:
1. Run the libguestfs test suite.

Actual results:

Lots of failures like the one above.
Look for "FAIL:" in the following file:
https://kojipkgs.fedoraproject.org//work/tasks/9782/10249782/build.log

Comment 1 Josh Boyer 2015-06-30 13:10:16 UTC
The first non-tainted oops is below.

Jeff, Mike, do either of you have any ideas on this?

[ 1293.900959] XFS (dm-0): Unmounting Filesystem
libguestfs: trace: umount_all = 0
libguestfs: trace: lvm_remove_all
guestfsd: main_loop: proc 47 (umount_all) took 0.33 seconds
guestfsd: main_loop: new request, len 0x28
lvm lvs -o vg_name,lv_name --noheadings --separator /
  /run/lvm/lvmetad.socket: connect failed: No such file or directory
  WARNING: Failed to connect to lvmetad. Falling back to internal scanning.
lvm lvchange -an /dev/VG/LV
  /run/lvm/lvmetad.socket: connect failed: No such file or directory
  WARNING: Failed to connect to lvmetad. Falling back to internal scanning.
[ 1296.560994] BUG: unable to handle kernel NULL pointer dereference at 0000000000000a00
[ 1296.560994] IP: [<ffffffff8140b1c0>] __blkg_lookup+0x40/0xe0
[ 1296.560994] PGD 0 
[ 1296.560994] Oops: 0000 [#1] SMP 
[ 1296.560994] Modules linked in: xfs nls_utf8 isofs btrfs xor raid6_pq kvm_amd kvm snd_pcsp snd_pcm snd_timer snd soundcore serio_raw ata_generic pata_acpi libcrc32c crc8 crc_itu_t crc_ccitt virtio_pci virtio_mmio virtio_input virtio_balloon virtio_scsi sym53c8xx scsi_transport_spi megaraid_sas megaraid_mbox megaraid_mm megaraid ideapad_laptop rfkill sparse_keymap video virtio_net virtio_gpu ttm drm_kms_helper drm virtio_console virtio_rng virtio_blk virtio_ring virtio crc32
[ 1296.560994] CPU: 0 PID: 2405 Comm: lvm Not tainted 4.2.0-0.rc0.git2.1.fc23.x86_64 #1
[ 1296.560994] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.8.1-20150619_004852- 04/01/2014
[ 1296.560994] task: ffff88001b9b8000 ti: ffff880013564000 task.ti: ffff880013564000
[ 1296.560994] RIP: 0010:[<ffffffff8140b1c0>]  [<ffffffff8140b1c0>] __blkg_lookup+0x40/0xe0
[ 1296.560994] RSP: 0018:ffff880013567a28  EFLAGS: 00000046
[ 1296.560994] RAX: 0000000000000001 RBX: 0000000000000000 RCX: 0000000000000001
[ 1296.560994] RDX: 0000000000000001 RSI: 0000000000000000 RDI: 0000000000000046
[ 1296.560994] RBP: ffff880013567a48 R08: 0000000000000000 R09: 0000000000000000
[ 1296.560994] R10: ffff88001b9b8000 R11: ffffffff81e56740 R12: 0000000000000000
[ 1296.560994] R13: ffffffff832bdb20 R14: 0000000000000000 R15: 0000000000000000
[ 1296.560994] FS:  00007faf4c6ac880(0000) GS:ffff88001ec00000(0000) knlGS:0000000000000000
[ 1296.560994] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 1296.560994] CR2: 0000000000000a00 CR3: 000000001bbde000 CR4: 00000000000007f0
[ 1296.560994] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 1296.560994] DR3: 0000000000000000 DR6: 0000000000000000 DR7: 0000000000000000
[ 1296.560994] Stack:
[ 1296.560994]  ffffffff832bdb20 ffff880019c2e800 ffff88001b870ed8 ffff88001b871930
[ 1296.560994]  ffff880013567a88 ffffffff8140f534 ffffffff8140f4b6 ffff88001b871930
[ 1296.560994]  ffff880013567a88 ffff88001b870ed8 ffffffff81ef1980 0000000000000000
[ 1296.560994] Call Trace:
[ 1296.560994]  [<ffffffff8140f534>] blk_throtl_drain+0xb4/0x220
[ 1296.560994]  [<ffffffff8140f4b6>] ? blk_throtl_drain+0x36/0x220
[ 1296.560994]  [<ffffffff8140c35d>] blkcg_drain_queue+0x2d/0x60
[ 1296.560994]  [<ffffffff813e6689>] __blk_drain_queue+0xc9/0x1a0
[ 1296.560994]  [<ffffffff813e8068>] ? blk_queue_bypass_start+0x68/0xb0
[ 1296.560994]  [<ffffffff813e8072>] blk_queue_bypass_start+0x72/0xb0
[ 1296.560994]  [<ffffffff8140a1c9>] blkcg_deactivate_policy+0x39/0x130
[ 1296.560994]  [<ffffffff8140f7f8>] blk_throtl_exit+0x38/0x50
[ 1296.560994]  [<ffffffff8140c3c9>] blkcg_exit_queue+0x39/0x40
[ 1296.560994]  [<ffffffff813ee73e>] blk_release_queue+0x1e/0xc0
[ 1296.560994]  [<ffffffff8141f16a>] kobject_release+0x7a/0x190
[ 1296.560994]  [<ffffffff8141f01f>] kobject_put+0x2f/0x60
[ 1296.560994]  [<ffffffff813e9e5b>] blk_cleanup_queue+0x18b/0x280
[ 1296.560994]  [<ffffffff8169df9e>] cleanup_mapped_device+0xde/0x110
[ 1296.560994]  [<ffffffff816a0ea8>] __dm_destroy+0x1a8/0x2c0
[ 1296.560994]  [<ffffffff816a23f3>] dm_destroy+0x13/0x20
[ 1296.560994]  [<ffffffff816a7e7d>] dev_remove+0x10d/0x170
[ 1296.560994]  [<ffffffff816a7d70>] ? dev_suspend+0x280/0x280
[ 1296.560994]  [<ffffffff816a8582>] ctl_ioctl+0x282/0x550
[ 1296.560994]  [<ffffffff81104e6d>] ? trace_hardirqs_on+0xd/0x10
[ 1296.560994]  [<ffffffff816a8863>] dm_ctl_ioctl+0x13/0x20
[ 1296.560994]  [<ffffffff8128316e>] do_vfs_ioctl+0x2ee/0x550
[ 1296.560994]  [<ffffffff8112532d>] ? debug_lockdep_rcu_enabled+0x1d/0x20
[ 1296.560994]  [<ffffffff8128f706>] ? __fget_light+0x66/0x90
[ 1296.560994]  [<ffffffff81283449>] SyS_ioctl+0x79/0x90
[ 1296.560994]  [<ffffffff8186062e>] entry_SYSCALL_64_fastpath+0x12/0x76
[ 1296.560994] Code: f4 41 89 d6 48 8b 9f 38 01 00 00 e8 6b a1 d1 ff 85 c0 74 09 80 3d f3 19 b5 00 00 74 56 48 85 db 74 09 4c 39 23 0f 84 9a 00 00 00 <49> 63 b4 24 00 0a 00 00 49 8d bd 28 01 00 00 e8 6c 73 01 00 48 
[ 1296.560994] RIP  [<ffffffff8140b1c0>] __blkg_lookup+0x40/0xe0
[ 1296.560994]  RSP <ffff880013567a28>
[ 1296.560994] CR2: 0000000000000a00
[ 1296.560994] ---[ end trace 4cbbdef4e1a3ff7c ]---
[ 1296.560994] BUG: sleeping function called from invalid context at include/linux/sched.h:2729
[ 1296.560994] in_atomic(): 1, irqs_disabled(): 1, pid: 2405, name: lvm
[ 1296.560994] INFO: lockdep is turned off.
[ 1296.560994] irq event stamp: 115080
[ 1296.560994] hardirqs last  enabled at (115079): [<ffffffff8185fbd0>] _raw_spin_unlock_irq+0x30/0x50
[ 1296.560994] hardirqs last disabled at (115080): [<ffffffff8185f99f>] _raw_spin_lock_irq+0x1f/0x80
[ 1296.560994] softirqs last  enabled at (115070): [<ffffffff810b18ba>] __do_softirq+0x39a/0x630
[ 1296.560994] softirqs last disabled at (115057): [<ffffffff810b1d9a>] irq_exit+0x11a/0x120

Comment 2 Josh Boyer 2015-06-30 13:11:31 UTC
*** Bug 1237137 has been marked as a duplicate of this bug. ***

Comment 3 Jeff Moyer 2015-06-30 13:35:33 UTC
My first guess would be the cgroup writeback patch set.

Comment 4 Richard W.M. Jones 2015-06-30 13:47:52 UTC
Any reasons why oopses would be tainted?  It's all free software
and I'm not doing any kind of kernel "poking" as far as I know.

Comment 5 Jeff Moyer 2015-06-30 14:19:14 UTC
(In reply to Richard W.M. Jones from comment #4)
> Any reasons why oopses would be tainted?  It's all free software
> and I'm not doing any kind of kernel "poking" as far as I know.

Oopses and BUGs will taint the kernel so that anyone looking at the stack traces will know they aren't looking at the first occurrence.  Usually, we only care about the first oops or bug that happened in the logs, since the rest are typically collateral damage.

Comment 6 Richard W.M. Jones 2015-07-03 10:12:39 UTC
FWIW still happens in 4.2.0-0.rc0.git4.1.fc23, with the same
stack trace.

Comment 7 Richard W.M. Jones 2015-07-22 14:26:55 UTC
Looks as if this has 'fixed itself' in 4.2.0-0.rc3.git1.1.fc24.

Comment 8 Mike Snitzer 2015-07-22 14:46:40 UTC
(In reply to Richard W.M. Jones from comment #7)
> Looks as if this has 'fixed itself' in 4.2.0-0.rc3.git1.1.fc24.

Quite likely it was one of Tejun's writeback fixes.

Comment 9 Richard W.M. Jones 2015-08-01 06:21:00 UTC
This one showed up again in Rawhide.

lvm vgchange -a n
  /run/lvm/lvmetad.socket: connect failed: No such file or directory
  WARNING: Failed to connect to lvmetad. Falling back to internal scanning.
[  112.461126] BUG: unable to handle kernel NULL pointer dereference at 00000000000009d0
[  112.461126] IP: [<ffffffff8140c3e0>] __blkg_lookup+0x40/0xe0
[  112.461126] PGD 0 
[  112.461126] Oops: 0000 [#1] SMP 
[  112.461126] Modules linked in: kvm_amd kvm snd_pcsp snd_pcm snd_timer snd soundcore serio_raw ata_generic pata_acpi libcrc32c crc8 crc_itu_t crc_ccitt virtio_pci virtio_mmio virtio_input virtio_balloon virtio_scsi sym53c8xx scsi_transport_spi megaraid_sas megaraid_mbox megaraid_mm megaraid ideapad_laptop rfkill sparse_keymap video virtio_net virtio_gpu ttm drm_kms_helper drm virtio_console virtio_rng virtio_blk virtio_ring virtio crc32
[  112.461126] CPU: 0 PID: 249 Comm: lvm Not tainted 4.2.0-0.rc4.git3.1.fc24.x86_64 #1
[  112.461126] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.8.2-20150714_191134- 04/01/2014
[  112.461126] task: ffff88001bb89d00 ti: ffff88001bafc000 task.ti: ffff88001bafc000
[  112.461126] RIP: 0010:[<ffffffff8140c3e0>]  [<ffffffff8140c3e0>] __blkg_lookup+0x40/0xe0
[  112.461126] RSP: 0018:ffff88001baffa28  EFLAGS: 00000046
[  112.461126] RAX: 0000000000000001 RBX: 0000000000000000 RCX: 0000000000000001
[  112.461126] RDX: 0000000000000001 RSI: 0000000000000000 RDI: 0000000000000046
[  112.461126] RBP: ffff88001baffa48 R08: 0000000000000000 R09: 0000000000000000
[  112.461126] R10: ffff88001bb89d00 R11: ffffffff81e58200 R12: 0000000000000000
[  112.461126] R13: ffffffff832c5b20 R14: 0000000000000000 R15: 0000000000000000
[  112.461126] FS:  00007f1ff7696880(0000) GS:ffff88001ec00000(0000) knlGS:0000000000000000
[  112.461126] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  112.461126] CR2: 00000000000009d0 CR3: 0000000019f11000 CR4: 00000000000006f0
[  112.461126] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[  112.461126] DR3: 0000000000000000 DR6: 0000000000000000 DR7: 0000000000000000
[  112.461126] Stack:
[  112.461126]  ffffffff832c5b20 ffff88001b9ea400 ffff88001bf31d50 ffff88001bf32778
[  112.461126]  ffff88001baffa88 ffffffff81410754 ffffffff814106d6 ffff88001bf32778
[  112.461126]  ffff88001baffa88 ffff88001bf31d50 ffffffff81ef3560 0000000000000000
[  112.461126] Call Trace:
[  112.461126]  [<ffffffff81410754>] blk_throtl_drain+0xb4/0x220
[  112.461126]  [<ffffffff814106d6>] ? blk_throtl_drain+0x36/0x220
[  112.461126]  [<ffffffff8140d57d>] blkcg_drain_queue+0x2d/0x60
[  112.461126]  [<ffffffff813e77b9>] __blk_drain_queue+0xc9/0x1a0
[  112.461126]  [<ffffffff813e9198>] ? blk_queue_bypass_start+0x68/0xb0
[  112.461126]  [<ffffffff813e91a2>] blk_queue_bypass_start+0x72/0xb0
[  112.461126]  [<ffffffff8140b4b9>] blkcg_deactivate_policy+0x39/0x100
[  112.461126]  [<ffffffff81410a18>] blk_throtl_exit+0x38/0x50
[  112.461126]  [<ffffffff8140d5e9>] blkcg_exit_queue+0x39/0x40
[  112.461126]  [<ffffffff813ef86e>] blk_release_queue+0x1e/0xc0
[  112.461126]  [<ffffffff8142042a>] kobject_release+0x7a/0x190
[  112.461126]  [<ffffffff814202df>] kobject_put+0x2f/0x60
[  112.461126]  [<ffffffff813eaf8b>] blk_cleanup_queue+0x18b/0x280
[  112.461126]  [<ffffffff816ae3c1>] cleanup_mapped_device+0xe1/0x110
[  112.461126]  [<ffffffff816b12a8>] __dm_destroy+0x1a8/0x2c0
[  112.461126]  [<ffffffff816b27f3>] dm_destroy+0x13/0x20
[  112.461126]  [<ffffffff816b827d>] dev_remove+0x10d/0x170
[  112.461126]  [<ffffffff816b8170>] ? dev_suspend+0x280/0x280
[  112.461126]  [<ffffffff816b8982>] ctl_ioctl+0x282/0x550
[  112.461126]  [<ffffffff813812dc>] ? SYSC_semtimedop+0x30c/0x1090
[  112.461126]  [<ffffffff816b8c63>] dm_ctl_ioctl+0x13/0x20
[  112.461126]  [<ffffffff8128361e>] do_vfs_ioctl+0x2ee/0x550
[  112.461126]  [<ffffffff811254ed>] ? debug_lockdep_rcu_enabled+0x1d/0x20
[  112.461126]  [<ffffffff8128fcd6>] ? __fget_light+0x66/0x90
[  112.461126]  [<ffffffff812838f9>] SyS_ioctl+0x79/0x90
[  112.461126]  [<ffffffff81870f2e>] entry_SYSCALL_64_fastpath+0x12/0x76
[  112.461126] Code: f4 41 89 d6 48 8b 9f 38 01 00 00 e8 0b 91 d1 ff 85 c0 74 09 80 3d 16 48 b5 00 00 74 56 48 85 db 74 09 4c 39 23 0f 84 9a 00 00 00 <49> 63 b4 24 d0 09 00 00 49 8d bd 28 01 00 00 e8 0c 74 01 00 48 
[  112.461126] RIP  [<ffffffff8140c3e0>] __blkg_lookup+0x40/0xe0
[  112.461126]  RSP <ffff88001baffa28>
[  112.461126] CR2: 00000000000009d0
[  112.461126] ---[ end trace 0f2782cad1fe2f18 ]---
[  112.461126] BUG: sleeping function called from invalid context at include/linux/sched.h:2739
[  112.461126] in_atomic(): 1, irqs_disabled(): 1, pid: 249, name: lvm
[  112.461126] INFO: lockdep is turned off.
[  112.461126] irq event stamp: 120680
[  112.461126] hardirqs last  enabled at (120679): [<ffffffff818704c0>] _raw_spin_unlock_irq+0x30/0x50
[  112.461126] hardirqs last disabled at (120680): [<ffffffff8187028f>] _raw_spin_lock_irq+0x1f/0x80
[  112.461126] softirqs last  enabled at (120670): [<ffffffff810b185a>] __do_softirq+0x39a/0x630
[  112.461126] softirqs last disabled at (120655): [<ffffffff810b1d3a>] irq_exit+0x11a/0x120
[  112.461126] CPU: 0 PID: 249 Comm: lvm Tainted: G      D         4.2.0-0.rc4.git3.1.fc24.x86_64 #1
[  112.461126] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.8.2-20150714_191134- 04/01/2014
[  112.461126]  0000000000000000 000000002bb716f3 ffff88001baff6a8 ffffffff81867795
[  112.461126]  0000000000000000 ffff88001bb89d00 ffff88001baff6d8 ffffffff810da131
[  112.461126]  ffff88001baff738 ffffffff81c7f5d5 0000000000000ab3 0000000000000000
[  112.461126] Call Trace:
[  112.461126]  [<ffffffff81867795>] dump_stack+0x4c/0x65
[  112.461126]  [<ffffffff810da131>] ___might_sleep+0x181/0x240
[  112.461126]  [<ffffffff810da23d>] __might_sleep+0x4d/0x90
[  112.461126]  [<ffffffff810be953>] exit_signals+0x33/0x150
[  112.461126]  [<ffffffff810ae889>] do_exit+0xc9/0xc30
[  112.461126]  [<ffffffff8111c431>] ? kmsg_dump+0x151/0x1a0
[  112.461126]  [<ffffffff8111c314>] ? kmsg_dump+0x34/0x1a0
[  112.461126]  [<ffffffff810206fe>] oops_end+0x9e/0xd0
[  112.461126]  [<ffffffff81070195>] no_context+0x135/0x390
[  112.461126]  [<ffffffff81070478>] __bad_area_nosemaphore+0x88/0x200
[  112.461126]  [<ffffffff81026e49>] ? sched_clock+0x9/0x10
[  112.461126]  [<ffffffff81070603>] bad_area_nosemaphore+0x13/0x20
[  112.461126]  [<ffffffff81070905>] __do_page_fault+0xd5/0x470
[  112.461126]  [<ffffffff811059cd>] ? __lock_acquire+0x63d/0x1d00
[  112.461126]  [<ffffffff81070ccf>] do_page_fault+0x2f/0x80
[  112.461126]  [<ffffffff81871b77>] ? native_iret+0x7/0x7
[  112.461126]  [<ffffffff81873148>] page_fault+0x28/0x30
[  112.461126]  [<ffffffff8140c3e0>] ? __blkg_lookup+0x40/0xe0
[  112.461126]  [<ffffffff8140c42d>] ? __blkg_lookup+0x8d/0xe0
[  112.461126]  [<ffffffff81410754>] blk_throtl_drain+0xb4/0x220
[  112.461126]  [<ffffffff814106d6>] ? blk_throtl_drain+0x36/0x220
[  112.461126]  [<ffffffff8140d57d>] blkcg_drain_queue+0x2d/0x60
[  112.461126]  [<ffffffff813e77b9>] __blk_drain_queue+0xc9/0x1a0
[  112.461126]  [<ffffffff813e9198>] ? blk_queue_bypass_start+0x68/0xb0
[  112.461126]  [<ffffffff813e91a2>] blk_queue_bypass_start+0x72/0xb0
[  112.461126]  [<ffffffff8140b4b9>] blkcg_deactivate_policy+0x39/0x100
[  112.461126]  [<ffffffff81410a18>] blk_throtl_exit+0x38/0x50
[  112.461126]  [<ffffffff8140d5e9>] blkcg_exit_queue+0x39/0x40
[  112.461126]  [<ffffffff813ef86e>] blk_release_queue+0x1e/0xc0
[  112.461126]  [<ffffffff8142042a>] kobject_release+0x7a/0x190
[  112.461126]  [<ffffffff814202df>] kobject_put+0x2f/0x60
[  112.461126]  [<ffffffff813eaf8b>] blk_cleanup_queue+0x18b/0x280
[  112.461126]  [<ffffffff816ae3c1>] cleanup_mapped_device+0xe1/0x110
[  112.461126]  [<ffffffff816b12a8>] __dm_destroy+0x1a8/0x2c0
[  112.461126]  [<ffffffff816b27f3>] dm_destroy+0x13/0x20
[  112.461126]  [<ffffffff816b827d>] dev_remove+0x10d/0x170
[  112.461126]  [<ffffffff816b8170>] ? dev_suspend+0x280/0x280
[  112.461126]  [<ffffffff816b8982>] ctl_ioctl+0x282/0x550
[  112.461126]  [<ffffffff813812dc>] ? SYSC_semtimedop+0x30c/0x1090
[  112.461126]  [<ffffffff816b8c63>] dm_ctl_ioctl+0x13/0x20
[  112.461126]  [<ffffffff8128361e>] do_vfs_ioctl+0x2ee/0x550
[  112.461126]  [<ffffffff811254ed>] ? debug_lockdep_rcu_enabled+0x1d/0x20
[  112.461126]  [<ffffffff8128fcd6>] ? __fget_light+0x66/0x90
[  112.461126]  [<ffffffff812838f9>] SyS_ioctl+0x79/0x90
[  112.461126]  [<ffffffff81870f2e>] entry_SYSCALL_64_fastpath+0x12/0x76
[  112.461126] note: lvm[249] exited with preempt_count 2
guestfsd: error: vgchange:   /run/lvm/lvmetad.socket: connect failed: No such file or directory

kernel 4.2.0-0.rc4.git3.1.fc24

Comment 11 Mike Snitzer 2015-08-01 16:37:22 UTC
(In reply to Richard W.M. Jones from comment #9)
> This one showed up again in Rawhide.

Tejun Heo needs to be made aware of this.

Comment 12 Richard W.M. Jones 2015-08-29 19:56:46 UTC
(In reply to Mike Snitzer from comment #11)
> (In reply to Richard W.M. Jones from comment #9)
> > This one showed up again in Rawhide.
> 
> Tejun Heo needs to be made aware of this.

How?

Anyway it's now started to affect Fedora 23 too.

lvm vgchange -a n
  /run/lvm/lvmetad.socket: connect failed: No such file or directory
  WARNING: Failed to connect to lvmetad. Falling back to internal scanning.
[   36.157672] BUG: unable to handle kernel NULL pointer dereference at 0000000000000558
[   36.157672] IP: [<ffffffff81389746>] __blkg_lookup+0x26/0x70
[   36.157672] PGD 0 
[   36.157672] Oops: 0000 [#1] SMP 
[   36.157672] Modules linked in: kvm_amd kvm snd_pcsp snd_pcm snd_timer snd soundcore serio_raw ata_generic pata_acpi libcrc32c crc8 crc_itu_t crc_ccitt virtio_pci virtio_mmio virtio_input virtio_balloon virtio_scsi sym53c8xx scsi_transport_spi megaraid_sas megaraid_mbox megaraid_mm megaraid ideapad_laptop rfkill sparse_keymap video virtio_net virtio_gpu ttm drm_kms_helper drm virtio_console virtio_rng virtio_blk virtio_ring virtio crc32
[   36.157672] CPU: 0 PID: 248 Comm: lvm Not tainted 4.2.0-0.rc8.git0.1.fc23.x86_64 #1
[   36.157672] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.8.2-20150714_191134- 04/01/2014
[   36.157672] task: ffff88001b4e2580 ti: ffff88001ac0c000 task.ti: ffff88001ac0c000
[   36.157672] RIP: 0010:[<ffffffff81389746>]  [<ffffffff81389746>] __blkg_lookup+0x26/0x70
[   36.157672] RSP: 0018:ffff88001ac0fa58  EFLAGS: 00000046
[   36.157672] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000
[   36.157672] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffffffff820176a0
[   36.157672] RBP: ffff88001ac0fa78 R08: ffff88001ac0c000 R09: ffff88001ce6e800
[   36.157672] R10: 0000000000000002 R11: 000000000001eaa5 R12: ffff88001cc57000
[   36.157672] R13: ffff88001ccf99c8 R14: ffff88001ccf9f38 R15: ffff88001ccba8d8
[   36.157672] FS:  00007f9a21711880(0000) GS:ffff88001f000000(0000) knlGS:0000000000000000
[   36.157672] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[   36.157672] CR2: 0000000000000558 CR3: 000000001b41d000 CR4: 00000000000006f0
[   36.157672] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[   36.157672] DR3: 0000000000000000 DR6: 0000000000000000 DR7: 0000000000000000
[   36.157672] Stack:
[   36.157672]  ffffffff82017700 ffffffff820176a0 ffff88001cc57000 ffff88001ccf99c8
[   36.157672]  ffff88001ac0faa8 ffffffff8138d14a ffff88001cc570b8 ffff88001ccf99c8
[   36.157672]  ffffffff81cb1c20 0000000000000000 ffff88001ac0fab8 ffffffff8138a108
[   36.157672] Call Trace:
[   36.157672]  [<ffffffff8138d14a>] blk_throtl_drain+0x5a/0x110
[   36.157672]  [<ffffffff8138a108>] blkcg_drain_queue+0x18/0x20
[   36.157672]  [<ffffffff81369a70>] __blk_drain_queue+0xc0/0x170
[   36.157672]  [<ffffffff8136a101>] blk_queue_bypass_start+0x61/0x80
[   36.157672]  [<ffffffff81388c59>] blkcg_deactivate_policy+0x39/0x100
[   36.157672]  [<ffffffff8138d328>] blk_throtl_exit+0x38/0x50
[   36.157672]  [<ffffffff8138a14e>] blkcg_exit_queue+0x3e/0x50
[   36.157672]  [<ffffffff8137016e>] blk_release_queue+0x1e/0xc0
[   36.157672]  [<ffffffff8139bcba>] kobject_release+0x7a/0x190
[   36.157672]  [<ffffffff8139bb6f>] kobject_put+0x2f/0x60
[   36.157672]  [<ffffffff8136a2b1>] blk_cleanup_queue+0x111/0x140
[   36.157672]  [<ffffffff815f13fc>] cleanup_mapped_device+0xdc/0x100
[   36.157672]  [<ffffffff815f2311>] __dm_destroy+0x161/0x260
[   36.157672]  [<ffffffff815f45d3>] dm_destroy+0x13/0x20
[   36.157672]  [<ffffffff815f9ebd>] dev_remove+0x10d/0x170
[   36.157672]  [<ffffffff815f9db0>] ? dev_suspend+0x280/0x280
[   36.157672]  [<ffffffff815fa572>] ctl_ioctl+0x232/0x4d0
[   36.157672]  [<ffffffff8130cd80>] ? SYSC_semtimedop+0x2b0/0xeb0
[   36.157672]  [<ffffffff810136f1>] ? __switch_to+0x261/0x4b0
[   36.157672]  [<ffffffff815fa823>] dm_ctl_ioctl+0x13/0x20
[   36.157672]  [<ffffffff8122ebd5>] do_vfs_ioctl+0x295/0x470
[   36.157672]  [<ffffffff8130b259>] ? sem_security+0x9/0x10
[   36.157672]  [<ffffffff8122ee29>] SyS_ioctl+0x79/0x90
[   36.157672]  [<ffffffff817750ae>] entry_SYSCALL_64_fastpath+0x12/0x71
[   36.157672] Code: eb bf 0f 1f 00 66 66 66 66 90 55 48 89 e5 41 55 41 54 53 48 83 ec 08 48 8b 87 c8 00 00 00 48 85 c0 74 05 48 39 30 74 45 48 89 f3 <48> 63 b6 58 05 00 00 49 89 fd 48 8d bf b8 00 00 00 41 89 d4 e8 
[   36.157672] RIP  [<ffffffff81389746>] __blkg_lookup+0x26/0x70
[   36.157672]  RSP <ffff88001ac0fa58>
[   36.157672] CR2: 0000000000000558
[   36.157672] ---[ end trace a6310b2924d6c01e ]---

Comment 13 Richard W.M. Jones 2015-09-06 18:24:13 UTC
Upstream fix is:

https://lkml.org/lkml/2015/9/5/205

I ran 1100 iterations with 4.3.0-0.rc0.git7.1.fc24.x86_64 + this
patch without hitting the problem.

Comment 14 Josh Boyer 2015-09-08 16:10:25 UTC
Fixed in Fedora git.  Will be in the 4.3-rc0-git9 build.

Also backported to F23.

Thanks much for the report and testing.

Comment 15 Fedora Update System 2015-09-22 12:05:10 UTC
kernel-4.2.1-300.fc23 has been submitted as an update to Fedora 23. https://bodhi.fedoraproject.org/updates/FEDORA-2015-16417

Comment 16 Fedora Update System 2015-09-23 03:55:21 UTC
kernel-4.2.1-300.fc23 has been pushed to the Fedora 23 testing repository. If problems still persist, please make note of it in this bug report.
If you want to test the update, you can install it with
$ su -c 'dnf --enablerepo=updates-testing update kernel'
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2015-16417

Comment 17 Fedora Update System 2015-09-24 05:07:36 UTC
kernel-4.2.1-300.fc23 has been pushed to the Fedora 23 stable repository. If problems still persist, please make note of it in this bug report.


Note You need to log in before you can comment on or make changes to this bug.