Description of problem: I'm hunting for some other crash with recent 4.0 kernels and I wanted to use kmemleak tracer. However it seems detector cannot be enabled. It's also unclear to me why there are 2 boot command line option processing. But one could notice: kmemleak: Kernel memory leak detector disabled kmemleak: Early log buffer exceeded (2329), please increase DEBUG_KMEMLEAK_EARLY_LOG_SIZE even thought they were enabled with 'kmemleak=on' on grub cfg. [ 0.000000] Initializing cgroup subsys cpuset [ 0.000000] Initializing cgroup subsys cpu [ 0.000000] Initializing cgroup subsys cpuacct [ 0.000000] Linux version 4.0.0-0.rc3.git0.1.fc23.x86_64+debug (mockbuild.fedoraproject.org) (gcc version 5.0.0 20150226 (Red Hat 5.0.0-0.18) (GCC) ) #1 SMP Mon Mar 9 13:22:53 UTC 2015 [ 0.000000] Command line: BOOT_IMAGE=/vmlinuz-4.0.0-0.rc3.git0.1.fc23.x86_64+debug root=/dev/mapper/vg_bot--frawhide---lv_root ro rd.md=0 rd.dm=0 rd.lvm.lv=vg_bot-frawhide-/lv_root KEYTABLE=us rd.lvm.lv=vg_bot-frawhide-/lv_swap rd.luks=0 SYSFONT=True LANG=en_US.UTF-8 console=ttyS1,115200n8 console=tty kmemleak=on [ 0.000000] Disabled fast string operations [ 0.000000] e820: BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009dbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009dc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x000000007a5fcfff] usable [ 0.000000] BIOS-e820: [mem 0x000000007a5fd000-0x000000007a5fffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffbc000-0x00000000ffffffff] reserved [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.4 present. [ 0.000000] DMI: Red Hat KVM, BIOS 0.5.1 01/01/2007 [ 0.000000] Hypervisor detected: KVM [ 0.000000] e820: update [mem 0x00000000-0x00000fff] usable ==> reserved [ 0.000000] e820: remove [mem 0x000a0000-0x000fffff] usable [ 0.000000] e820: last_pfn = 0x7a5fd max_arch_pfn = 0x400000000 [ 0.000000] MTRR default type: write-back [ 0.000000] MTRR fixed ranges enabled: [ 0.000000] 00000-9FFFF write-back [ 0.000000] A0000-BFFFF uncachable [ 0.000000] C0000-FFFFF write-protect [ 0.000000] MTRR variable ranges enabled: [ 0.000000] 0 base 00E0000000 mask FFE0000000 uncachable [ 0.000000] 1 disabled [ 0.000000] 2 disabled [ 0.000000] 3 disabled [ 0.000000] 4 disabled [ 0.000000] 5 disabled [ 0.000000] 6 disabled [ 0.000000] 7 disabled [ 0.000000] PAT not supported by CPU. [ 0.000000] found SMP MP-table at [mem 0x000fda30-0x000fda3f] mapped at [ffff8800000fda30] [ 0.000000] Scanning 1 areas for low memory corruption [ 0.000000] Base memory trampoline at [ffff880000097000] 97000 size 24576 [ 0.000000] init_memory_mapping: [mem 0x00000000-0x000fffff] ...[ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] setup_percpu: NR_CPUS:1024 nr_cpumask_bits:2 nr_cpu_ids:2 nr_node_ids:1 [ 0.000000] PERCPU: Embedded 481 pages/cpu @ffff880077800000 s1931096 r8192 d30888 u2097152 [ 0.000000] pcpu-alloc: s1931096 r8192 d30888 u2097152 alloc=1*2097152 [ 0.000000] pcpu-alloc: [0] 0 [0] 1 [ 0.000000] kvm-stealtime: cpu 0, msr 7780e3c0 [ 0.000000] Built 1 zonelists in Node order, mobility grouping on. Total pages: 493292 [ 0.000000] Policy zone: DMA32 [ 0.000000] Kernel command line: BOOT_IMAGE=/vmlinuz-4.0.0-0.rc3.git0.1.fc23.x86_64+debug root=/dev/mapper/vg_bot--frawhide---lv_root ro rd.md=0 rd.dm=0 rd.lvm.lv=vg_bot-frawhide-/lv_root KEYTABLE=us rd.lvm.lv=vg_bot-frawhide-/lv_swap rd.luks=0 SYSFONT=True LANG=en_US.UTF-8 console=ttyS1,115200n8 console=tty kmemleak=on [ 0.000000] PID hash table entries: 4096 (order: 3, 32768 bytes) [ 0.000000] Memory: 1918444K/2004580K available (8746K kernel code, 1347K rwdata, 3564K rodata, 3524K init, 16568K bss, 86136K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=2, Nodes=1 [ 0.000000] Hierarchical RCU implementation. [ 0.000000] RCU dyntick-idle grace-period acceleration is enabled. [ 0.000000] RCU lockdep checking is enabled. [ 0.000000] RCU restricting CPUs from NR_CPUS=1024 to nr_cpu_ids=2. [ 0.000000] RCU: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=2 [ 0.000000] Running RCU self tests [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] NR_IRQS:65792 nr_irqs:440 16 [ 0.000000] Offload RCU callbacks from all CPUs [ 0.000000] Offload RCU callbacks from CPUs: 0-1. [ 0.000000] Console: colour dummy device 80x25 [ 0.000000] console [tty0] enabled [ 0.000000] console [ttyS1] enabled [ 0.000000] Lock dependency validator: Copyright (c) 2006 Red Hat, Inc., Ingo Molnar [ 0.000000] ... MAX_LOCKDEP_SUBCLASSES: 8 [ 0.000000] ... MAX_LOCK_DEPTH: 48 [ 0.000000] ... MAX_LOCKDEP_KEYS: 8191 [ 0.000000] ... CLASSHASH_SIZE: 4096 [ 0.000000] ... MAX_LOCKDEP_ENTRIES: 32768 [ 0.000000] ... MAX_LOCKDEP_CHAINS: 65536 [ 0.000000] ... CHAINHASH_SIZE: 32768 [ 0.000000] memory used by lock dependency info: 8671 kB [ 0.000000] per task-struct memory footprint: 2688 bytes [ 0.000000] kmemleak: Early log buffer exceeded (2329), please increase DEBUG_KMEMLEAK_EARLY_LOG_SIZE [ 0.000000] tsc: Detected 2400.084 MHz processor Version-Release number of selected component (if applicable): 4.0.0-0.rc3.git0.1.fc23.x86_64+debug How reproducible: Steps to Reproduce: 1. boot in qemu guest 2. 3. Actual results: Expected results: Additional info:
(In reply to Zdenek Kabelac from comment #0) > Description of problem: > > I'm hunting for some other crash with recent 4.0 kernels and I wanted to use > kmemleak tracer. However it seems detector cannot be enabled. > > It's also unclear to me why there are 2 boot command line option processing. There isn't. > But one could notice: > > kmemleak: Kernel memory leak detector disabled > kmemleak: Early log buffer exceeded (2329), please increase > DEBUG_KMEMLEAK_EARLY_LOG_SIZE > > even thought they were enabled with 'kmemleak=on' on grub cfg. The problem here is that when you pass kmemleak=on, it enables kmemleak from the beginning of the boot. Before the kmemleak subsystem is initialized, it logs entries to a statically allocated buffer. If the boot logs more entries than the buffer can hold, then the logging code calls kmemleak_disable which is why you see the disabled printk. CONFIG_DEBUG_KMEMLEAK_EARLY_LOG_SIZE=1024 in our configs. We could adjust that up, but I'm not confident it will actually help you here. Unless you're trying to debug a leak issue that happens very early, perhaps you would be better off enabling kmemleak at runtime via the sysfs parameters.
Doing my own build with: CONFIG_DEBUG_KMEMLEAK_EARLY_LOG_SIZE=5000 let the kernel boot with running kmemleak: kmemleak: Kernel memory leak detector initialized kmemleak: Automatic memory scanning thread started
Hm. 5000 means the kernel allocates a static array of 5000 early_log structures. Each struct is 168 bytes on x86_64, which is ~821k worth of space. It is marked as __initdata, so theoretically the kernel should reclaim that. Will have to think about this a bit.
I've recompiled with 2048 - and it remained to work. Maybe for this variable it might be actually good if it could be somehow configured on boot grub line...
Is 'thinking' still in progress after 2 years ? Otherwise it makes no sense to even compile 'kmemleak' as it's simply useless.
This should have been updated in a rawhide build after Tuesday, can you see if that value works for you?
Superb I can confirm 4.10.0-0.rc7.git1.1.fc26.x86_64 which seems to be showing 4096 entries in /boot/config-4.10.0-0.rc7.git1.1.fc26.x86_64 DOES WORK!
Okay, closing this as rawhide, it will trickle down to releases with the rebase.
(In reply to Zdenek Kabelac from comment #2) > Doing my own build with: > > CONFIG_DEBUG_KMEMLEAK_EARLY_LOG_SIZE=5000 > > let the kernel boot with running kmemleak: > > kmemleak: Kernel memory leak detector initialized > kmemleak: Automatic memory scanning thread started How did you do your own build? Is it that you manually increased the buffer size to 5000 from 400 (in my case version 4.17) in all configuration files and then recompiled and rebooted or something else need to be done?