Bug 713675

Summary: F15 regression: perf top does not work on Intel(R) Pentium(R) 4 CPU 3.00GHz
Product: [Fedora] Fedora Reporter: Uros Bizjak <ubizjak>
Component: kernelAssignee: Kernel Maintainer List <kernel-maint>
Status: CLOSED ERRATA QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 15CC: dzickus, gansalmon, itamar, jonathan, kernel-maint, madhu.chinakonda
Target Milestone: ---   
Target Release: ---   
Hardware: i686   
OS: Linux   
Whiteboard:
Fixed In Version: kernel-2.6.41.1-1.fc15 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2011-11-17 23:28:47 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Uros Bizjak 2011-06-16 08:12:07 UTC
Description of problem:

"perf top" does not work on Intel(R) Pentium(R) 4 CPU 3.00GHz. It just sits there, without any action:

----------------------------------------------------------------------------------------------
   PerfTop:       0 irqs/sec  kernel:-nan%  exact: -nan% [1000Hz cycles],  (all, 2 CPUs)
----------------------------------------------------------------------------------------------

             samples  pcnt  DSO
             _______ _____  


[root@localhost uros]# 


Version-Release number of selected component (if applicable):

[root@localhost uros]# perf --version
perf version 2.6.38.8-32.fc15.i686

How reproducible:

>"perf top" as a root

Steps to Reproduce:
1.
2.
3.
  
Actual results:

----------------------------------------------------------------------------------------------
   PerfTop:       0 irqs/sec  kernel:-nan%  exact: -nan% [1000Hz cycles],  (all, 2 CPUs)
----------------------------------------------------------------------------------------------

             samples  pcnt  DSO
             _______ _____  


NMI interrupt counts are not increasing:

            CPU0       CPU1       
   0:        126          0   IO-APIC-edge      timer
   1:       1312          0   IO-APIC-edge      i8042
   7:          0          0   IO-APIC-edge      parport0
   8:          1          0   IO-APIC-edge      rtc0
   9:          0          0   IO-APIC-fasteoi   acpi
  14:      19444          0   IO-APIC-edge      ata_piix
  15:       4148          0   IO-APIC-edge      ata_piix
  16:      84993        600   IO-APIC-fasteoi   uhci_hcd:usb2, uhci_hcd:usb5, radeon
  17:      14008     260972   IO-APIC-fasteoi   Intel ICH5
  18:          0          0   IO-APIC-fasteoi   uhci_hcd:usb4
  19:          0          0   IO-APIC-fasteoi   uhci_hcd:usb3
  20:      11700          0   IO-APIC-fasteoi   eth0
  23:          2          0   IO-APIC-fasteoi   ehci_hcd:usb1
 NMI:          4          2   Non-maskable interrupts
 LOC:     222219     187651   Local timer interrupts
 SPU:          0          0   Spurious interrupts
 PMI:          4          2   Performance monitoring interrupts
 IWI:          0          0   IRQ work interrupts
 RES:      89204     101056   Rescheduling interrupts
 CAL:        322        464   Function call interrupts
 TLB:       1239       2265   TLB shootdowns
 TRM:          0          0   Thermal event interrupts
 THR:          0          0   Threshold APIC interrupts
 MCE:          0          0   Machine check exceptions
 MCP:          4          4   Machine check polls
 ERR:          0
 MIS:          0

Expected results:

Some output from the perf, it worked in F14.  NMIs should increase when "perf top" is running.

Additional info:

more cpuinfo
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 15
model		: 2
model name	: Intel(R) Pentium(R) 4 CPU 3.00GHz
stepping	: 9
cpu MHz		: 3000.000
cache size	: 512 KB
physical id	: 0
siblings	: 2
core id		: 0
cpu cores	: 1
apicid		: 0
initial apicid	: 0
fdiv_bug	: no
hlt_bug		: no
f00f_bug	: no
coma_bug	: no
fpu		: yes
fpu_exception	: yes
cpuid level	: 2
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe pebs bts cid xtpr
bogomips	: 5984.01
clflush size	: 64
cache_alignment	: 128
address sizes	: 36 bits physical, 32 bits virtual
power management:

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 15
model		: 2
model name	: Intel(R) Pentium(R) 4 CPU 3.00GHz
stepping	: 9
cpu MHz		: 3000.000
cache size	: 512 KB
physical id	: 0
siblings	: 2
core id		: 0
cpu cores	: 1
apicid		: 1
initial apicid	: 1
fdiv_bug	: no
hlt_bug		: no
f00f_bug	: no
coma_bug	: no
fpu		: yes
fpu_exception	: yes
cpuid level	: 2
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe pebs bts cid xtpr
bogomips	: 5983.76
clflush size	: 64
cache_alignment	: 128
address sizes	: 36 bits physical, 32 bits virtual
power management:

[    0.004082] CPU: Physical Processor ID: 0
[    0.004085] CPU: Processor Core ID: 0
[    0.004089] mce: CPU supports 4 MCE banks
[    0.004104] CPU0: Thermal monitoring enabled (TM1)
[    0.004747] ACPI: Core revision 20110112
[    0.009041] ftrace: allocating 22973 entries in 45 pages
[    0.010091] Enabling APIC mode:  Flat.  Using 1 I/O APICs
[    0.010387] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1
[    0.020920] CPU0: Intel(R) Pentium(R) 4 CPU 3.00GHz stepping 09
[    0.020998] Performance Events: Netburst events, Netburst P4/Xeon PMU driver.
[    0.020998] ... version:                0
[    0.020998] ... bit width:              40
[    0.020998] ... generic registers:      18
[    0.020998] ... value mask:             000000ffffffffff
[    0.020998] ... max period:             0000007fffffffff
[    0.020998] ... fixed-purpose events:   0
[    0.020998] ... event mask:             000000000003ffff
[    0.020998] NMI watchdog enabled, takes one hw-pmu counter.
[    0.020998] CPU 1 irqstacks, hard=f3ccc000 soft=f3cce000
[    0.020998] Booting Node   0, Processors  #1 Ok.
[    0.002999] Initializing CPU#1
[    0.002999] CPU1: Thermal monitoring handled by SMI
[    0.092040] NMI watchdog enabled, takes one hw-pmu counter.
[    0.092104] Brought up 2 CPUs

Comment 1 Jarod Wilson 2011-06-20 18:12:38 UTC
kernel-utils isn't a valid component, moving to kernel.

Comment 2 Chuck Ebbert 2011-06-24 14:46:32 UTC
Try disabling the nmi watchdog.

Comment 3 Uros Bizjak 2011-07-13 07:16:27 UTC
(In reply to comment #2)
> Try disabling the nmi watchdog.

Indeed, adding nmi_watchdog=0 to kernel command line restores perf top functionality.

Comment 4 Don Zickus 2011-10-13 18:40:45 UTC
*sigh* yes, nmi_watchdog, perf and p4 machines have a terrible history together.

this should be all fixed in 3.0 btw.  for now only one user of perf is allowed, either the nmi watchdog or perf itself.

nmi_watchdog=0 on the command line is one workaround

'echo 0 > /proc/sys/kernel/nmi_watchdog' is another.

Sorry about that.  3.0 has it fixed, which I believe is F16.

Cheers,
Don

Comment 5 Josh Boyer 2011-10-13 18:44:58 UTC
(In reply to comment #4) 
> Sorry about that.  3.0 has it fixed, which I believe is F16.

Actually, F15 is running 2.6.40.6 which is just 3.0.6 rebranded.

Uros, could you let us know if this is working for you now?

Comment 6 Uros Bizjak 2011-10-18 06:58:55 UTC
(In reply to comment #5)

> Actually, F15 is running 2.6.40.6 which is just 3.0.6 rebranded.
> 
> Uros, could you let us know if this is working for you now?

Yes, everything seems OK now on:

$ uname -r
2.6.40.6-0.fc15.i686

/proc/interrupts after short "perf top":

 NMI:        672        384   Non-maskable interrupts
 ...
 PMI:        672        384   Performance monitoring interrupts

Thanks!

Comment 7 Uros Bizjak 2011-10-18 07:09:08 UTC
(In reply to comment #6)
> (In reply to comment #5)
> 
> > Actually, F15 is running 2.6.40.6 which is just 3.0.6 rebranded.
> > 
> > Uros, could you let us know if this is working for you now?
> 
> Yes, everything seems OK now on:

Oops, I didn't notice that I still have nmi_watchdog=0 in kernel command line.

Unfortunately, removing this option, "perf top" does not work. It again just sits there, and NMI count isn't increasing.

   PerfTop:       0 irqs/sec  kernel:-nan%  exact: -nan% [1000Hz cycles],  (all, 2 CPUs)
-------------------------------------------------------------------------------

             samples  pcnt  DSO
             _______ _____

Comment 8 Don Zickus 2011-10-18 13:12:05 UTC
Hi Uros,

Sorry I suck.  The commit that fixed that was committed in 3.1 (f91298709790b9a483752ca3c967845537df).  Let me see how difficult it is to backport that to 2.6.40/3.0.

Cheers,
Don

Comment 9 Don Zickus 2011-10-19 18:32:30 UTC
I built a kernel with a patch in koji.

https://koji.fedoraproject.org/koji/taskinfo?taskID=3440337

Can you give it a try and let me know if that fixes your problem?

Cheers,
Don

Comment 10 Uros Bizjak 2011-10-20 09:39:33 UTC
(In reply to comment #9)
> I built a kernel with a patch in koji.
> 
> https://koji.fedoraproject.org/koji/taskinfo?taskID=3440337
> 
> Can you give it a try and let me know if that fixes your problem?

Yes, I can confirm that "perf top" works OK with the kernel from the link above:

$ uname -r
2.6.40.7-0.fc15.i686

 NMI:        630        824   Non-maskable interrupts
 ...
 PMI:        630        824   Performance monitoring interrupts

(I did double check that there is no extra option in kernel command line.)

Comment 11 Josh Boyer 2011-10-20 12:54:00 UTC
(In reply to comment #10)
> (In reply to comment #9)
> > I built a kernel with a patch in koji.
> > 
> > https://koji.fedoraproject.org/koji/taskinfo?taskID=3440337
> > 
> > Can you give it a try and let me know if that fixes your problem?
> 
> Yes, I can confirm that "perf top" works OK with the kernel from the link
> above:

Great!  Thank you for testing.  I'll include Don's patch in the next official F15 kernel build.

Comment 12 Fedora Update System 2011-10-24 15:12:39 UTC
kernel-2.6.40.7-3.fc15 has been submitted as an update for Fedora 15.
https://admin.fedoraproject.org/updates/kernel-2.6.40.7-3.fc15

Comment 13 Fedora Update System 2011-10-27 03:59:01 UTC
Package kernel-2.6.40.7-3.fc15:
* should fix your issue,
* was pushed to the Fedora 15 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing kernel-2.6.40.7-3.fc15'
as soon as you are able to, then reboot.
Please go to the following url:
https://admin.fedoraproject.org/updates/FEDORA-2011-14978
then log in and leave karma (feedback).

Comment 14 Fedora Update System 2011-10-27 14:12:52 UTC
kernel-2.6.40.8-2.fc15 has been submitted as an update for Fedora 15.
https://admin.fedoraproject.org/updates/kernel-2.6.40.8-2.fc15

Comment 15 Fedora Update System 2011-11-01 19:59:03 UTC
kernel-2.6.40.8-4.fc15 has been submitted as an update for Fedora 15.
https://admin.fedoraproject.org/updates/kernel-2.6.40.8-4.fc15

Comment 16 Fedora Update System 2011-11-12 00:08:06 UTC
kernel-2.6.41.1-1.fc15 has been submitted as an update for Fedora 15.
https://admin.fedoraproject.org/updates/kernel-2.6.41.1-1.fc15

Comment 17 Fedora Update System 2011-11-17 23:28:47 UTC
kernel-2.6.41.1-1.fc15 has been pushed to the Fedora 15 stable repository.  If problems still persist, please make note of it in this bug report.